Public archive
Every published test has a place to live.
The narrative pages explain the main comparisons. This archive is the workbench underneath: benchmark reports, security rounds, migration runs, audit notes, raw artifact indexes and rerun scripts.
75public entries
Grouped by method, benchmark, competitor, security, migration, audit and rerun file.
Methodology
4 public entries
Full methodology reviewClaude Code review of the competitive claims, caveats, competitor settings and public wording.Claude Code review transcriptEnglish transcript used to challenge the benchmark story before turning it into public pages.Executive benchmark reportHigh-level benchmark summary for Sibyl memory, Scale365 and the answer-only layer.Golden suiteRegression map for future Sibyl builds: core, dependent and paid API test groups.
Scale365 benchmark
6 public entries
Sibyl Scale365 retrieval500 companies, 365 days, 191k final records and 350 category retrieval questions.Sibyl Scale365 answer-onlyClaude Sonnet answer layer over the Sibyl Scale365 retrieved context.Token economy mega runPrompt-token reduction test showing the cost difference between top-k memory and full context.Three-arm demoA/B style demo comparing planner, Sibyl and full-context answers for public positioning.Product memory phase AProduct-memory core run focused on corrections, current state and answer behavior.Product memory answer-onlyClaude Sonnet answer-only layer for the product-memory core v3 run.
Competitors
13 public entries
Honcho vs Sibyl bundle42k-record baseline bundle with README, reports, raw outputs and the Honcho runner.Sibyl 42k Sonnet reportSibyl side of the 42k, 250-question Honcho comparison.Honcho 42k Sonnet reportHoncho side of the 42k, 250-question comparison, including context and cost numbers.Honcho 10-company retrieval baselineSmaller Honcho retrieval baseline used before the larger 42k comparison.Mem0 Scale365 retrievalLocal Mem0 baseline with Qdrant, fastembed, sparse BM25 and infer=False.Mem0 Scale365 answer-onlyClaude Sonnet answer layer over Mem0 retrieved context.Mem0 50-company retrievalSmaller Mem0 retrieval run used to inspect scale sensitivity and smoke-test setup.Hindsight Scale365 retrievalLocal Hindsight retrieval baseline with provider none, consolidation disabled and chunks off.Hindsight Scale365 answer-onlyClaude Sonnet answer layer over Hindsight recalled context.Hindsight 10-company smokeSmall Hindsight smoke run used to validate the harness before full Scale365.Mnemosyne Scale365 retrievalFTS-only Mnemosyne run, useful but explicitly marked as the fragile comparison.Mnemosyne Scale365 answer-onlyClaude Sonnet answer layer over Mnemosyne retrieved context.Mnemosyne 10-company smokeSmall Mnemosyne smoke run used to validate setup before the larger FTS-only test.
MCP security
14 public entries
MCP security repro packReproducible pack for MCP validation, storage, output hygiene and boundary tests.MCP isolation and migration suite31-run baseline covering profile isolation, symlink escapes, migration and dangerous paths.MCP input securityInput validation, malformed payloads and hardening behavior in the MCP surface.MCP profile isolationCross-profile isolation check for storage and memory reads.MCP runtime securityRuntime behavior and error-surface test for MCP calls.MCP search injection expandedSearch-injection and adversarial query behavior around memory retrieval.MCP path traversal namesPath-traversal style names and their effect on storage boundaries.MCP output hygieneError and output hygiene test for validation leaks and backend details.MCP secret log hygieneChecks whether secrets leak into logs or user-facing outputs.MCP prompt injection contentPrompt-injection content persisted in memory and reread through the MCP surface.MCP archive leakageArchive leakage and deep-history exposure test.MCP tenant isolationTenant isolation deep test around cross-scope retrieval and storage.MCP category isolationCategory isolation and category-boundary behavior in memory operations.MCP limit enforcementLimit enforcement test for resource bounds and oversized calls.
Migration
20 public entries
Migration clean preserves dataMigration run checking that a clean migration preserves existing data.Migration twice idempotentRepeated migration test to verify idempotence.Migration backup symlink outside HOMESymlinked backup path test for writes resolving outside the expected home directory.Migration DB symlink outside HOMEDatabase path symlink test for cross-home storage boundary escape.Migration credentials symlink outside HOMECredentials path symlink test for secret and profile boundary exposure.Migration memory dir symlink outside HOMEHome memory directory symlink test for writes outside the allowed profile.Migration backup path with spacesBackup path robustness test for paths containing spaces.Migration backup path is fileMigration failure behavior when the backup path points to a file.Migration DB path directoryMigration behavior when a database path resolves to a directory.Migration backup parent readonlyMigration behavior when the backup parent is readonly.Migration large existing DBLarge existing database migration test.Migration after invalid credentials JSONMigration behavior after corrupt credentials input.Migration backup dir many existing filesBackup directory collision and many-files behavior during migration.HOME symlink isolationIsolation test when HOME itself resolves through a symlink to another home.HOME parent symlink isolationIsolation test when the home parent resolves through a symlink.Env DB symlink isolationIsolation test for a database path provided through env and pointing at another home.Env DB parent symlink isolationIsolation test for an env-provided DB parent resolving to another home.DB hardlink isolationIsolation test for database hardlinks crossing the expected profile boundary.Storage TOCTOU DB symlinkTime-of-check to time-of-use test for database symlink replacement after validation.Storage TOCTOU dir symlinkTime-of-check to time-of-use test for directory symlink replacement after validation.
Hermes security
10 public entries
Hermes security round 6Hermes security series focused on memory, prompts and local boundary behavior.Hermes security round 7Follow-up Hermes security round for prompt and local profile hardening.Hermes security round 8Additional Hermes security round covering agent-facing memory behavior.Hermes learning security round 9Learning-provider and self-learning security checks.Hermes local boundary round 10Critical local-boundary candidates around install and uninstall behavior.Migration symlink secret round 11Critical symlink chain where migration can persist a secret into Sibyl memory.Hermes critical chain round 12Secret migration, learning provider prompt and subagent critical candidate chains.Hermes prefetch boundary round 13Prefetch marker and boundary escape test around stored content.Hermes learning authority round 14Tests whether untrusted journal material can become trusted learned reference.Hermes profile diagnostics round 15Profile-boundary diagnostics and learned-reference scope checks.
Audit
2 public entries
Rerun files
6 public entries
Public artifact manifestMachine-readable list of published benchmark artifacts, reports, raw outputs and scripts.Sibyl Scale365 runnerRunner for the Sibyl 365-day, 500-company category benchmark.Mem0 Scale365 runnerRunner for the local Mem0 Scale365 baseline.Hindsight Scale365 runnerRunner for the local Hindsight Scale365 baseline.Mnemosyne Scale365 runnerRunner for the Mnemosyne Scale365 baseline.Golden suite runnerRunner that groups high-value regression tests for a new Sibyl build.