Honcho
Sibyl vs Honcho: same 42k corpus, same 250-question suite
Sibyl passes more questions with 4.5x smaller average context and an estimated Sonnet cost about 3.5x lower.
Read the pageDashboard
Tests, costs, caveats and rerun files.
This page replaces the old raw dashboard with a public view: readable results, links to narrative pages, `.md` artifacts and rerun scripts.
Sibyl Scale365350/350
retrieval
Sibyl answers344/350
Sonnet answer-only
Honcho baseline243/250
Sibyl answers, 42k
Security audit1 high
2026-06-09 audit
Retrieval Scale365
Questions passed
100%Sibyl
Average context
Tokens sent
228tokens
Ingestion
Load 191k records
48sSibyl
Comparisons
Narrative pages by competitor.
Mem0
Sibyl vs Mem0: Scale365, 191k records and extraction cost
Mem0 retrieves 92/350 and reaches 105/350 after Sonnet, with 1,720 average context tokens and about 92 minutes of ingestion.
Read the pageHindsight
Sibyl vs Hindsight: good milestone recall, high context cost
Hindsight passes 152/350 in retrieval and answers, with 11,892 average context tokens and 98 minutes of ingestion.
Read the pageMnemosyne
Sibyl vs Mnemosyne: the most fragile methodological result
Mnemosyne passes 5/350 in retrieval and 55/350 after Sonnet, but this result should be read carefully because vectors are disabled.
Read the pageTest notes
The long-form pages.
Sibyl
Sibyl Scale365: the control run before the fight starts
I would rather not open with a leaderboard. The honest starting point is the control run: a year of synthetic business life, 500 companies, 1,500 stakeholders, 209,000 write calls, and the one question I keep coming back to. Can an agent retrieve current operational state without dumping the whole past into the prompt?
Read the test noteHoncho
Sibyl vs Honcho: the clean test with one async scar
Honcho is the page I would show first to a technical reader, because it is closest to a real head-to-head: same 42k corpus, same 250 questions, same Sonnet model. The scar is visible too: the runner probably under-waits Honcho async processing.
Read the test noteMem0
Sibyl vs Mem0: the setting that turns memory into a bill
The Mem0 story is really about cost, not a dunk. The local baseline I tested is real and reproducible, and limited on purpose. A stronger Mem0 mode exists, but the reports put a price tag on it: extraction stops being a footnote once the corpus hits 41M tokens.
Read the test noteHindsight
Sibyl vs Hindsight: recall works until context eats the room
Hindsight is the subtle page. It does not collapse like a bad retrieval toy. It finds meaningful memories. The problem is that it often sends so much context that Sonnet becomes the expensive part of the memory system.
Read the test noteMnemosyne
Sibyl vs Mnemosyne: the result that needs a warning label
This is the page where storytelling needs restraint. The number looks brutal, but the report also says vectors were disabled. The honest story is not victory-lap content. It is a warning label plus a next rerun.
Read the test noteSecurity
MCP, Hermes, symlinks, prefetch and audit.
Security
MCP isolation baseline: 31 strict runs, 4 confirmed symlink failures
25 PASS, 6 strict FAIL. Four failures are confirmed symlink/cross-profile isolation issues; two are harness exception failures.
Read the testSecurity
Hermes retests: prompt fence improved, symlink and learning issues remain open
The 0.4.17 retest is mixed: selected prompt-boundary escapes are fixed, while symlink isolation, migrated secret prefetch and learning-reference cases still need patch confirmation.
Read the testSecurity
Code quality and security audit: what is solid, what still needs work
1 high, 2 medium, 1 low. The code is considered well hardened for such a young project, but several points still need fixes.
Read the testArchive coverage
75 public entries are indexed outside the charts.
The dashboard shows the main measurements. The archive carries the extra security rounds, migration runs, smoke tests, audit notes and runner files.