Competitor test
Sibyl vs Honcho: the clean test with one async scar
A direct 42k-record comparison with the same 250-question suite, the same answering model and a clear caveat around Honcho async processing.
Thesis
Honcho is the page I would show first to a technical reader, because it is closest to a real head-to-head: same 42k corpus, same 250 questions, same Sonnet model. The scar is visible too: the runner probably under-waits Honcho async processing.
What I pulled from the reports
- The bundle README says both systems used the same 42,000-record corpus, the same 250-question suite and Claude Sonnet 4.6.
- The Honcho baseline report records 219/250 Honcho retrieval, 214/250 Honcho + Sonnet answers, 1,312.52 average Honcho context tokens and $1.830699 estimated answer cost.
- The Sibyl side records 243/250 passed, 290.76 average context tokens and $0.534255 estimated answer cost.
- The methodology review flags two Honcho-specific objections: status favors current-state construction, and ingestion used `time.sleep(3)`, likely too short for async processing.
Benchmark signal
Plugin settings
same source corpus for both systems
same benchmark questions
same answer-only evaluator
append-only memory in this run
async wait may be too short
What I am comparing
The cleanest comparison starts with a boring sentence: same corpus, same questions, same answer model. That boring sentence is the reason this page matters.
There are 42,000 generated memory records, 250 questions and a Claude Sonnet 4.6 answer step. Sibyl retrieves its context. Honcho retrieves its context. Then the answer layer has to live with what retrieval gave it.
The Honcho settings
Honcho is run as an SDK baseline. In this test, it behaves like an append-only memory layer. Sibyl, by contrast, is using current-state structure when the question asks for current state.
That difference is not a bug in Honcho, it is the design gap the test is meant to expose. The async wait is the more fragile part: the report says the runner used `time.sleep(3)`, and the methodology review thinks that may be too short for 42k records.
- Same 42k corpus.
- Same 250 questions.
- Same Sonnet answer model.
- Honcho async processing may be under-waited.
Why Sibyl is tested in its strong mode
Sibyl is not pretending to be raw search here. I test it as the thing I am actually building: a memory stack that keeps current state accessible and hands the model a small, answerable context.
That is also why Honcho is not the villain here. The problem is not that Honcho is bad, it is that append-only memory carries more history than the question needs, and the surplus turns into prompt cost.
Where the test turns
The category table is more useful than the headline. Honcho is perfect on milestone, marker, role and negative controls. The pain shows up in status and segment questions, exactly where current state and clean segmentation matter.
The X version is short: less context, more correct answers, lower bill. The serious version is narrower and stronger: on this state-heavy benchmark, Sibyl's state-first retrieval gives Sonnet a smaller and more useful prompt.
Fairness note
This is the strongest competitor page, but I would still rerun Honcho with a longer async wait before turning it into a hard external claim. The page is strong enough to explain the signal. It is not the end of the investigation.
Rerun the test
python runs/honcho-vs-sibyl-42k-baseline/05-honcho-baseline-runner.py