Comparison | Honcho
Sibyl vs Honcho: same 42k corpus, same 250-question suite
A 1:1 comparison between Sibyl and Honcho on 42,000 records, 250 questions and Claude Sonnet 4.6.
Full test note
Honcho is the page I would show first to a technical reader, because it is closest to a real head-to-head: same 42k corpus, same 250 questions, same Sonnet model. The scar is visible too: the runner probably under-waits Honcho async processing.
Read the full test pageVerdict
Sibyl passes more questions with 4.5x smaller average context and an estimated Sonnet cost about 3.5x lower.
This is the cleanest comparison in the set: same 42k corpus, same 250-question suite, same answering model. The weak point to flag is the Honcho async wait time, probably too short according to the methodology review.
Comparison table
| Metric | Sibyl | Honcho |
|---|---|---|
| Retrieval passed | 243/250 | 219/250 |
| Sonnet answers passed | 243/250 | 214/250 |
| Average context | 291 tokens | 1,313 tokens |
| Estimated Sonnet cost | $0.53 | $1.83 |
Methodological caveat
The current-state tier favors Sibyl. Honcho is append-only in this test, and the runner uses `sleep(3)`, which may underestimate asynchronous processing.
Why not the best possible configuration?
- Honcho was tested as an SDK baseline on the existing corpus, not as a heavily tuned product integration.
- The report recommends rerunning with a more generous async wait before using the result as a hard external claim.
Rerun the test
python runs/honcho-vs-sibyl-42k-baseline/05-honcho-baseline-runner.py