# Honcho vs Sibyl 42k Baseline

This folder contains the complete files needed to share the 1:1 baseline.

## Files

1. `01-sibyl-42k-sonnet-report.md`
   - Original Sibyl 42k report.
   - 42,000 memory records, 250 questions, Claude Sonnet 4.6 answering.

2. `02-sibyl-42k-sonnet-raw_result.json`
   - Full raw result for the Sibyl run.
   - Includes per-question scoring, retrieval flags, answer scoring, token estimates, and cost estimates.

3. `03-honcho-42k-sonnet-baseline-report.md`
   - Honcho baseline report generated from the same 42k corpus and same 250 questions.
   - Uses Honcho retrieval plus Claude Sonnet 4.6 answering.

4. `04-honcho-42k-sonnet-baseline-raw_result.json`
   - Full raw result for the Honcho baseline.
   - Includes per-question retrieval context, answer scoring, token estimates, and cost estimates.

5. `05-honcho-baseline-runner.py`
   - Script used to generate the Honcho baseline from the existing Sibyl source artifact.

## Headline Result

| Metric | Sibyl | Honcho |
|---|---:|---:|
| Retrieval passed | 243 / 250 | 219 / 250 |
| Claude Sonnet answer passed | 243 / 250 | 214 / 250 |
| Average retrieved context | 291 tokens/query | 1,313 tokens/query |
| Estimated Sonnet cost | $0.53 | $1.83 |

Both systems used the same 42,000-record corpus, the same 250-question suite, and Claude Sonnet 4.6 for answer generation.

## Category Pattern

| Category | Sibyl | Honcho retrieval | Honcho + Sonnet |
|---|---:|---:|---:|
| status | 48 / 48 | 40 / 48 | 40 / 48 |
| milestone | 48 / 48 | 48 / 48 | 48 / 48 |
| marker | 48 / 48 | 48 / 48 | 48 / 48 |
| role | 48 / 48 | 48 / 48 | 48 / 48 |
| negative controls | 10 / 10 | 10 / 10 | 10 / 10 |
| segment | 41 / 48 | 25 / 48 | 20 / 48 |

Main takeaway: on the same dataset, questions, and Sonnet model, Sibyl achieved higher accuracy while retrieving much less context and using lower estimated Sonnet cost.