# Sibyl Noir - Three-Arm Memory Demo

Run: `run-2026-06-04-sibyl-hackathon-three-arm-demo`

## What this shows

The same natural, paraphrased question is answered three ways:

- `no memory`: the detective gets no case file. It cannot answer. This is the floor.
- `full context`: the detective gets the entire 300-statement case
  file on every question. Correct, but heavy and expensive, and accuracy degrades as the real
  evidence is buried among distractors.
- `planner+Sibyl`: a Sonnet planner turns the question into a retrieval key, Sibyl returns the top
  5 statements, and the detective answers from those only. Correct, and
  far cheaper. Planner tokens are counted in this arm.

## Scoreboard

| Arm | Total tokens | Total cost | Correct |
| --- | ---: | ---: | ---: |
| no memory | 688 | $0.0046 | 0 / 3 |
| full context | 73443 | $0.2259 | 3 / 3 |
| planner+Sibyl | 3991 | $0.0190 | 3 / 3 |

planner+Sibyl matched full-context accuracy (3/3 vs 3/3) using 94.6% fewer tokens and 11.9x cheaper.

## Per-question detail

### Q1: Where did the cook say she was around half past nine?

| Arm | Correct | Tokens | Cost | Answer |
| --- | :-: | ---: | ---: | --- |
| no memory | no | 228 | $0.0015 | There are no witness statements in the provided context, so I have no evidence to answer where the cook said she was around half past nine. |
| full context | yes | 24476 | $0.0752 | Ada Cross (the cook) gave conflicting accounts of her whereabouts at 21:30: in round 1 she said she was in the kitchen, and in round 2 she ... |
| planner+Sibyl | yes | 1316 | $0.0062 | Ada Cross (the cook) gave two conflicting locations for 21:30: in Round 1 she said she was in the kitchen, and in Round 2 she said she was ... |

### Q2: Did the banker change his story near quarter to ten?

| Arm | Correct | Tokens | Cost | Answer |
| --- | :-: | ---: | ---: | --- |
| no memory | no | 227 | $0.0015 | There are no witness statements in the provided context, so I have no evidence to determine whether the banker changed his story near quart... |
| full context | yes | 24475 | $0.0752 | Yes, the banker Ben Lyle changed his story at 21:45 (quarter to ten). In round 1 he placed himself in the library, but in round 2 he claime... |
| planner+Sibyl | yes | 1334 | $0.0064 | Yes, the banker Ben Lyle contradicted himself about his location at 21:45 (quarter to ten). In Round 1 he placed himself in the library, bu... |

### Q3: Where was the gardener at ten in the evening?

| Arm | Correct | Tokens | Cost | Answer |
| --- | :-: | ---: | ---: | --- |
| no memory | no | 233 | $0.0016 | There are no witness statements in the provided context, so I have no evidence about the gardener's whereabouts at ten in the evening and c... |
| full context | yes | 24492 | $0.0755 | The gardener, Cora Voss, gave contradictory statements about her location at 22:00 (ten in the evening): in round 1 she claimed to be in th... |
| planner+Sibyl | yes | 1341 | $0.0064 | Cora Voss (the gardener) gave two different locations for 22:00: the garage in round 1 and the garden in round 2. This is a direct contradi... |

## Raw

Full answers and token usage per arm are in the `.raw_result.json` next to this file.