# Sibyl Memory Benchmark Report

Date: 2026-06-07

## Executive Summary

Sibyl performed materially better than Mem0, Hindsight, and Mnemosyne on the 500-company, 365-day benchmark. The clearest signal is retrieval quality: Sibyl retrieved 350 / 350 expected answers, while Mem0 retrieved 92 / 350, Hindsight 152 / 350, and Mnemosyne 5 / 350.

The answer-only LLM layer did not change the main conclusion. Sibyl answered 344 / 350 with Claude Sonnet 4.6 on top of perfect retrieval. Mem0 improved slightly to 105 / 350, Hindsight stayed at 152 / 350, and Mnemosyne rose to 55 / 350, mostly because the LLM could answer negative-trap questions without retrieved facts.

The Phase A product-memory tests also support the same conclusion in a more realistic usage shape. In the 365-day real-usage extraction run, Sibyl extracted 600 / 600 conversations correctly, retrieved 140 / 140 answers, and answered 138 / 140 with Claude. The 2 answer failures were LLM omissions: the retrieved context contained the required facts.

## Headline Stats

| Test | Dataset | Retrieval | LLM answer-only | Estimated LLM cost |
| --- | --- | ---: | ---: | ---: |
| Sibyl benchmark | 500 companies, 365 days, 350 questions | 350 / 350 | 344 / 350 | $0.639780 |
| Mem0 benchmark | 500 companies, 365 days, 350 questions | 92 / 350 | 105 / 350 | $2.756979 |
| Hindsight benchmark | 500 companies, 365 days, 350 questions | 152 / 350 | 152 / 350 | $18.682209 |
| Mnemosyne benchmark | 500 companies, 365 days, 350 questions | 5 / 350 | 55 / 350 | $2.778108 |
| Sibyl Phase A1 | 50 companies, structured product memory, 120 questions | 120 / 120 | Not run | $0 |
| Sibyl Phase A2 | Same as A1, Claude answer-only | 120 / 120 baseline | 120 / 120 | $0.517665 |
| Sibyl Phase A3, 67 days | 50 companies, 200 raw conversations, 140 questions | 135 / 140 raw | 135 / 140 raw | $1.669770 |
| Sibyl Phase A3, 365 days | 50 companies, 600 raw conversations, 140 questions | 140 / 140 | 138 / 140 | $3.418686 |

## Key Takeaways

1. Sibyl's advantage is retrieval, not just answer formatting.
   On the main benchmark, Sibyl retrieved every expected fact from a 500-company, 365-day corpus. The competitors lost most of the benchmark before the LLM answer step.

2. More context is not automatically better.
   Hindsight sent far more context to Claude, averaging about 11,892 context tokens per question and costing $18.68 for answer-only. It still answered 152 / 350 because the retrieved context often missed the required structured facts or included too much irrelevant material.

3. A weak retrieval layer cannot be reliably fixed by an LLM.
   Mem0 improved from 92 / 350 retrieval to 105 / 350 answer-only. Mnemosyne improved from 5 / 350 to 55 / 350, but the gain came mostly from negative-trap questions, not from recovering missing positive facts.

4. Sibyl is strongest when the memory shape matches real operational work.
   The Phase A tests cover company state, people, provenance, corrections, aliases, agent tool continuity, and temporal reasoning. Sibyl passed the retrieval layer across structured ingest and raw-conversation extraction.

5. The 365-day Phase A run is the strongest production-style signal so far.
   It used one Claude extraction call per incoming conversation, stored extracted memory in Sibyl, then retrieved and answered later. Across 50 companies and 12 sessions per company, Sibyl extracted 600 / 600 conversations and retrieved 140 / 140 question contexts.

## Main Benchmark Breakdown

### Sibyl

- Retrieval: 350 / 350.
- LLM answer-only: 344 / 350.
- Retrieval was perfect across all 7 categories: context_stat, marker, milestone, negative_trap, role, status, and temporal_topic.
- The 6 answer-only misses came after successful retrieval, so they are answer-layer issues rather than memory retrieval failures.

### Mem0

- Retrieval: 92 / 350.
- LLM answer-only: 105 / 350.
- Strongest category: milestone, 42 / 50 retrieval.
- Weak categories: marker 0 / 50, negative_trap 0 / 50, context_stat 1 / 50.
- The LLM added only 13 additional correct answers over retrieval.

### Hindsight

- Retrieval: 152 / 350.
- LLM answer-only: 152 / 350.
- Strong categories: marker 50 / 50, milestone 50 / 50, temporal_topic 50 / 50.
- Weak categories: context_stat 0 / 50, negative_trap 0 / 50, role 1 / 50, status 1 / 50.
- Cost was high because the answer-only run used about 6.15M prompt tokens.

### Mnemosyne

- Retrieval: 5 / 350.
- LLM answer-only: 55 / 350.
- The LLM score rose mostly because negative_trap reached 50 / 50, while positive factual categories stayed near zero.
- This makes the answer-only result look better than the actual retrieval capability.

## Phase A Product-Memory Results

### A1: Core Retrieval

- Companies: 50.
- Questions: 120.
- Result: 120 / 120 retrieval.
- Cost: $0.
- Purpose: isolate Sibyl's memory engine with structured entities, states, references, and journal events.

### A2: Answer-Only On Retrieved Context

- Companies: 50.
- Questions: 120.
- Result: 120 / 120 answer-only.
- Cost: $0.517665.
- Purpose: verify that Claude can answer correctly when Sibyl provides the right context.

### A3: Real Usage Extraction, 67-Day Timeline

- Companies: 50.
- Conversations: 200.
- Questions: 140.
- Extraction: 200 / 200.
- Raw retrieval: 135 / 140.
- Raw answer-only: 135 / 140.
- Cost: $1.669770.
- Note: the 5 misses were caused by an oracle issue. The test forbade a phrase that was actually the correct latest next step for those accounts. This should be treated as a benchmark scoring defect, not a Sibyl retrieval defect.

### A3: Real Usage Extraction, 365-Day Timeline

- Timeline: 2026-02-02 to 2027-02-01.
- Companies: 50.
- Sessions per company: 12.
- Conversations: 600.
- Questions: 140.
- Extraction: 600 / 600.
- Retrieval: 140 / 140.
- Answer-only: 138 / 140.
- Cost: $3.418686.
- The 2 answer failures were LLM omissions in multi-session answers. In both cases, Sibyl retrieved the required next-step fact, but Claude did not include it in the final answer text.

## Why Sibyl Beats The Competitors On This Benchmark

Sibyl stores operational memory as typed records: entities, state documents, journal events, and references. That matters for this benchmark because many questions ask for exact company state, current versus previous status, people, source references, aliases, negative traps, and temporal updates.

Mem0, Hindsight, and Mnemosyne behave more like general memory or retrieval layers in these runs. They can find some text matches, but they do not consistently preserve the current operational state of a company across a long, noisy timeline. Once the retrieval layer misses the structured fact, the LLM has little chance to answer reliably.

## Limitations

- The benchmark is synthetic and deterministic. It is useful for isolating memory behavior, but it is not a full substitute for messy real customer data.
- The competitor runs used the integrations and settings available in this environment. Different production configurations could change their scores.
- The answer-only runs measure Claude answering from retrieved context. They do not measure a full autonomous agent loop with retries, repair, or tool planning.
- The Phase A3 extraction runs include Claude as the extractor, so their total cost includes both extraction and answering.

## Source Runs

- Sibyl retrieval: `runs/run-2026-06-06-sibyl-365d-500c-50q-category-retrieval.md`
- Sibyl answer-only: `runs/run-2026-06-06-sibyl-365d-500c-50q-category-sonnet46.md`
- Mem0 retrieval: `runs/run-2026-06-06-mem0-365d-500c-50q-category-option1.md`
- Mem0 answer-only: `runs/run-2026-06-06-mem0-365d-500c-sonnet46-answeronly.md`
- Hindsight retrieval: `runs/run-2026-06-07-hindsight-365d-500c-50q-category-retrieval.md`
- Hindsight answer-only: `runs/run-2026-06-07-hindsight-365d-500c-sonnet46-answeronly.md`
- Mnemosyne retrieval: `runs/run-2026-06-07-mnemosyne-365d-500c-50q-category-retrieval.md`
- Mnemosyne answer-only: `runs/run-2026-06-07-mnemosyne-365d-500c-sonnet46-answeronly.md`
- Sibyl Phase A1: `runs/run-2026-06-07-sibyl-phase-a-product-memory-core-v3.md`
- Sibyl Phase A2: `runs/run-2026-06-07-sibyl-phase-a-product-memory-core-v3-sonnet46-answeronly.md`
- Sibyl Phase A3, 67 days: `runs/run-2026-06-07-sibyl-phase-a-real-usage-extraction.md`
- Sibyl Phase A3, 365 days: `runs/run-2026-06-07-sibyl-phase-a-real-usage-extraction-1y.md`