# Sibyl Memory Quality: Full Test Review and Competitive Benchmark Assessment

- **Date**: 2026-06-10
- **Scope**: all memory quality test runs in `runs/` (late May to June 8), the four competitive benchmarks (Honcho, Mem0, Hindsight, Mnemosyne), and the 2026-06-09 security and code quality audit.
- **Goal**: answer two questions. Is the plugin good on memory quality? Are the competitive benchmarks defensible?

## 1. Executive Summary

**Memory quality: strong, with one structural condition.** Write integrity, retention, and exact recall are essentially flawless at every scale tested (up to 191,000 records). Retrieval reaches 100% on the hardest internal benchmarks, but only through the full retrieval stack (intent planner + exact-entity/state fallback + repair resolver). Raw natural-language search without that stack scores near zero. The product is the stack, not the bare search.

**Competitive benchmarks: directionally right, not yet publication-proof.** Sibyl wins every comparison by a wide margin, but the protocol gives Sibyl a home-field advantage that a competitor could attack. The cost and efficiency claims, however, are architectural and effectively unattackable. One cheap fix (a Mnemosyne hybrid rerun at 50 companies) would close the only objection that costs nothing to raise.

## 2. Internal Memory Quality Results

### What is excellent

| Area | Result |
|---|---|
| Write integrity at scale | 42,000/42,000 writes (200 companies, 180 days), zero loss after restart; 191,000 records ingested in 47.6 s on Scale365 |
| Chronology retention | 1,152/1,152 writes, 8/8 checks, 40/40 milestone recalls after restart |
| Dense and scale 10x (paid tier) | 100% of writes up to 11,520; all checks pass. Free-tier failures are the 2 MB cap, not data quality |
| Mega benchmark (42k records, 250 questions) | 243/250 baseline, 248/250 with hybrid exact fallback, 250/250 after follow-up repair (the last 2 were date formatting, retrieval was correct) |
| Scale365 benchmark (191k records, 350 questions, 7 categories) | 350/350 retrieval, 344/350 Sonnet answers, 228 avg context tokens, $0.64 total |
| High-density retrieval (mcp-run series) | Runs 16/17 initially collapsed (16% and 11% search pass at 450 to 770 entities). The agent-side repair resolver fixed it completely: runs 25 to 29 at 100% pass, recall 1.00, zero hallucination, up to 100 companies / 1,520 entities |
| Natural paraphrase via Sonnet planner | Role aliases, unseen time phrasings: 20/20 and 15/15 where the deterministic planner scores 0/20. All four hackathon go/no-go gates passed |

### Known weaknesses (all documented in runs)

1. **Raw natural-language search without the planner is a non-starter**: 0/32 on non-canonical business memory, 0/10 on the retrieval go/no-go. Everything recovers to 100% with the exact-state fallback, but the planner layer is mandatory.
2. **Product Update Suite: 3/7.** Vague current-state queries (new owner, current priority, paraphrased buried details) do not surface in direct search. Same root cause as point 1.
3. **Self-learning poisoning unguarded** (Track 4): 10 untrusted proposals accepted with no safeguard.
4. **Ambiguous queries return 0 hits** instead of asking for clarification (UX round 2 verdict: NEEDS_UX_PATCHES).
5. **Symlink isolation**: 6/31 failures in the isolation/migration suite (cross-HOME write leakage through symlinks).
6. From the 2026-06-09 audit (security, not memory quality): 1 HIGH finding, silent config destruction on corrupted settings during setup.

## 3. Competitive Benchmark Results

### Honcho (2026-06-05, mega 42k corpus, 250 questions)

| Metric | Sibyl | Honcho |
|---|---:|---:|
| Retrieval passed | 243/250 (97.2%) | 219/250 (87.6%) |
| Sonnet answers passed | 243/250 | 214/250 (85.6%) |
| Avg context tokens | 291 | 1,313 (4.5x) |
| Estimated Sonnet cost | $0.53 | $1.83 |

### Scale365 (500 companies, 191k records, 365 days, 350 questions, 7 categories)

| System | Retrieval | Sonnet answers | Avg ctx tokens | LLM cost | Ingestion |
|---|---:|---:|---:|---:|---:|
| **Sibyl** | 350/350 (100%) | 344/350 (98%) | 228 | $0.64 | 48 s |
| Hindsight | 152/350 (43%) | 152/350 | 11,892 | $18.68 | 98 min |
| Mem0 | 92/350 (26%) | 105/350 | 1,720 | $2.76 | 92 min |
| Mnemosyne | 5/350 (1.4%) | 55/350 | 1,662 | $2.78 | 6 h 12 |

All four ran the same corpus, same questions, same substring scoring, same Sonnet 4.6 answer step.

## 4. Methodology Assessment

### What holds up

- True 1:1 on corpus, questions, scoring, model, temperature, and follow-up repair policy. Raw per-question results and runner scripts are preserved and reproducible.
- The cost and efficiency comparison is architectural and robust: 228 context tokens vs 1,700 to 12,000; $0.64 vs $2.76 to $18.68; 48 s local ingestion vs 1.5 to 6+ hours. No configuration choice can explain that away.
- Negative-trap categories and documented competitor failure modes are good practice.

### Where the protocol is vulnerable

1. **Asymmetric intelligence.** Sibyl runs with its app-side hybrid exact fallback routed by question intent. Competitors run with their intelligence disabled: Mem0 with `infer=False` (its core feature), Hindsight with `llm_provider="none"` and consolidation off, Mnemosyne with `fts_weight=1.0, vec_weight=0.0` (vectors cut entirely). The comparison is Sibyl-with-its-brain vs competitors-without-theirs.
2. **Home-field data.** The corpus is synthetic, generated in Sibyl's record format (entity/state/journal tiers, structured JSON, markers). Competitors ingest JSON blobs that do not resemble their natural input (conversations).
3. **Canonical queries only.** Retrieval queries use exact entity names, which is exactly Sibyl's sweet spot. Our own tests show Sibyl scores 0/32 on non-canonical raw queries; the non-canonical suite was never run against competitors.
4. **Honcho-specific**: the status category is won by data-model construction (state tier last-write-wins vs append-only duplicates), and ingestion used a `time.sleep(3)` wait, likely too short for Honcho's async processing of 42k records.

### The cost rebuttal: why most objections die anyway

Enabling the disabled features has a price, computed from the actual corpus (191k records, ~215 tokens each, ~41M tokens minimum through any extraction LLM):

| Objection | Cost to enable | Verdict |
|---|---|---|
| Mem0 `infer=True` | ~$50-100 (gpt-5-mini) to ~$445 (Sonnet), plus 8 to 25 h of ingestion | Self-defeating: this is precisely the architectural cost Sibyl avoids ($0, 48 s). Note: LLM extraction paraphrases content and may destroy the exact markers the scoring requires, so it could even lower the score |
| Hindsight LLM provider + consolidation | Same order of magnitude, plus its already 7x higher read-side cost ($18.68/run) | Same self-defeating economics |
| Mnemosyne `vec_weight > 0` | **~$0** (local embeddings, or under $1 via API). Requires re-ingestion (~6 h+ machine time at 500c; the previous DB in /tmp is gone) | **The one objection not protected by economics.** Anyone can rerun this cheaply and the 1.4% will not survive |

Mnemosyne's 6 h 12 ingestion (28x slowdown from 50c to 500c) is itself a publishable finding, but the FTS-only configuration is not defensible as a product comparison.

## 5. Recommendations (prioritized)

1. **Publish the cost/efficiency claim now.** Tokens, dollars, and ingestion time are the unattackable core of the story. The three-arm demo (no memory / full context / planner+Sibyl at 1/12th the cost, same accuracy) is the best pitch asset.
2. **Rerun Mnemosyne at 50 companies in hybrid mode** (`fts/vec 0.5/0.5`), ~45 min total. If hybrid scores 40 to 60%, publish 500c with a transparent config note; if it stays under 20%, the current number is defensible as-is. Do not publish the 1.4% without this.
3. **Add one disclosure line per competitor** stating the configuration and the cost to enable the disabled features (the table above). It converts the main methodological criticism into a selling point.
4. **Close the Honcho ingestion objection**: rerun with a generous async wait instead of `sleep(3)`.
5. **Run one neutral public benchmark** (LongMemEval or LoCoMo) before any external-facing claim of general superiority. All current benchmarks are home-made.
6. Product-side, before wider exposure: poisoning safeguards for self-learning, ambiguity detection in UX, and the HIGH audit finding (silent config destruction in setup).

## 6. Bottom Line

Yes, the plugin is genuinely good on memory quality: perfect write integrity, perfect retention at 191k records, 100% retrieval on the hardest internal suite, at a cost and latency no tested competitor approaches. The honest framing is that Sibyl's advantage is the full retrieval stack plus a state-tier data model, not raw search. The benchmarks prove the architecture decisively on structured state memory; they do not yet prove general-purpose superiority on neutral ground, and one cheap Mnemosyne rerun plus a few disclosure lines would make the whole series publication-proof.