Control test
Sibyl Scale365: the control run before the fight starts
The source-backed control story: 209k write calls, 191k final records, 350 retrieval questions, 344 Sonnet answers and one very important caveat.
Thesis
I would rather not open with a leaderboard. The honest starting point is the control run: a year of synthetic business life, 500 companies, 1,500 stakeholders, 209,000 write calls, and the one question I keep coming back to. Can an agent retrieve current operational state without dumping the whole past into the prompt?
What I pulled from the reports
- The Sibyl Scale365 retrieval report documents 209,000 write calls, 191,000 final records, 350/350 retrieval, 2.10 average context rows, 227.76 average context tokens and 47.604 seconds of ingestion.
- The executive report says the answer-only layer reached 344/350 with Claude Sonnet 4.6, and that the 6 misses happened after successful retrieval.
- The methodology review adds the important caveat: Sibyl reaches 100% on the hardest internal benchmarks through the full retrieval stack. Raw natural-language search without that stack is near zero.
- The same executive report points to a more production-shaped Phase A 365-day run: 600/600 extraction, 140/140 retrieval and 138/140 answers, with 2 LLM omissions.
Benchmark signal
Scale365 corpus
category questions
Sonnet answer-only
average retrieved context
local structured memory
estimated Sonnet answer cost
Plugin settings
365 days, 500 companies
category retrieval and answer checks
answer-only evaluation
entities, category state, current facts
the reference before competitor pages
What I am comparing
I want the reader to picture the dataset before the score lands. Forget the cute todo-list memory demos. This is 500 companies moving through a simulated year: statuses, milestones, roles, markers, temporal topics, context stats and negative traps.
The control question is practical. After a year of activity, can the memory system find the exact current state for the right company without sending Sonnet a messy archive of everything that ever happened?
- The corpus is large enough to make naive recall expensive.
- The questions ask for concrete operational facts, not general semantic similarity.
- The answer model is used after retrieval, so a retrieval miss still poisons the final answer.
- Negative traps matter because a good memory must know when not to invent.
The catch: the product is the stack
The methodology review has one sentence that should stay near every public claim: the product is the stack, not bare search. Sibyl does not win this run because raw search magically understands business memory.
It wins because the app-side retrieval stack routes intent, uses exact entity and state fallback, and repairs the kinds of misses that destroy a long-memory benchmark. That is a strength, but it is also the disclosure.
- Full stack: 350/350 retrieval on Scale365.
- Bare natural-language search: documented as near zero in related internal tests.
- Real claim: structured-state memory plus the retrieval planner is efficient.
- Bad claim: every search layer is beaten on every possible task.
The result in human terms
The headline is clean: Sibyl retrieves 350/350 questions and answers 344/350 after Sonnet, while keeping average context around 228 tokens.
The more interesting story is what that does to the prompt. Retrieving two tight rows is not only cheaper on tokens; it also leaves the model less room to wander off. For agent work, a smaller context is itself a product feature.
What would make the claim stronger
The report is honest about what still has to happen before any big public claim. A neutral benchmark like LongMemEval or LoCoMo would help, and so would a third-party harness and a competitor-tuned rerun.
So I would frame this as the control run and the architectural signal. It proves that Sibyl can be extremely efficient on structured operational memory. It does not prove general-purpose superiority on neutral ground.
Fairness note
This is a home-system baseline, not a neutral third-party benchmark. I use it to make the other tests intelligible. The useful claim is that the structured-state stack is unusually efficient on this workload, not that Sibyl wins every memory task forever.
Rerun the test
python scripts/run_sibyl_365d_500c_category_benchmark.py