Competitor test
Sibyl vs Hindsight: recall works until context eats the room
Hindsight finds meaningful memories in local mode, but the average context climbs so high that Sonnet becomes the expensive part of the test.
Thesis
Hindsight is the subtle page. It does not collapse like a bad retrieval toy. It finds meaningful memories. The problem is that it often sends so much context that Sonnet becomes the expensive part of the memory system.
What I pulled from the reports
- The Hindsight retrieval report says the tested mode is an embedded local daemon with LLM provider `none`, observations and consolidation disabled, Hindsight `recall`, max_tokens=4000, budget low and include_chunks=False.
- The same report shows 152/350 retrieval, 43.40 average context rows, 11,891.85 average context tokens, 5,869.615 seconds of ingestion and 467.616 seconds of retrieval.
- The category table shows the nuance: milestone, marker and temporal_topic are each 50/50, while status is 1/50, role is 1/50, context_stat is 0/50 and negative_trap is 0/50.
- The answer-only report records 6,149,418 prompt tokens and $18.682209 estimated cost, with no answer-score improvement over retrieval.
Benchmark signal
Plugin settings
Scale365 benchmark corpus
tested local service mode
no provider-backed consolidation
local recall baseline
include_chunks=false
What I am comparing
The Hindsight page is not a simple win-loss story. It is about a memory system that can surface real events and still lose the product argument.
The local daemon retrieves useful material on milestones, markers and temporal topics. The trouble begins when the question asks for current status, roles, context stats or negative traps. Then the model receives a lot of surrounding history but not the exact structured answer.
The Hindsight settings
The tested setup is Hindsight local daemon, provider `none`, consolidation disabled, recall max_tokens at 4000, budget low and include_chunks false.
That is a very important sentence. Hindsight is not getting a provider-backed consolidation layer. The page should therefore be read as a local-mode measurement, not as a verdict on every Hindsight deployment.
Why I did not enable provider-backed consolidation
Turning on a provider would move this test into the same cost argument as Mem0. Maybe quality improves. But then the benchmark has to charge the system for the preprocessing intelligence it consumes.
I am not saying provider-backed consolidation is useless. I am saying the price and delay become part of the product. A memory system cannot ask the evaluator to ignore the meter.
The strange failure mode
The answer-only report shows the weird part clearly. Hindsight stays at 152/350 after Sonnet. The model sees 6.15M prompt tokens across the run, but the answer score does not move.
One failure example says the company reached a support-escalation checkpoint on day 365. That is true context, but the expected answer asks for final status `active`. This is the trap: relevant context is not the same thing as the exact operational state.
The readable takeaway is simple: recall can work and still be too expensive, or too loose, for agent memory. Context is not just data, it is the room the model has left to think in.
Fairness note
This is a local Hindsight test without an LLM provider or consolidation. It is useful for measuring the tested mode, especially context cost. It should not be sold as a final judgment on a fully optimized Hindsight setup.
Rerun the test
python scripts/run_hindsight_365d_500c_category_baseline.py