Comparisons
These pages are the readable layer between the headline numbers and the raw files. Each comparison explains the setup, the tested mode, the strongest signal and the reason the result should not be oversold.
Honcho is the cleanest head-to-head. Mem0 and Hindsight are cost stories. Mnemosyne needs a hybrid rerun.
retrieval
Sonnet answer-only
Sibyl side, 42k corpus
Honcho, Mem0, Hindsight, Mnemosyne
Read order
The point is not to shout a universal leaderboard. The point is to show where structured memory changes retrieval quality, prompt cost, ingestion time and benchmark fairness.
Honcho
Sibyl passes more questions with 4.5x smaller average context and an estimated Sonnet cost about 3.5x lower.
The current-state tier favors Sibyl. Honcho is append-only in this test, and the runner uses `sleep(3)`, which may underestimate asynchronous processing.
Read comparisonMem0
Mem0 retrieves 92/350 and reaches 105/350 after Sonnet, with 1,720 average context tokens and about 92 minutes of ingestion.
Mem0 is not tested with `infer=True`. That is an explicit choice: enabling LLM extraction on this corpus would introduce major cost and ingestion time into the benchmark.
Read comparisonHindsight
Hindsight passes 152/350 in retrieval and answers, with 11,892 average context tokens and 98 minutes of ingestion.
Hindsight is configured without an LLM provider or consolidation. The comparison measures the cost and quality of the tested local mode, not an optimized cloud configuration.
Read comparisonMnemosyne
Mnemosyne passes 5/350 in retrieval and 55/350 after Sonnet, but this result should be read carefully because vectors are disabled.
This is the result I would present with the most caution. The report recommends a 50-company rerun in hybrid FTS/vector mode before any strong public claim.
Read comparison