Competitor test
Sibyl vs Mem0: the setting that turns memory into a bill
The Mem0 test is not a dunk. It is a cost question: what happens when a memory plugin needs LLM extraction before it becomes competitive?
Thesis
The Mem0 story is really about cost, not a dunk. The local baseline I tested is real and reproducible, and limited on purpose. A stronger Mem0 mode exists, but the reports put a price tag on it: extraction stops being a footnote once the corpus hits 41M tokens.
What I pulled from the reports
- The Mem0 retrieval report states the setup explicitly: open-source local baseline, Qdrant local, fastembed `BAAI/bge-small-en-v1.5`, sparse BM25 true, `infer=False`, 191,000 stored memories.
- The same report shows 92/350 retrieval, 1,720.41 average context tokens, 5,512.309 seconds of ingestion and 902.035 seconds of retrieval.
- The Mem0 answer-only report shows 105/350 answers, 853,538 prompt tokens, $2.756979 estimated cost and only 13 extra passes over retrieval.
- The methodology review estimates that `infer=True` on the corpus means roughly 41M tokens through an extraction LLM, around $50 to $100 with a cheaper model or around $445 with Sonnet, plus 8 to 25 hours of ingestion.
Benchmark signal
Plugin settings
Scale365 benchmark corpus
local vector store
local embedding baseline
sparse retrieval enabled
LLM extraction disabled
estimated extraction input size
What I am comparing
The Mem0 test starts like a classic plugin benchmark and quickly becomes an economics page. I put 191,000 structured memories through a local Mem0 setup, then asked the same Scale365 questions.
The immediate objection is fair: Mem0 is not running with `infer=True`. That is exactly why the page has to exist. The choice is not cosmetic. It changes the ingestion bill, the time budget and the meaning of the result.
The Mem0 settings
The report is very explicit: Qdrant local, fastembed, sparse BM25 enabled, raw structured records, no LLM extraction. In other words, this is a local Mem0 baseline, not the most intelligent Mem0 path.
That would be a cheap trick if I hid it. I do the opposite: I make it the story. If the stronger setting requires an extraction pass over the whole corpus, then the cost of that pass belongs in the benchmark.
- Qdrant runs locally.
- fastembed provides the embedding baseline.
- BM25 sparse retrieval is enabled.
- `infer=False` keeps LLM extraction off.
- Sonnet is still used for answer-only evaluation after retrieval.
Why I did not run infer=True
The methodology review estimates roughly 41 million input tokens for extraction on this corpus. With a cheaper model, that is around 50 to 100 dollars. With Sonnet, around 445 dollars. The same section flags 8 to 25 hours of ingestion for some configurations.
That is the actual comparison I want readers to see. Sibyl pays with structure and a retrieval planner. Mem0 can pay with extraction. Both are legitimate product choices. Only one of them keeps this benchmark local, fast and cheap.
Where Mem0 actually struggled
The category breakdown is the story. Mem0 does well enough on milestones at 42/50. But marker is 0/50, negative traps are 0/50 at retrieval, and context_stat is 1/50. Sonnet only raises the final score from 92 to 105.
That means the model cannot simply rescue weak retrieval. It can refuse some negative traps and phrase some answers better, but it cannot reliably invent missing structured state. The context has to be right before the model sees it.
Fairness note
This page should never be presented as Mem0 in its best possible configuration. It is a local, cost-controlled baseline. The useful claim is about the tradeoff: if the stronger mode requires a large LLM extraction pass, the benchmark has to include that cost.
Rerun the test
python scripts/run_mem0_365d_500c_category_baseline.py