Comparison | Mnemosyne
Sibyl vs Mnemosyne: the most fragile methodological result
Mnemosyne is tested in FTS-only mode, which makes the score weak but calls for a hybrid rerun before any hard public claim.
Full test note
This is the page where storytelling needs restraint. The number looks brutal, but the report also says vectors were disabled. The honest story is not victory-lap content. It is a warning label plus a next rerun.
Read the full test pageVerdict
Mnemosyne passes 5/350 in retrieval and 55/350 after Sonnet, but this result should be read carefully because vectors are disabled.
The run uses `fts_weight=1.0`, `vec_weight=0.0`, extraction disabled and top_k=8. Ingestion takes 22,343 seconds, more than 6 hours.
Comparison table
| Metric | Sibyl | Mnemosyne |
|---|---|---|
| Retrieval | 350/350 | 5/350 |
| Sonnet answers | 344/350 | 55/350 |
| Average context | 228 tokens | 1,662 tokens |
| Ingestion | 48 s | 22,343 s |
| Estimated answer cost | $0.64 | $2.78 |
Methodological caveat
This is the result I would present with the most caution. The report recommends a 50-company rerun in hybrid FTS/vector mode before any strong public claim.
Why not the best possible configuration?
- Unlike Mem0 and Hindsight, enabling vectors may cost little in dollars.
- The real observed barrier is ingestion time: more than 6 hours on 500 companies in the FTS-only run.
- The defensible conclusion is therefore the ingestion cost gap, not the 1.4% as a definitive ranking.
Rerun the test
python scripts/run_mnemosyne_365d_500c_category_baseline.py