Competitor test
Sibyl vs Mnemosyne: the result that needs a warning label
Mnemosyne is tested in FTS-only mode, so this page explains the result, the missing vector path and the rerun needed before any strong claim.
Thesis
This is the page where storytelling needs restraint. The number looks brutal, but the report also says vectors were disabled. The honest story is not victory-lap content. It is a warning label plus a next rerun.
What I pulled from the reports
- The Mnemosyne retrieval report says the tested mode is `mnemosyne_public_remember_no_extract`, extraction disabled, recall top_k=8, `fts_weight=1.0`, `vec_weight=0.0` and `importance_weight=0.0`.
- The same report shows 5/350 retrieval, 7.95 average context rows, 1,662.09 average context tokens, 22,342.618 seconds of ingestion and 8.709 seconds of retrieval.
- The answer-only report shows 55/350 answers, but negative_trap alone accounts for 50/50. Positive factual categories stay near zero.
- The methodology review calls Mnemosyne the one objection not protected by economics, because enabling vectors could be cheap while re-ingestion costs time. It recommends a 50-company hybrid rerun.
Benchmark signal
Plugin settings
Scale365 benchmark corpus
full-text search baseline
text search fully weighted
vectors disabled
no extraction layer
retrieval limit
What I am comparing
This test begins with the caveat, not the score. Mnemosyne is run in FTS-only mode on the Scale365 corpus. The report says vectors are disabled.
That means I cannot write this as a knockout article. I can write it as a test diary: here is what happened, here is why it matters, and here is why the next honest step is a hybrid rerun.
The Mnemosyne settings
The exact settings are the story: extraction disabled, top_k=8, `fts_weight=1.0`, `vec_weight=0.0`, `importance_weight=0.0`. This is full-text retrieval, not hybrid retrieval.
The run still matters because ingestion takes 22,342.618 seconds. That is more than 6 hours before any debate about whether vectors would improve quality.
- FTS-only search is active.
- Vector retrieval is disabled.
- Extraction is disabled.
- The observed ingestion time is the clearest signal.
Why I do not treat the score as a final ranking
The retrieval score is 5/350. The answer-only score is 55/350. That second number can mislead too, because the answer report shows negative_trap alone jumps to 50/50 while positive factual categories stay near zero.
So the serious position is narrower: the tested setup is not competitive, ingestion is slow, and the retrieval claim needs a 50-company hybrid FTS/vector rerun before I would post a strong public ranking.
What the result means
If I wanted X drama, I could post the 5/350 and pretend the story is done. That would get attention and weaken the work.
The useful takeaway is sharper: the benchmark already found a real issue, ingestion time. The retrieval score is a red flag, but not a finished verdict until hybrid mode is tested.
Fairness note
This is the most fragile competitor result. The page exists because hiding fragile results is bad testing, but overselling them is also bad testing. The public claim should focus on the observed ingestion cost and the need for a hybrid rerun.
Rerun the test
python scripts/run_mnemosyne_365d_500c_category_baseline.py