Independent agent benchmarks

I run reproducible benchmarks on AI agent systems.

I measure what agents keep and retrieve, what they hand to the prompt, how much it costs, and what breaks once MCP enters the execution path. I am an independent tester with no affiliation to the systems I measure. Memory is where I started, comparing options like Honcho, Mem0, Hindsight, Mnemosyne and Sibyl, and the results, caveats and rerun files are all below.

Systems compared5Honcho, Mem0, Hindsight, Mnemosyne and Sibyl

Retrieval questions350Same question set asked to every system

Test window365 daysSynthetic business year of memory writes

Public artifacts75Reports, scripts and raw run files, all rerunnable

No sponsor, no affiliation with any system tested. The results live further down, next to the settings, caveats and rerun files that produced them.

Read my test notes Open the archive Methodology

Tested stack

Memory is tested across agents, plugins and MCP surfaces.

Benchmark proof

The claim is backed by a corpus, competitors and security runs.

Final records191kScale365 memory corpus

Write calls209kSynthetic business year

Competitors4Honcho, Mem0, Hindsight, Mnemosyne

Security baseline31 runs4 confirmed symlink issues

Narrative tests

Start with the story, then open the files.

The main pages explain what is being compared, why memory, MCP and plugin settings matter, and where the caveats are. The archive underneath keeps the raw reports and runners close to the claim.

Control test

Sibyl Scale365: the control run before the fight starts

The source-backed control story: 209k write calls, 191k final records, 350 retrieval questions, 344 Sonnet answers and one very important caveat.

Read the control run

HonchoSibyl vs Honcho: the clean test with one async scar Mem0Sibyl vs Mem0: the setting that turns memory into a bill HindsightSibyl vs Hindsight: recall works until context eats the room MnemosyneSibyl vs Mnemosyne: the result that needs a warning label

Scale365 snapshot

The chart is readable, the caveat stays visible.

Each system is tested in a documented mode: some in a full structured-state stack, others in reproducible local modes, and every page explains what a more expensive or smarter configuration would change.

Read the methodology note

Retrieval benchmark

Questions passed

350Scale365

Sibyl Scale365 retrieval350/350

Hindsight retrieval152/350

Mem0 retrieval92/350

Mnemosyne retrieval5/350

Test archive

75 public entries across benchmarks, security, migration and rerun files.

The old dashboard is no longer the only place to find the work. The archive lists the reports, scripts and raw artifact index behind the public pages.

Full methodology reviewClaude Code review of the competitive claims, caveats, competitor settings and public wording.Reviewmd Claude Code review transcriptEnglish transcript used to challenge the benchmark story before turning it into public pages.Transcriptmd Executive benchmark reportHigh-level benchmark summary for Sibyl memory, Scale365 and the answer-only layer.Reportmd Golden suiteRegression map for future Sibyl builds: core, dependent and paid API test groups.Suitemd Sibyl Scale365 retrieval500 companies, 365 days, 191k final records and 350 category retrieval questions.350/350md Sibyl Scale365 answer-onlyClaude Sonnet answer layer over the Sibyl Scale365 retrieved context.344/350md Token economy mega runPrompt-token reduction test showing the cost difference between top-k memory and full context.Passedmd Three-arm demoA/B style demo comparing planner, Sibyl and full-context answers for public positioning.Demomd

View full archive

Long-form writing

The context before the claim.

Why I test agent memoryThe starting point for this lab: Hermes, MCP, Codex, Claude Code, then the conviction that memory is becoming a product layer.How I use MCP in the Codex and Claude Code harnessThe role of MCP in the tests: write, reread, attack, isolate, migrate, then verify artifacts with Codex and Claude Code.Why some competitors are not tested in max modeThe methodological rationale: LLM cost, ingestion time, local configuration, and the difference between architectural efficiency and a general ranking.

MCP and Hermes security

Agent tooling is also an attack surface.

MCP isolation baseline: 31 strict runs, 4 confirmed symlink failures25 PASS, 6 strict FAIL. Four failures are confirmed symlink/cross-profile isolation issues; two are harness exception failures.Hermes retests: prompt fence improved, symlink and learning issues remain openThe 0.4.17 retest is mixed: selected prompt-boundary escapes are fixed, while symlink isolation, migrated secret prefetch and learning-reference cases still need patch confirmation.Code quality and security audit: what is solid, what still needs work1 high, 2 medium, 1 low. The code is considered well hardened for such a young project, but several points still need fixes.

Research paths

Follow the threads behind the benchmarks.

Persistent memory for Hermes agentsHermes AI agents, long-running context, retrieval quality and reproducible benchmark evidence.MCP memory harnessCodex, Claude Code, MCP security, migrations and rerunnable benchmark artifacts.Memory plugins for AI coding agentsLocal retrieval, structured state, prompt context and competitor memory-plugin tradeoffs.