# Sibyl Golden Suite

This document closes the current Sibyl plugin test phase and defines the short regression suite to run when Sibyl ships a new build.

## Decision

The broad exploration phase is complete for the current plugin surface.

The remaining useful work is regression mode:

- rebuild the sandbox with the new Sibyl wheels;
- run the golden suite;
- compare fixed versus still failing;
- keep raw outputs immutable;
- send a short fixed or still failing report.

## Golden Commands

Run these first. They are the high-value offline tests.

```bash
python3 scripts/run_sibyl_golden_suite.py --execute --group core
```

To see the commands without running them:

```bash
python3 scripts/run_sibyl_golden_suite.py
```

Optional dependent checks need prior local DB artifacts under `/tmp` or paid API credentials:

```bash
python3 scripts/run_sibyl_golden_suite.py --execute --group dependent
python3 scripts/run_sibyl_golden_suite.py --execute --group paid-api
```

## Closure Matrix

| Area | Status | Last artifact | Golden command | Expected fixed behavior |
|---|---|---|---|---|
| MCP validation and storage repro pack | Open regressions, ready for retest | `runs/repro-pack-2026-06-01.md` | `scripts/run_mcp_security_repro_pack.py` | Sanitized errors, no raw validation or backend leaks, no storage path escape. |
| Migration symlink secret chain | Open regressions, ready for retest | `patches/sibyl-memory-migration-round11-symlink-secret-disclosure.md` | `scripts/run_sibyl_security_round11_migration_symlink_suite.py` | Migration refuses or safely skips symlinked project and memory paths. |
| Hermes prefetch boundary escape | Open regressions, ready for retest | `patches/sibyl-memory-hermes-round13-prefetch-boundary-escape.md` | `scripts/run_hermes_security_round13_prefetch_boundary_suite.py` | Prefetch renders exactly one begin marker and one end marker regardless of stored content. |
| Hermes self-learning authority | Open product-security gap, ready for retest | `patches/sibyl-memory-hermes-round14-learning-authority.md` | `scripts/run_hermes_security_round14_learning_authority_suite.py` | Untrusted or sensitive journal material is not promoted into trusted learned references without guardrails. |
| Hermes profile diagnostics | Mixed, one profile-boundary finding | `patches/sibyl-memory-hermes-round15-profile-diagnostics.md` | `scripts/run_hermes_security_round15_profile_diagnostics_suite.py` | `active_profile` symlink is refused or ignored, learned references stay profile-scoped. |
| Product memory update behavior | Storage passes, natural retrieval gaps remain | `patches/sibyl-memory-product-memory-update-suite-results.md` | `scripts/run_mcp_memory_product_update_suite.py` | Current-state retrieval handles corrections, old-new conflicts, ownership changes, priority drift, and paraphrases. |
| UX and first-run memory questions | UX patches needed | `runs/run-2026-06-04-sibyl-ux-qol-round2.md` | `scripts/run_sibyl_ux_qol_round2.py` | CLI explains setup/init flow; vague memory questions return clarification or useful context instead of zero results. |
| Install and uninstall profile boundary | Open local boundary finding | `patches/sibyl-memory-hermes-round10-local-boundary-critical-candidates.md` | covered by round 10 scripts and run artifacts | Installer and uninstaller reject symlinked `plugins` parents and never modify another profile. |
| Scale and retention | Passed, no need to rerun every patch | `runs/memory-scale10x-paid-mega-200c-180d-2026-06-03.md` | optional only | 42k writes, 200 companies, 600 stakeholders, 180 days still pass exact recall and marker retrieval. |
| Token economy | Passed, marketable proof | `runs/run-2026-06-04-mcp-run311-token-economy-mega-50q.md` | optional only | Sibyl top-k context keeps large prompt-token reduction versus full company context. |
| Demo and A/B evidence | Passed enough for positioning | `runs/run-2026-06-04-sibyl-hackathon-three-arm-demo.md` | optional only | Planner plus Sibyl matches full-context answers at materially lower token cost. |
| Obsidian lab | Planned isolated product workflow | `docs/sibyl-obsidian-lab-test.md` | manual | Fresh agent recalls canonical lab note and does not confuse lab DB with personal vault. |

## What Not To Reopen

Do not rerun broad exploration unless a new feature changes the attack surface.

Avoid spending time on:

- another generic symlink variant without a new trust boundary;
- another volume test below the existing 42k-write benchmark;
- another raw keyword retrieval benchmark unless it changes the retrieval strategy;
- paid LLM comparisons unless the model, retrieval planner, or context strategy changed.

## Retest Protocol

1. Record the tested package versions.
2. Run the golden suite.
3. Keep new `.raw_result.json`, `.assessment.json`, `.log`, and `.md` files unchanged.
4. Classify each item as `fixed`, `still failing`, `changed behavior`, or `blocked`.
5. Send only a short delta report.

## Current Read

The plugin stores and retains memory well at scale. The strongest remaining product issue is assistant-facing natural retrieval:

- vague current-state questions;
- paraphrases that do not share exact terms;
- ranking current state above historical journal rows;
- clarification when the query is ambiguous.

Security retesting should wait for a new Sibyl build unless the team asks for a specific new threat model.