Evaluation

RememberStack is design-first and gate-heavy. Evaluation is not a marketing scoreboard first — it is how regressions in truth, retrieval honesty, and agent curriculum are caught.

TL;DR for agents and operators

Public docs state methodology posture, not a live SLA score. Published numbers require pinned dataset, protocol version, and revision.

What “good” means here

1. Correctness over recall theater — projections may lag; they must not serve dead facts as current.
2. Grain honesty — claims never answer “is it true now?”
3. Agent curriculum — a cold harness with only SKILL.md plans correctly.
4. Replayability — decisions, costs, and traces are ledgered where the product requires.
5. Comparable benchmarks — only under a fixed protocol fingerprint.

Design-conformance evals

Acceptance checks in the repository judge the code against the architecture the product documents — not against ad-hoc prompts.

Use these when changing spine, retrieval, or lifecycle behavior: failing a design eval is a product regression even if a demo still “looks fine.”

Cold-agent skill check

Input: rendered deployment SKILL.md + a task.
No: design corpus, repository context, or hidden cheats.

Must demonstrate:

See Mounts for the skill surface.

Golden sets and regression canaries

Examples that ship with the system (not exhaustive):

Treat measured spike numbers as capacity evidence, not promises.

LoCoMo (full-system protocol)

Repository adapter: benchmarks/locomo/.

Posture

• Unshipped as a vendored dataset — you supply pinned locomo10.json after confirming CC BY-NC 4.0 terms.
• Operator-invoked real provider runs; not silent CI spend.
• Current protocol is RS-LoCoMo-Full-v36, versioned and fingerprinted in the adapter README.
• Answer agent can use the complete shipped read plane: assured operations, direct primitives, open SQL/live graph helpers and saved queries, P1, and the ordinary published P3 mount. It has no private database/object-store shortcut.
• Readiness is machine-verified before answer; live graph health and P3 publication are separate capabilities.
• Costs, tool calls, envelopes, and failures are checkpointed.
• Primary metric: judge accuracy; secondary: official LoCoMo F1; diagnostic: session-grain recall from envelope evidence only (not turn Recall@k).

Development retrieval-access ablations

The D124 answer-only runner can reuse one processed Full-v36 sample across four OSS-only access profiles: native Codex with a locally copied P3 directory, native Codex with P3 plus the read-only RememberStack MCP server, an OpenRouter Luna agent with only that MCP surface, and the same provider-neutral loop with three bounded P3 list/search/read tools. It does not require UMC, RememberFS or FUSE and it never changes the source run.

Treat only the within-runtime pairs as causal comparisons. Native Codex uses an instruction-and-audit policy rather than a hard security sandbox, and its score is not comparable until an operator has reviewed its action log and marked it clean. Full commands and recovery rules are in the adapter README.

Safe first command (no API spend)

uv sync --extra benchmark
uv run --extra benchmark python -m benchmarks.locomo prepare \
  --dataset /absolute/path/locomo10.json \
  --tier smoke \
  --protocol full-v36 \
  --output .benchmark-runs/locomo-smoke

Do not run remote stages until you have read the adapter README and accepted cost/isolation gates.

Full-v30 also prevents an identity-only lookup from becoming a terminal Unknown: the answer harness requires one ordinary content-bearing testimony, fact, context, primitive, row-returning query, or P3 search/read attempt first. It uses the existing call and cost budgets and records unknown_guard_retries.

Honesty about scores

• Protocol version, model IDs, surface hashes, and dataset pin must match to compare runs.
• A weak reader/harness can tank accuracy with a healthy store — diagnose retrieval vs reader separately.
• Optional Langfuse tracing for answer/judge requires the observability extra and all three Langfuse env vars (Configuration).
• Sharding across hosts is supported for large publication runs; merge via documented summarize flags.

How agents should use this page

What evaluation is not

• Not a substitute for honest product claims on the managed cloud site
• Not automatic continuous LoCoMo spend on every PR
• Not permission to game judges with protocol-specific hacks
• Not a finished public scoreboard until methodology and results are published together

Progressive disclosure

Next

• Troubleshooting
• Configuration
• Project status