RememberStackremember.dev/docs

Benchmarks and how we measure

A memory benchmark feeds a system a long history, then asks questions whose answers are somewhere in it. The score says how often the system's answers are right. The number depends on much more than the memory: the model that reads the retrieved context, the model that grades the answers, the prompt, the retrieval tools the reader may call, and the exact engine build. A score means something only next to all of those.

RememberStack's harness therefore fixes all of them in a named protocol, checks that the running engine matches it, and refuses to run otherwise.

The dependence runs one way that matters when you read any score, ours included: a weak answer model or a clumsy harness lowers accuracy even when the memory holds every answer. A wrong answer has two possible causes, and they need different fixes:

  • Retrieval: the evidence for the gold answer never reached the answer model.
  • Answering: the evidence was in what the model retrieved, and it answered wrong anyway, or chose a tool that could not find it.

Diagnose them separately. The run directory keeps, for every question, the gold evidence, each tool call with its result, and the answer, so you can check which of the two failed before blaming either the memory or the model.

89.94% accuracy on LoCoMo

Agents using RememberStack answered 1,385 of 1,540 questions correctly across ten long conversations and 272 sessions. The full publication run used RS-LoCoMo-Full-v45; see Recorded results.

LoCoMo

LoCoMo (Maharana et al., 2024) is a set of 10 long conversations between two people, each spread over many sessions, with questions about what was said. The pinned copy holds 272 sessions, 5,882 turns and 1,986 questions in five categories.

The harness keeps categories 1 to 4 (1,540 questions) and leaves out category 5, whose questions are adversarial and have no gold answer:

CategoryQuestionsWhat it tests
1282Multi-hop: combining several facts.
2321Temporal: when things happened.
396Open-domain and commonsense inference.
4841Single-hop: one fact.

Three fixed question sets (tiers) are committed with the harness:

TierQuestionsUse
smoke8Checking that a run works end to end.
development200Comparing changes during development.
publication1,540The full set; the only tier a published score may come from.

The dataset file is locomo10.json from the LoCoMo repository at commit 3eb6f2c585f5e1699204e3c3bdf7adc5c28cb376. The harness checks its SHA-256 (79fa87e9…a698ff4) and refuses any other file.

Retrieval depth

Prepare-time flags --retrieval-k and --retrieval-candidate-k configure the context tools used by the answer agent. Defaults are 50 confirmed results per type and 200 internal candidates per search channel. Counts apply separately to facts, claims and chunks; a combined call may return 150 primary records plus linked fact evidence. Candidate counts are not returned-result counts.

For example, prepare --retrieval-k 50 --retrieval-candidate-k 200 freezes that budget in run.json and its fingerprint. Effective arguments are enforced and saved on every context-tool call. A different budget requires a new run; changing it in an existing checkpoint fails fingerprint verification. Larger budgets have no established accuracy advantage and increase returned text and reader cost. The run directory retains generated answers and complete tool traces so they can be regraded without repeating ingestion or answering.

How a LoCoMo run works

The current Full-v47 protocol uses per-version publication markers, ready folder identities and frozen file inventories, including binary reader files and navigation. It rejects changed local bytes and requires the same evidence for resume and ablations. Global P3 version/timestamps are absent. This attestation change establishes no new score; earlier results remain tied to their recorded protocol and source revision.

The harness lives in benchmarks/locomo/ and runs as python -m benchmarks.locomo.

  1. Prepare (local, no calls). Validates the dataset, selects the tier's questions, fixes the protocol and records the repository revision in a run directory.
  2. Ingest, per conversation. Checks that the deployment runs the engine build, pipeline component versions and model bindings the protocol requires, then uploads each session as one Markdown document with a stable source_kind/source_ref.
  3. Answer, per conversation, after processing has finished. An answer model gets each question and may call retrieval tools: the four assured operations, eight lower-level lookups (resolve, lookup_relations, transcript_relation, lookup_observations, search_claims, search_chunks, adjacent_chunks, hydrate_relation), the seven SQL query tools, and three tools over the live corpus with frozen run inventories. It may make at most 8 tool calls and 9 model calls per question.
  4. Judge, per conversation. A judge model labels each answer CORRECT or not against the gold answer.
  5. Summarize (local). Scores the whole tier. A question with no answer or no judgement counts as wrong.

Remote stages run only with --execute, a clean git worktree at the prepared revision, and a confirmation flag that repeats the conversation id. Each stage takes hard ceilings on calls and on evaluator spend.

The summary reports:

  • Judge accuracy: the share of questions labelled CORRECT. This is the headline number.
  • Official F1: LoCoMo's token-overlap F1 between answer and gold answer.
  • Both per category, plus model calls, retries, tokens, evaluator cost and failure counts. Ingest cost is not in the summary; it is on the deployment's cost ledger.

Scoring retained ingestion after a serving-only upgrade

Use reuse-ingestion to prepare a new current-code run over an already processed conversation. First migrate its existing database with the normal setup command and publish/verify its current projections. Do not rerun ingestion. Then run:

uv run --extra benchmark python -m benchmarks.locomo reuse-ingestion \
  --source-run /private/original-run --sample conv-42 \
  --output /private/current-serving-run --execute

This command makes readiness and query-surface requests, but no model call and no upload. It requires identical document bytes, dataset selection, processing generations, model bindings and per-version processing rows. Changed extraction or embedding generations require a fresh ingestion instead. Current code/build, public query-surface and exact live-document guards still apply.

The new run records the current serving revision and surface fingerprint. Its retained_ingestion origin records the original processing revision/fingerprint and hashes of the original configuration and state; their exact bytes are saved under provenance/retained-ingestion/. Original files and results stay unchanged. New answer/judge state starts empty. Ingestion is refused for the derived run. Always derive from the original processing run, rather than another scoring fork. Normal summarize --run ... --run ... can combine disjoint retained forks only when their serving identities and original processing revision/fingerprint match. Its summary retains all per-sample origins; mixing ordinary and retained runs fails.

Keep the matching P3 mount root and frozen dataset available to the new run; this command does not republish or copy the mount. Use normal answer/judge or retrieval-ablation --profile full with that run. Native store backups then name the actual serving revision for restoration, while retaining the paid processing origin. A partially prepared output is inspected and removed explicitly before retrying; the command refuses a nonempty output. No automatic paid retry occurs.

The current protocol

Reader refinements have distinct saved prompt hashes and protocol fingerprints. Reports must identify that fingerprint, not only the family name. A prior-prompt run remains inspectable but cannot resume with changed instructions.

Full-v46 changes the image-description fingerprint to preserve bounded legible identity labels alongside OCR. It needs matching new visual readings; historical bundles and published results remain frozen. For reader-only retained v45 comparisons use the PR593 revision; current v46 code does not execute retired protocol names. No v46 score is claimed.

Execution uses one profile, RS-LoCoMo-Full-v46 (key full-v46). It selects Luna 6 with low reasoning for ingestion and visual descriptions, Jev (jev-latest) for adjudication, and Luna 6 with high reasoning for answering and judging. All these Luna seats prefer OpenAI Flex and permit normal OpenAI Luna as the only fallback. Standard processing can cost more; provider-returned usage determines the recorded cost. Embeddings remain Qwen through Nebius. The historical branch's semantic judge policy is restored: identifying or encompassing the reference's core fact, entity, date or event is correct. Synonyms and additional context do not invalidate that core match. Contradictions, a fundamentally different fact/entity/time and missing/Unknown answers are wrong. Dates must be semantically equivalent; no fixed date-error tolerance is introduced. Luna high still judges. Historical answers/grades retain their own identities. This is distinct from the independently reported Mem0-like sidecar below. The limits remain eight tool calls and nine model calls per question, with no hard answer length cap.

The answerer must give its best-supported answer rather than abstain with Unknown. It may use ordinary general knowledge to connect retrieved clues and qualify an indirect conclusion with likely or possibly. It must still preserve the question's entity and date, and may not invent supporting dialogue or personal details. This evaluation policy does not change the engine's typed negative results or the judge. Old runs retain their original prompts and grades; this protocol requires a new run identity. Its offline checks establish no score.

The saved run records its non-secret ingestion environment, component/model bindings, approved provider order and 300-second API transport budget. CI checks the native System One backup remains disabled (REMEMBERSTACK_TYPESAFE_BACKUP=none), so adding an optional product adapter cannot silently change benchmark providers. CI also checks all component generations against the current engine. The shard runner attests workers against the profile, and backups preserve it. Superseded protocols are read-only historical records; they cannot be selected for execution or silently used as a fallback. Changing configuration does not relabel or re-ingest an old store.

Image preparation uses the shared Mistral OCR and Luna vision converter over actual source pixels, without questions or reference answers. prepare-images plans locally by default; paid execution requires --execute, provider keys and finite cache-absolute conversion/cost ceilings. Successful independent lanes are reused. Uncertain interrupted paid attempts require inspection. The reported-cost limit stops new admissions; already admitted calls finish. OCR uses its configured page-price meter; vision records provider-returned usage.

Offline prepare --images <bundle> requires complete image coverage, copies the original pixels, readings, native provenance, binary assets and accounting into the run, and fingerprints them. The renderer attaches labeled derived readings to their exact source turns and speakers; downstream sessions are ordinary E0 Markdown, not native image-origin representations. It never silently substitutes a caption when an available image's reading is missing. Permanently inaccessible source URLs require explicit --allow-unavailable with typed failures, visible markers and coverage counts. Provider failures still stop preparation. Frozen run inputs are independent of ambient caches and are included by the ordinary GCS run archive. Changed image input requires a new run and fresh ingestion. Image restoration keeps retrieval budgets and core generations unchanged. Full-v42 additionally restores the historical branch's semantic judge template. Full-v45 extended the answer instructions without changing that judge or the models; Full-v46 inherits that reader, including PR593, and adds new desc-prompt-v2 visual readings.

The answerer checks question completeness, distinct events and participants, and adjacent dialogue. If a relevant reply lacks the question that elicited it, the answerer is instructed to read adjacent chunks even when another candidate looks plausible. It checks which source turn and speaker own an attached description and names a work's supported subject or premise when asked what it is about. These are prompt instructions within the existing tool/model-call budgets, not a reranker or an accuracy guarantee. No score is established for this protocol by its offline tests. The partial historical campaign remains a separate result.

The answer instructions preserve relevant items with their precise source labels when a short question uses informal wording. They check object kind and medium, compare unnamed preference candidates without requiring a resolved-person link, and include concrete subjects from the owning image. Answers keep brief source qualifications without invented categories or plots. These are general prompt instructions, with no accuracy guarantee or new ingestion work.

For emotional recall, the instructions include attitudes conveyed by a person's wording and distinguish the time of an attitude from an event it concerns. Person, subject and time constraints still apply; no expected state names are provided. This remains a prompt instruction, not an accuracy guarantee.

The reader's displayed tool catalog retains input schemas and tool contracts, but omits repeated output JSON Schemas. Public tool schemas, runtime validation and retrieved evidence remain complete. Full-v45 introduced this prompt-rendering change; a smaller prompt alone does not establish a score or latency improvement.

Independent Mem0-like grading of saved answers

python -m benchmarks.locomo.mem0_style_judge prepares a separate grading result without changing answers or canonical grades. It does not retrieve, ingest, or regenerate answers. The following command plans the work without model calls:

python -m benchmarks.locomo.mem0_style_judge \
  --run /data/locomo/frozen-run \
  --output /data/locomo/mem0-like-results \
  --max-calls 2000 --max-cost-usd 10

For saved retrieval-ablation answers, supply their output directory separately:

python -m benchmarks.locomo.mem0_style_judge \
  --run /data/locomo/retained-run \
  --answers /data/locomo/saved-answers \
  --output /data/locomo/mem0-style-results \
  --judge-model openai/gpt-6-luna-pro \
  --max-calls 2000 --max-cost-usd 10

This reads the existing answers and validates their execution/source identities. The saved answer set must be complete before execution. Neither source directory may overlap the grading output. A changing source snapshot stops grading.

Execution requires --execute and OpenRouter credentials. The judge defaults to Luna 6 with high reasoning; --judge-model can select Luna Pro. Routing prefers Flex and permits normal OpenAI as fallback. Its policy accepts one correct list item, dates within fourteen days, durations within fifty percent and broad paraphrases. This is a policy inspired by the pinned Mem0 source, not an exact reproduction of Mem0's published evaluation.

Optional --with-evidence supplies gold-referenced source turn text and dates to the judge and permits evidence only to increase acceptance. It does not supply original image pixels. Category-three gold answers use the text before the first semicolon, as in that upstream policy. This preprocessing is local to the sidecar; original data stays unchanged.

The output has a separate identity, per-item prompt hashes, model settings, usage, explanations and result files. Resume requires the identical frozen input snapshot. A call interrupted before its outcome was saved remains visible as in flight and must be reconciled before resuming. Received unbilled capacity rejections can resume in the same directory, reusing prior successful grades and retaining rejection attempts. Missing answers, source failures and judge failures are reported separately. The cost setting is a reported-spend stop threshold; the provider account cap is the hard monetary boundary. Keep alternate-policy scores separate from canonical scores.

BEAM

BEAM ("Beyond a Million Tokens", Tavakoli et al., ICLR 2026) tests memory over very long chat histories. Its probing questions cover abilities such as information extraction, contradiction resolution, event ordering and abstention, and each question comes with rubric items ("nuggets").

The harness in benchmarks/rs_harness_beam/ has three commands:

CommandWhat it does
answer-retrievalAnswers the probing questions of a prepared run directory against a running engine, using combined_context and SQL queries, with an OpenRouter model (default openai/gpt-5.6-luna).
score-officialThe BEAM paper's scorer: an LLM judge scores every nugget 0, 0.5 or 1; event-ordering questions also get Kendall's τ-b. Default rubrics: the committed 100K/1 fixture.
scoreA simple containment check, kept as a placeholder.

Limits of the BEAM harness today:

  • It has no ingest step. The run directory (questions.json, state.json) and the ingested conversation must be prepared by other means.
  • The answer agent contains search hints written for the committed 100K/1 fixture, so its answers on that fixture are not a fair measurement.
  • score-official prints an overall_mean that is not a BEAM metric; BEAM reports per-ability scores.

BEAM is used for internal checks only. No BEAM score is published.

Recorded results

These are all the scores recorded in the repository (plan/analysis/, design/benchmarks/, decisions.md). Each belongs to its protocol and engine revision.

DateProtocolEngine revisionTierJudge accuracyOfficial F1
2026-11-13RS-LoCoMo-Full-v45serving 1187ca0f, processing 13ce4a52publication1,385 / 1,540 (89.94%)—
2026-07-31RS-LoCoMo-Full-v5-strongextractor 07jpublication517 / 1,540 (33.6%)0.3054
before 2026-08-10RS-LoCoMo-Full-v8-strong0ef54549publication, assembled from two partial runs1,100 / 1,540 (71.43%)not recorded
2026-08-10RS-LoCoMo-Full-v11213551c7publication979 / 1,540 (63.57%)0.5417
2026-09-01RS-LoCoMo-Full-v18not recordedsmoke (conversation 26)7 / 8not recorded
2026-08-07BEAM smoke, 100K/1 fixturenot recorded1 conversationoverall_mean ≈ 0.5625 (not a BEAM metric)n/a

The v8 and v11 runs used different answer tools and are not directly comparable. The analysis of the drop between them points mainly at the answer agent choosing a weaker retrieval tool for 393 questions, rather than at worse ingestion. It is in plan/analysis/locomo_v11_score_regression_analysis.md.

The Full-v45 publication run used openai/gpt-6-luna-pro at high reasoning for answering and grading, Luna at low reasoning for ingestion and vision, and Jev for adjudication. Retrieval used k=50, candidate_k=200 per search channel and eight simultaneous questions per conversation. The saved run retains its answers, tool traces, model settings, costs and failure records for regrading.

Reproduce a LoCoMo run

You need a clone of writeitai/remember-stack with the development dependencies, a self-hosted engine you can wipe, and an OpenRouter key for the answer and judge models.

git clone https://github.com/writeitai/remember-stack.git
cd remember-stack
uv sync

Start the engine with the protocol's model bindings; see Install with Docker Compose and Models and providers. Point the harness at it:

export REMEMBER_API_URL=http://127.0.0.1:8000
export REMEMBER_API_KEY="<token>"
export REMEMBERSTACK_OPENROUTER_API_KEY=<openrouter key>

Then, stage by stage, for one conversation (conv-26) of the smoke tier:

uv run --extra benchmark python -m benchmarks.locomo prepare-images \
  --dataset /path/to/locomo10.json --tier smoke \
  --cache /private/locomo-images-cache --output /private/locomo-image-inputs \
  --max-conversions 1000 --max-cost-usd 10
 
# The command above only plans. After reviewing it, add --execute and supply
# REMEMBERSTACK_MISTRAL_OCR_API_KEY and REMEMBERSTACK_IMAGE_DESCRIPTION_API_KEY.
# Then prepare offline from the resulting immutable bundle:
uv run python -m benchmarks.locomo prepare \
  --dataset /path/to/locomo10.json --tier smoke \
  --output .benchmark-runs/smoke --protocol full-v47 \
  --images /private/locomo-image-inputs
 
uv run python -m benchmarks.locomo ingest \
  --run .benchmark-runs/smoke --sample conv-26 \
  --max-documents 100 --max-evaluator-cost-usd 5 \
  --execute --confirm-isolated-deployment conv-26
 
# Wait for each served document's p3 readiness. Mount/copy the live corpus
# read-only (see Filesystem views) and pass its directory as --p3-root below.
 
uv run python -m benchmarks.locomo answer \
  --run .benchmark-runs/smoke --sample conv-26 \
  --p3-root /path/to/p3 --max-questions 8 --max-agent-calls 72 \
  --max-evaluator-cost-usd 5 --execute
 
uv run python -m benchmarks.locomo judge \
  --run .benchmark-runs/smoke --sample conv-26 \
  --max-judge-calls 8 --max-evaluator-cost-usd 5 --execute
 
uv run python -m benchmarks.locomo summarize --run .benchmark-runs/smoke

Run ingest, answer and judge once per conversation in the tier. For the publication tier, summarize accepts several --run directories and checks that they belong to the same protocol and do not overlap.

The maintained path for full runs is the sharding script, which wipes the stack, ingests, waits for processing, answers, judges and backs up each conversation in turn:

LOCOMO_PROTOCOL=full-v46 LOCOMO_IMAGE_INPUTS=/private/publication-image-inputs \
  LOCOMO_MAX_EVALUATOR_COST_USD=60 \
  bash benchmarks/locomo/sharding/run_shard.sh conv-26 .benchmark-runs/my-run /path/to/locomo10.json

It takes a comma-separated list of conversation ids. Its limits (tier, questions, calls, cost, drain timeout) are set with LOCOMO_* environment variables at the top of the script.

For relative-time questions, the answer prompt asks for calendar dates, ranges or named periods at the precision supported by the source. The source timestamp anchors relative wording; it cannot supply an otherwise unknown event date or justify extra precision.

Compare answer prompts or models without ingestion

Use retrieval-ablation --profile full to re-answer one completely processed sample with the same live store, source-pinned 22 SDK/P3 tools, high reasoning, retrieval budget, call limits, and canonical judge. An optional UTF-8 --answer-prompt file uses the existing answer-template format placeholders. Without it, the current canonical template is used.

To compare an answering model, supply --answer-model PROVIDER/MODEL with --profile full. The exact model is frozen in the experiment fingerprint and sent to the provider; omission inherits the source model. Keep the prompt unchanged for a model-only comparison. The judge, retrieval settings and processed store stay fixed. Model changes require a new output directory, and no ingestion or image conversion is repeated.

--answer-workers 4 answers up to four independent questions concurrently (default one, allowed range 1–16). Each question retains its own trace and completed answers are checkpointed by one writer. Shared admission keeps model-call and reported-spend accounting consistent. After a technical error, new calls stop while in-flight calls finish and are saved. Previously admitted calls can complete after their combined cost reaches the spending threshold; the provider/account allowance is the hard dollar limit. Judging stays serial.

For a model served by another provider, supply --answer-provider HOST (repeat for an ordered allowlist). This changes only the answer provider; the judge keeps its canonical OpenAI policy. Model, provider policy and worker count are fingerprinted, so changing them requires a fresh output directory.

Use --judge-model MODEL for an explicit full-profile grader comparison. It keeps the judge prompt, schema and HIGH reasoning effort; the effective model is frozen into the experiment and controls the actual grader request. Default judging remains unchanged. Repeat --item ITEM_ID to select a development subset from the sample. IDs must be distinct and belong to that sample; their canonical order is fingerprinted. Omit --item to test the entire sample. The reader still searches the complete ingested source. A passing subset is not a full-conversation score, and changing the selection rejects resume.

For a timeout-only replay, --api-timeout-seconds 600 explicitly gives each full-profile SDK request a longer timeout. The override is recorded in the configuration, fingerprint and summary; changing it requires a new output directory. Omission keeps the source run's saved timeout (currently300 seconds) and existing experiment identities. This does not extend the server's separate PostgreSQL retrieval budget: configure that deployment through REMEMBERSTACK_SELFHOST_RETRIEVAL_DATABASE_BUDGET_S (default120 seconds). Record server tuning alongside the replay, retain the original results, and select timeout-affected items without using their judge labels. No ingestion is repeated.

uv run python -m benchmarks.locomo retrieval-ablation \
  --run .benchmark-runs/development --sample conv-47 --profile full \
  --p3-root /path/to/p3 --answer-prompt /path/to/answer-template.txt \
  --output .benchmark-runs/prompt-trial-1 --max-questions 20 \
  --max-agent-calls 180 --max-judge-calls 20 \
  --max-evaluator-cost-usd 2 --execute

The output saves the frozen template, its hash, source and runner revisions, complete new answers/tool traces/grades, usage, and summary. It leaves the source run untouched. Identical commands resume missing work; changed prompts or source identity or answer models require a new output directory. Live readiness, exact ingests, models, query surface and P3 must still match. The source judge schema must equal the current judge. Judge wording must also match unless the explicit --use-current-judge-prompt option selects current canonical wording. That choice and effective prompt hash enter the new immutable identity; changing it rejects resume. Source metadata, golds and old grades remain untouched, and all other compatibility checks still apply. No re-ingestion is needed for this evaluation-only choice. These are development trials; retain every trial and do not report a selected trial as held-out evaluation. Archive the output alongside the processed-store backups.

The current answer prompt preserves defining criteria in constraint deductions, compares credible references across sessions before narrowing to one candidate, and retains relevant restrictions with their source qualifiers without adding unsupported denials. If resolved-entity filters leave required details missing, it searches claims and chunks without that semantic filter and verifies attribution from the source. It reads returned chunk text as well as extracted claims, because claims can omit items mentioned only in a reply or attachment. List answers keep strongly implied items with a qualifier such as "likely" instead of dropping them. Repeated episodes are kept apart by when they happened. An answer names a different participant only when the event in the question belongs uniquely to them. Deployment, visibility and time boundaries remain in force. These rules do not change ingestion, ranking or the primary judge. The current full-profile comparison runner can read a source with an older prompt-only protocol name when its entire non-prompt contract still matches. It first verifies models, provider policy, stage generations, schemas and loop budgets, then checks current-contract readiness against the recorded source build, exact ingests, query surface and P3. A source with changed ingestion settings is rejected. This keeps one executable protocol; source configuration and fingerprints stay unchanged.