Operating the pipeline
Every piece of pipeline work is a row in PostgreSQL: which version, which stage, how many attempts, and what went wrong last. Workers claim these rows, retry failures, and set aside work that keeps failing. This page shows how to see that state and act on it.
Running operator commands
The operator commands are part of the remember CLI, under
remember ops. They talk to PostgreSQL directly rather than to the API, so
they run inside the engine image, where the database settings already are.
Run them with docker compose exec api remember ops …; the image enables
them, so no extra setting is needed.
Read the deployment id from .env once per shell:
DEPLOYMENT_ID=$(grep '^REMEMBERSTACK_SELFHOST_DEPLOYMENT_ID=' .env | cut -d= -f2)Then run each command in the api container:
docker compose exec -T api \
remember ops inspect --deployment "$DEPLOYMENT_ID"Every remember ops command prints one JSON document on standard output.
Pipe it through python3 -m json.tool to read it.
| Command | What it does |
|---|---|
remember ops inspect --deployment ID | Report pipeline, dead-letter, projection and consistency state |
remember ops replay PROCESSING_ID --deployment ID | Give one dead-lettered item more attempts |
remember ops resume-no-route --deployment ID | Release conversions parked for lack of a route |
remember ops graph-catalog ensure | Check and repair the PostgreSQL graph views and functions |
remember ops cost-export --deployment ID | Print one page of the cost ledger (Observability) |
Inspect the pipeline
docker compose exec -T api \
remember ops inspect --deployment "$DEPLOYMENT_ID" | python3 -m json.toolThe report has five parts:
| Field | Contents |
|---|---|
routes | For each stage and lane, how many items are in each status (pending, running, succeeded, failed while waiting to retry, dead_letter, skipped), split by defer_reason: no_route, admission or budget for parked work, scheduled for work due later, retry_backoff for a failure waiting to retry, null otherwise. A large pending count with no reason on one stage shows where the pipeline is behind. |
dead_letters | Items that ran out of attempts: a total, groups by stage and error class, and individual items with their processing_id, attempts and last_error. |
poison_targets | Items that dead-lettered under two or more component versions: an upgrade did not fix them. |
latest_projections | Historical projection metadata; live corpus readiness is checked per requested document through /readiness. |
currency | A consistency check between each claim's cached "current" flag and the ledger it is derived from. mismatch_total should be 0. |
Lists in the report are capped at REMEMBERSTACK_OPERATIONAL_SAMPLE_LIMIT
entries (default 20).
Retries and dead letters
When a stage fails on an item:
- A retryable failure, such as a provider timeout or a
5xx, is retried after a back-off: 2 seconds after the first failure, doubling each time, never more than 60 seconds (REMEMBERSTACK_WORK_RETRY_BACKOFF_BASE_S,REMEMBERSTACK_WORK_RETRY_BACKOFF_MAX_S). - An item gets 3 attempts in total: the first try and two retries.
- A non-retryable failure, such as a file that cannot be converted, goes straight to the dead letters.
- An item that uses its last attempt goes to the dead letters.
- A worker that dies while running an item (killed, out of memory,
restarted) stops renewing its hold on the item. Once the hold runs out
(
REMEMBERSTACK_WORK_LEASE_S, 60 seconds by default), another worker puts the item back in the queue, or sends it to the dead letters if that was its last attempt. Thelast_errorthen starts withlease expired.
HTTP 429 answers from OpenRouter are handled inside the provider call and do not use attempts (Models and providers).
A dead-lettered item stays where it is. Its version is not ready, and the work after it in the pipeline waits. Nothing retries it on its own.
Replay a dead letter
Find the processing_id in the dead_letters.items of the inspect report,
fix the cause (a key, a model, a converter route), then replay it:
docker compose exec -T api \
remember ops replay 3f2a9c1e-8b7d-4e21-9a0f-5c6d7e8f9a0b --deployment "$DEPLOYMENT_ID"| Option | Default | Meaning |
|---|---|---|
--attempts N | 1 | How many more attempts to grant |
--lane steady|backfill | the item's lane | Run it in another lane |
--not-before 2026-09-24T08:00:00+00:00 | now | Do not run it before this time |
The command prints the item's new state: its route, attempts,
max_attempts and not_before. The worker for that stage picks it up.
Parked conversions
An uploaded file whose MIME type has no conversion route is parked with the
reason no_route rather than failed. It uses no attempts. After you add a
route and restart with docker compose up -d, release it:
docker compose exec -T api \
remember ops resume-no-route --deployment "$DEPLOYMENT_ID"The output lists the released processing_id values. Items whose MIME type
is still unrouted stay parked. See File formats and converters.
Spend budgets
A budget caps what one pipeline stage may spend on model calls, in US dollars, within a fixed time window. Before a worker runs an item, it checks the budget for that stage and lane. If the window's spend has reached the ceiling, the item is parked until the window ends, and the worker moves on. Nothing is dropped.
REMEMBERSTACK_WORK_BUDGETS is a JSON list; each entry names the
deployment, the stage, the lane (steady or backfill), the window in
seconds, and the ceiling:
# .env
REMEMBERSTACK_WORK_BUDGETS=[{"deployment_id":"<your deployment id>","stage":"extract_claims","lane":"steady","window_seconds":86400,"ceiling_usd":"5.00"},{"deployment_id":"<your deployment id>","stage":"normalize_relations","lane":"steady","window_seconds":86400,"ceiling_usd":"5.00"}]Only one budget may exist per deployment, stage and lane. Stages without a budget are not capped.
Put it in .env and run docker compose up -d; the workers enforce it.
Work a budget parked shows as pending with defer_reason budget, both
in remember ops inspect and in readiness.
The spend itself is in the cost ledger; read it with remember ops cost-export (Observability).
The graph catalog
The memory's graph is a set of PostgreSQL views over the stored relations,
plus the functions that walk them and the database roles and grants they
use. Migrations create them. If a manual database change or a restore leaves
them missing or different, remember ops graph-catalog ensure compares them
with what this release expects, recreates them if needed, and reports the
result:
docker compose exec -T api \
remember ops graph-catalog ensureThe output has ready, changed, problems_before and problems_after.
It also checks the versions of pgvector,
pg_textsearch and pg_partman.
When a hard forget is in progress
While a hard forget is running, the API answers 503 with
{"code": "forget_in_progress"}. Ordinary claims, publication and reconcile
refuse to run. The coordinator alone can erase source bytes and regenerate
navigation under the closed barrier.
The Python hard-forget coordinator and ordinary workers use the same purge path.
There is no public CLI command that starts a hard forget. API startup and
project --plane p3 replay every portable manifest before serving or repairing
restored data, including manifests previously marked complete. Any error keeps
admission closed. A host with controlled mount sessions must acknowledge stopping
them and erasing their caches; operator-managed FUSE mounts must be stopped and
remounted. See restore ordering.