Troubleshooting

Most problems with a self-hosted RememberStack fall into four groups: the stack did not start, the client talks to the wrong place or with the wrong credential, a document is stuck somewhere in the pipeline, or the answer is correct but not what you expected. This page lists each symptom with the first thing to check and the fix.

Run the commands on this page from the directory that holds your compose.yaml and .env.

Start here

Running operator commands

Several fixes below use remember ops. These commands read PostgreSQL directly, so they run inside the api container (docker compose exec api remember ops …). Run anywhere else, remember ops exits with "'remember ops' runs inside the engine container".

Read the deployment id once per shell:

DEPLOYMENT_ID=$(grep '^REMEMBERSTACK_SELFHOST_DEPLOYMENT_ID=' .env | cut -d= -f2)

Then pass it to each command:

docker compose exec -T api \
  remember ops inspect --deployment "$DEPLOYMENT_ID" | python3 -m json.tool

Operating the pipeline describes every field of the report.

The API does not answer

/healthz answers {"status":"ok"} when the API process is up and can run a query on PostgreSQL. It needs no token.

docker compose ps -a
curl http://localhost:8000/healthz

docker compose ps -a also lists containers that have stopped. Read it this way:

The API, the workers, PostgreSQL and the object store have the restart policy unless-stopped: Docker starts a container that exits again, until you stop it yourself. A container that fails at start therefore shows as restarting rather than exited. /healthz keeps answering ok while a worker is down: it checks the API and the database, not the pipeline.

Configuration mistakes that stop a container at start:

Wrong port or wrong address

Compose publishes the API on the host port in REMEMBERSTACK_SELFHOST_API_PORT, on 127.0.0.1 only unless REMEMBERSTACK_SELFHOST_API_PUBLISH_ADDRESS says otherwise, so another machine gets "Connection refused" until you open it. The remember client and CLI do not read that variable. Without REMEMBER_API_URL they use http://127.0.0.1:8000, whatever port the API is published on.

If you changed the port, tell the client:

export REMEMBER_API_URL=http://localhost:8080

remember doctor prints the address it checks and whether the API answered:

remember doctor

It asks GET /deployment, which needs a read or write credential. With an ingest token it reports an authentication failure even though the address is right.

401 and 403

The API sends one of four detail strings:

A read token can call every route that only reads, including the four assured operations and GET /deployment. It cannot ingest or change anything; that needs ingest or write. The full table is in Authentication and scopes.

A browser app cannot reach the API

A web page on another origin that calls the API gets a network error in the browser, and the request may not show up in the API logs at all. The API is running; the browser refused the response because the API did not name the page's origin.

Check REMEMBERSTACK_SELFHOST_BROWSER_ORIGINS. It is empty by default, which allows no origin. Each entry must be written exactly as the browser sends it: lowercase, no path, no trailing slash. It must be https, unless the app runs on the same machine: http://localhost:3000, http://127.0.0.1:5173 and http://[::1]:8080 are accepted, and any other http origin stops the API at start.

Only GET, POST and DELETE with the Authorization and Content-Type headers are allowed, and cookies are not. See Browser origins.

A document never becomes queryable

Ingest returns as soon as the bytes are stored. Everything after that runs in the workers and takes minutes. Before you look for a fault, ask readiness about the version_id that ingest returned:

import uuid
 
import remember
from remember import ReadinessRequirements
 
client = remember.Client.from_env()
report = client.pipeline_readiness(
    version_ids=(uuid.UUID("6f1c2d0e-8a3b-4d5e-9f10-2a3b4c5d6e7f"),),
    require=ReadinessRequirements(pipeline=True, p1=True, live_graph=True, p3=False),
)
for version in report.versions:
    for stage in version.stages:
        print(stage.stage, stage.status)

The document status in GET /documents is not readiness: a document is ready there once it is converted and structured, before its claims and facts exist.

See what is waiting and why

Readiness gives each waiting stage a defer_reason, and the routes in remember ops inspect count work by stage, status and defer_reason, for the whole deployment:

docker compose exec -T api \
  remember ops inspect --deployment "$DEPLOYMENT_ID"

defer_reason is no_route for a file no converter accepts, budget for work a spend budget parked, scheduled for work due later, and retry_backoff for a failure waiting to be retried. Pending work with no reason is simply queued for its worker.

Parked conversions

A file whose MIME type has no conversion route is stored and its convert work is parked with the reason no_route. It uses no attempts and makes no model calls. The ingest result reports it at once with "parked": "no_route", and remember ingest prints a warning.

A stock deployment routes Markdown, plain text, HTML, .docx, .pptx and .xlsx. PDFs and images need an OCR route with your own provider key. Add a route for the type as described in File formats and converters, apply it with docker compose up -d, then release the parked work:

docker compose exec -T api \
  remember ops resume-no-route --deployment "$DEPLOYMENT_ID"

It prints {"released": [...]} with the processing ids it released. Work whose MIME type is still unrouted stays parked.

A Markdown file parked as no_route

The MIME type is the one the upload declared, not something the engine detects. These uploads arrive with a type the default routes do not cover:

• a file or filename with no extension, or an extension the client does not map and the sending machine's MIME table does not know: application/octet-stream;
• an older remember client on a Python installation that does not know .md: application/octet-stream;
• a curl call with a type such as text/x-markdown. The match is exact.

Send the same file again with the right type: remember ingest notes --mime text/markdown, or mime="text/markdown" in Python. RememberStack stores identical bytes once, with the type of their first upload, but a type with no route is replaced by a later upload's type that has one, and every parked conversion of those bytes is released. The ingest result's mime shows the type now recorded.

Alternatively, add a route for the type that was recorded, for example "application/octet-stream": "passthrough", then run resume-no-route. passthrough needs valid UTF-8, so a binary file sent under that type fails conversion instead of waiting. To see which types were recorded:

docker compose exec -T postgres sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"' <<'SQL'
SELECT mime, count(*) FROM content_objects GROUP BY mime;
SQL

The current client sends .md and .markdown as text/markdown on every Python installation, and takes the type of bytes from their filename. For a file whose extension does not say what it is, pass the type: remember ingest notes --mime text/markdown, or mime="text/markdown" in Python.

Dead letters

Find what failed and why:

docker compose exec -T api \
  remember ops inspect --deployment "$DEPLOYMENT_ID" | python3 -m json.tool

dead_letters.groups shows the stage and error class; each entry in dead_letters.items has its processing_id and last_error, the full traceback of the last attempt. Fix the cause first: a key, a model id, a converter route, a file that is not UTF-8. Then give the item more attempts:

docker compose exec -T api \
  remember ops replay 3f2a9c1e-8b7d-4e21-9a0f-5c6d7e8f9a0b --deployment "$DEPLOYMENT_ID"

replay handles one item. If poison_targets lists the item, it failed under two releases of the same stage, so an upgrade did not fix it; look at the input itself. See Operating the pipeline.

Processing is slow

Minutes per document is normal: most stages wait on model calls, and some cannot start until every chunk or claim of the version has finished the stage before. Check where work is piling up:

docker compose exec -T api \
  remember ops inspect --deployment "$DEPLOYMENT_ID" | python3 -m json.tool
docker compose logs -f --since 10m worker-extract-claims worker-normalize-relations

Model key and provider errors

A missing key stops the containers at start (see The API does not answer). A wrong key does not: the stack starts, and every model call fails. .env.example ships the placeholder replace-before-real-use, which fails the same way.

What you see: retryable failure in stage … lines in the worker logs, with OpenRouter /chat/completions returned 401 (or 402 when the account is out of credit) in the traceback. After three attempts, a few seconds apart, the items dead-letter.

docker compose logs --since 30m worker-structure worker-extract-claims | grep -i openrouter

Fix: put a valid key in .env, apply it with docker compose up -d, then replay the dead letters. Other provider errors follow the same path:

A search that cannot embed its query answers 503 model provider unavailable. The same key fixes it.

Work parked by a spend budget

If you set REMEMBERSTACK_WORK_BUDGETS, a stage that reaches its ceiling parks its work until the budget window ends. Nothing fails and nothing is dropped; readiness shows the stage as pending.

A defer_reason of budget (see See what is waiting and why) shows it. Wait for the window to end, or raise ceiling_usd in .env and run docker compose up -d. See Spend budgets.

Empty or surprising answers

First make sure the documents are processed: an answer from before readiness does not include them, and nothing in it says so. Then read the result, not only its list of facts.

• negative says why a result is empty. unknown_entity means the name did not resolve; known_empty means nothing matches; boundary means the question could not be answered as asked, with a workaround.
• truncation.truncated means there is more than was returned.
• dropped_by_hydration counts candidates the search found that no longer hold. A high value right after ingest usually means processing is still settling.

Most surprising answers are the result of how the question was asked:

A SQL query is rejected

SQL queries run against the query space, memory_v1: prepared, read-only views and functions. Every statement is parsed and checked against that query space before it runs, and anything outside it is rejected.

A statement the API refuses or that fails while running does not come back as an HTTP error. POST /query/sql answers 200 with termination_reason set to rejected or failed, an error_code, an error_message and no rows. The Python client returns that result as it is; it does not raise. Check termination_reason on every result before you read rows:

result = client.open_query("SELECT fact_label FROM facts_current LIMIT 5")
if result["termination_reason"] != "completed":
    print(result["error_code"], result["error_message"])

The codes you meet most often:

Every code is in Errors and status codes.

503 forget_in_progress

The API answers 503 with {"code": "forget_in_progress"} while a hard forget is running, and the operator commands that copy or replay data refuse to run. A deployment started from compose.yaml has no command that starts a hard forget, so it should not enter this state on its own. If you restored data from an installation that ran one, see Hard-forget manifests and When a hard forget is in progress.

setup refuses an existing database

Some releases change what stored data means in a way that cannot be converted. Their migrations stop instead of guessing. On a database that already holds claims or chunks, setup stops with an error that ends in "recreate the deployment and ingest its sources again", and the API and workers do not start.

docker compose logs setup shows the message. To continue:

1. If you may want to go back, take a backup of the old volumes first, as described in Back up, while the old release is still checked out.
2. Remove the old data: docker compose down -v. This deletes the memory.
3. Start the new release: docker compose up -d --wait.
4. Send your source documents again.

To stay on the earlier release instead, check out its tag again and start it with docker compose up -d. setup applies all pending migrations in one database transaction, so the failure rolled back the ones before it too. See Upgrades and migrations.

Before you report a problem

Collect:

• build_revision and model_bindings from curl http://localhost:8000/deployment;
• the remember ops inspect output;
• docker compose logs --since 1h for the services involved, with secrets removed;
• the version_id and its readiness stages, for a stuck document.

last_error tracebacks and logs can quote text from your documents. Read them before you share them. See Contributing.