Pipeline stages

Exhaustive stage reference for the per-document evidence chain. Parent overview: Ingestion.

Stage graph

ingest → convert → structure → chunk → embed(chunks)
                                      ↓
                         select propositions and source references
                                      ↓
                         ground claims → embed(claims)
                                      ↓
                              normalize / resolve entities
                                      ↓
                              adjudicate facts (supersession, contradictions)
                                      ↓
                              reconcile (currency, evidence counts)
                                      ↓
                         P1 updates · live graph · (scheduled) P3 · K triggers

All stages are idempotent with respect to their versioned inputs. Workers use the deployment work ledger: retry, then dead-letter for systemic failures (provider outage, database errors, unrecoverable handler bugs). That is not the same as a re-derivable soft drop inside a successful stage: a predicate outside the governed vocabulary is preserved through the other: escape when possible, otherwise it is dropped with structured telemetry and does not dead-letter the document version. Entity types and domain/range signature gates are not part of E3.

E0 — Files

Ingest

Job: accept bytes, compute content_hash, attach connector identity, create or advance lineage/version.

Rules:

• Content-identical payloads reuse the content object.
• Lineage identity is connector-native (source_kind, source_ref) — not title, path, or hash.
• source_modified_at feeds claim assertion time; cursor-only polls must not rewrite it casually.

Outputs: raw object URI, document/version rows, enqueue convert.

Convert

Job: produce an immutable representation: clean Markdown + source map (+ media sidecars when composed).

Shipped smoke profile: registers text/markdown and text/plain → Markdown passthrough. Unregistered MIME types are ingested and kept, but their conversion is parked rather than attempted, so nothing dead-letters — see Troubleshooting.

Designed / composable routes (register on the deployment; do not assume the Compose smoke stack has them):

Rules:

• Conversion is not regenerated in place when tools improve: a new run = new representation_id beside the old.
• Media claims inherit derivation kind and evidence mode (source_expression | model_observation | model_interpretation) for honest read-side disclosure when media routes are used.
• Raw originals remain the byte authority; Markdown is the navigation-promoted form.

Structure

Job: per-document section tree (PageIndex-class): headings, roles, spans, summaries, placement hints.

Rules:

• Prefer deterministic heading/skeleton construction.
• Bounded model seats for role/summary/sanity/fallback only — not unbounded freeform rewrite of the whole doc.
• Structure feeds selection for claim extraction and P3 navigation stubs.

E1 — Chunks

Blockizer

Job: deterministic sequence of paragraph-grain blocks from document.md.

Why blocks exist: section trees snap to a stable grid; chunk packing becomes editable without reshuffling the world.

Chunk packer

Job: pack whole blocks into non-overlapping, section-bounded, anchor-stabilized chunks.

Properties:

• Re-ingest after a small edit reuses unchanged chunk work (with lifecycle cache keys).
• Context prefixes (headers, titles) are generated for embedding/readability but stored separately from source body under the embedding-input policy.
• Chunks are evidence grain source text — never atomic claims, never adjudicated facts.

Chunk embedding (P1)

Job: write vector + FTS rows for chunks under the active embedder generation.

Policy notes (shipped):

• P1 text column holds normalized body (not silent header+body mash that poisons audit).
• Location headers return separately when present.
• FTS index bootstrap and write-path optimization are automatic; reads must not full-scan the corpus.

E2 — Claims

Extraction

Job: extract coherent, verifiable natural-language assertions from chunks.

Each claim carries:

• Immutable text identity (assertion-by-a-source)
• An origin span on the claim row, plus the complete list of supporting body ranges on the selected occurrence (and through those ranges to page/region/time locators)
• Source-asserted validity interval when the source states one (asserted validity — testimony, not system verdict)
• Extractor / representation / structure generation coordinates (basis)

Extraction has two model calls. Selection chooses propositions and identifies source-backed people, organizations, works and particular events. Its response is saved once, including an empty result. After all chunks finish Selection, Claimify writes standalone claims using their kept propositions and a bounded set of earlier references. A reference can cross a section boundary, but remains within the previous eight chunks of the same document representation.

Each earlier reference includes exact source passages. Its descriptive name helps orientation; the source passages must support the resulting claim and be cited. For example, a tournament introduced in chunk 2 can clarify “that tournament” in chunk 7. Two tournaments with the same date remain separate references. The engine supplies passage labels and resolves them to exact occurrence ranges.

The initial limits are four published references per Selection response and eight references, totaling at most 4,096 characters including evidence, per Claimify request. These are bounded context limits, not a guarantee that every distant reference is available. No additional extraction model call is introduced.

Version reuse includes the earlier chunks' source inputs even when they produced no references. An edit there can introduce a competing referent and must trigger rechecking. Unchanged inputs reuse the saved decisions and claim identities, remapping all evidence ranges onto the new occurrence.

Gates:

• Deterministic grounding checks reject unanchored inventions.
• Decision ledgers record extraction outcomes for replay and eval.

Claim embedding (P1)

Job: index claim text for semantic + lexical nomination. Default evidence search uses both channels (not double semantic).

Currency (bookkeeping, not validity)

When a new extraction basis completes, prior claims for that basis become non-current testimony via an append-only transitions ledger. Claim rows themselves stay immutable. See Lifecycle.

Source time and claim windows

The extractor receives the full source timestamp, using source_modified_at when present and otherwise published_at. A relative phrase such as “last Friday” or “three hours ago” is resolved against that instant. The resolved date is written into the claim text in ISO form (“printed the screenplay on 2022-01-21”), exactly as a pronoun is replaced with the name it refers to, and the same value is stored in the structured validity fields. The verbatim source wording stays in source_span. A phrase the extractor cannot resolve stays as the source spoke it, with no guessed date: a relative phrase still present in claim text means resolution failed, and it is read against asserted_at, the time the source made the statement.

The written date is grounded by the claim’s own validity fields rather than by the source text: a date that does not match the stored bounds, or appears with unknown precision, is rejected as an outside fact.

The prompt distinguishes event_time (a happening), effective_period (a state or arrangement), measurement_period (a reporting span), and proposition_validity (when a proposition holds). “Has been CEO since 2019” is an open-ended window. Kind and precision are separate: a year-long employment period is a state period, not automatically an event.

Missing dates stay unknown. Relative expressions without an anchor and part-of-day expressions without representable bounds are not given invented instants. Absolute dates stated in the text do not require a header timestamp. These instructions improve extraction; they do not guarantee that every model output has the correct kind or date.

E3 — Facts

Normalized assertions may name a few extra source-backed context entities (a tournament, company, or person the claim refers to besides the subject). Those bindings live on the assertion application. Candidate nomination still starts with the same-subject baseline and may add a small number of facts that share that context. Shared context never proves two assertions are the same fact, and an empty context search does not authorize a new fact.

Entity resolution

Job: map surface forms to canonical entity ids (write path).

Write-time cascade tiers:

Query-time resolve_entity is T0 exact today — see Concepts and Primitives. Merges are reversible with transcripts. Only canonical ids flow downstream.

Write-time T0 never accepts an exact name by itself. T4 prefers a compatible existing entity: missing overlap and different topics do not establish a new referent. new requires positive distinction from every supplied candidate; it may mint a second active entity with the same normalized lemma and stores exclusions only for those supplied candidates. Candidate truncation remains visible in the audit but does not create a third decision state. Lemmas that span multiple entity ids are marked promiscuous and rank below unguarded fuzzy/phonetic matches without being dropped from the candidate set. The shipped cascade has no exact-name auto-accept flag.

After supported observations or relations change, a deterministic refresher rebuilds the entity’s short summary and entity-profile-v3:dated-history vector. The summary includes completed history with explicit world dates. It is an identity description, not a list of statements guaranteed to hold today. Unchanged input hashes skip embedding. Supersession, merge/un-merge, terminal human review, normal deletion/reconciliation, and hard-forget use the same refresh seam; if no supported fact remains, every profile cache field clears together. A survivor profile includes evidence from its complete redirect closure. A merged member keeps a separately attested member-local profile for joint neighborhood re-decision, but remains excluded from public entity resolution and search. During the profile-policy cut, the self-host image's setup entrypoint clears name-only vectors, rebuilds active and merged entity profiles in bounded resumable pages, and publishes entity semantic search only after that pass completes.

Current profile publication invokes bounded local clustering over the touched entity's alias lemmas. Identical merge-review proposals deduplicate by their live roots and cluster configuration. The default remains fail-closed: auto_merge_enabled=false, so unattended convergence produces reviewable diagnostics rather than an uncalibrated silent merge.

Bare head nouns (game, app, system, card, photo, module, the system) are not minted as entities. Qualified names (FIFA 23, James's Unity strategy game) may mint. When the claim spelling differs from the canonical name (App vs Application), both surfaces are stored as aliases on the same id. Entities have no type class: extract emits names only. works_for between two people is allowed.

Predicate governance

Relations use a governed predicate vocabulary with escape values and promotion — not free-text edge soup.

Relations vs observations

Both have evidence links and one chosen world-time window. Claims preserve what a source said; adjudication can revise a fact's chosen dates when new context justifies a correction. The fact's system belief timestamps remain separate.

Both shapes pass through contextual adjudication before a fact is created or reused. Dates, wording and surrounding context help decide whether two reports refer to the same event. Repeating the same tournament name does not force a merge, and a corrected date does not force a new fact. A shared person or event is not a shared proposition: winning, participating, and enjoying stay distinct. A participation or enjoyment report is not positive evidence of a win just because it shares the event. Date corrections change the chosen window of a date-neutral statement; the writer does not rewrite that statement. There is no stored state/occurrence category or separate date-dispute workflow.

The adjudicator sees compact semantic evidence with attempt-local names (F1, C1, A1). Database IDs, generation strings, and membership hashes stay internal. The writer still validates the full snapshot. Distinct sources with identical wording remain distinct testimony.

Evidence links

Many-to-many fact_claim_evidence with stance supports | contradicts. Counts aggregate by distinct current-testimony lineages.

Supersession & contradiction

• A justified successor can end its predecessor at the successor's world start.
• Ending an interval does not retract the historical fact. Learning that a fact was wrong instead closes system belief with invalidated_at.
• An ordinary adjudication can change the whole chosen window, in either direction, or clear dates that lack grounding. Attaching evidence without a date revision preserves the chosen window.
• Contradictions group live disagreements; both sides remain.
• Every adjudication is a logged, replayable decision.

Withdrawal and world dates

Withdrawing testimony changes its evidence currency. If no supporting testimony remains, the engine closes system belief in the fact. Withdrawal does not choose an end date for the real-world event or relationship. A world-time end changes only through a grounded adjudication, such as evidence that employment ended in June—even if the source was withdrawn in September.

Reconciliation

Runs when basis or version changes:

1. Flip testimony currency for affected claims.
2. Recount evidence / contradict counts.
3. Close sole-supported facts under living removal rules.
4. Flag support-withdrawn cases from toolchain re-derivation for review.
5. Emit change signals for K / projections consumers.

Projection touchpoints

Failure modes & operations

Operator surfaces: deployment CLI, ledger queries, rebuild drills. See Deployment.

Next

• Lifecycle & versions
• Retrieval
• Architecture