Completes the data-contract layer. Highlights:
- `ResponseIX.context` is an internal mutable accumulator used by pipeline
steps (pages, files, texts, use_case classes, segment index). It MUST NOT
leak into the serialised response, so we mark the field with
`Field(exclude=True)` and carry the shape in a small `_InternalContext`
sub-model with `extra="allow"` so steps can stash arbitrary state without
schema churn. Tested: `model_dump()` and `model_dump_json()` both drop it.
- `FieldProvenance` gains `provenance_verified: bool | None` and
`text_agreement: bool | None` — the two MVP reliability flags written by
the new ReliabilityStep. Both default None so rows predating the
ReliabilityStep (empty LLM output, cloud-import replay) parse cleanly.
- `quality_metrics` stays a free-form `dict[str, Any]` — the MVP adds
`verified_fields` and `text_agreement_fields` counters without carving
them into the schema, which keeps future metric additions free.
- `Job.status` and `Job.callback_status` are `Literal[...]` so Pydantic
rejects unknown states at the edge. Invariant
(`status='done' iff response.error is None`) stays worker-enforced —
callers sometimes hydrate in-flight rows and we do not want validation
to reject them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dirk Riemann
02db3b05cc