Lands the two remaining provenance-subsystem pieces:
mapper.py — map_segment_refs_to_provenance:
- For each LLM SegmentCitation, pick seg-ids per source_type
(`value` vs `value_and_context`), cap at max_sources_per_field,
resolve each via SegmentIndex, track invalid references.
- Resolve field values by dot-path (`result.items[0].name` supported —
`[N]` bracket notation is normalised to `.N` before traversal).
- Skip fields that resolve to zero valid sources (spec §9.4).
- Write quality_metrics with fields_with_provenance / total_fields /
coverage_rate / invalid_references.
verify.py — verify_field + apply_reliability_flags:
- Dispatches per Pydantic field type: date → parse-both-sides compare;
int/float/Decimal → normalize + whole-snippet / numeric-token scan;
IBAN (detected via `iban` in field name) → upper+strip compare;
Literal / None → flags stay None; else string substring.
- _unwrap_optional handles BOTH typing.Union AND types.UnionType so
`Decimal | None` (PEP 604, what get_type_hints emits on 3.12+) resolves
correctly — caught by the integration-style test_writes_flags_and_counters.
- Number comparator scans numeric tokens in the snippet so labels
("Closing balance CHF 1'234.56") don't mask the match.
- apply_reliability_flags mutates the passed ProvenanceData in place and
writes verified_fields / text_agreement_fields to quality_metrics.
Tests cover each comparator, Literal/None skip, short-value skip (strings
and numerics), Decimal via optional union, and end-to-end flag+counter
writing against a Pydantic use-case schema that mirrors bank_statement_header.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure functions the ReliabilityStep will compose to compare extracted values
against OCR snippets (and context.texts). Kept in one module so every rule
is directly unit-testable without pulling in the step ABC.
Highlights:
- `normalize_string`: NFKC + casefold + strip common punctuation (. , : ; !
? () [] {} / \\ ' " `) + collapse whitespace. Substring-compatible.
- `normalize_number`: returns the canonical "[-]DDD.DD" form (always 2dp)
after stripping currency symbols. Heuristic separator detection handles
Swiss-German apostrophes ("1'234.56"), de-DE commas ("1.234,56"), and
plain ASCII ("1234.56" / "1234.5" / "1234"). Accepts native int/float/
Decimal as well as str.
- `normalize_date`: dateutil parse with dayfirst=True → ISO YYYY-MM-DD.
Date and datetime objects short-circuit to their isoformat().
- `normalize_iban`: uppercase + strip whitespace. Format validation is the
call site's job; this is pure canonicalisation.
- `should_skip_text_agreement`: dispatches on type + value. Literal → skip,
None → skip, numeric |v|<10 → skip, len(str) ≤ 2 → skip. Numeric check
runs first so `10` (len("10")==2) is treated on the numeric side
(not skipped) instead of tripping the string length rule.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
