feat(segmentation): SegmentIndex + prompt-text formatter (spec §9.1) #6

Merged

goldstein merged 1 commit from feat/segment-index into main

2026-04-18 08:54:03 +00:00

goldstein commented

2026-04-18 08:53:57 +00:00

Owner

Adds SegmentIndex.build(...), lookup_segment(id), and to_prompt_text(context_texts=...) per spec §9.1.

Page-tag lines (<page …> / </page>) excluded from IDs so they can never be cited as provenance.
IDs are p{global_1based}_l{line_idx_in_page} (dense within page).
Bboxes normalised 0-1 by dividing by page width/height.
File-index carried via a parallel PageMetadata list (ingestion-decoupled).
Prompt text appends context.texts untagged after a blank line.

CI trigger is flaky for now; local tests green (49 passed, ruff clean).

Adds `SegmentIndex.build(...)`, `lookup_segment(id)`, and `to_prompt_text(context_texts=...)` per spec §9.1. - Page-tag lines (`<page …>` / `</page>`) excluded from IDs so they can never be cited as provenance. - IDs are `p{global_1based}_l{line_idx_in_page}` (dense within page). - Bboxes normalised 0-1 by dividing by page width/height. - File-index carried via a parallel `PageMetadata` list (ingestion-decoupled). - Prompt text appends `context.texts` untagged after a blank line. CI trigger is flaky for now; local tests green (49 passed, ruff clean).

goldstein added 1 commit 2026-04-18 08:53:57 +00:00

feat(segmentation): SegmentIndex + prompt-text formatter (spec §9.1)

tests / test (push) Successful in 58s

Details

tests / test (pull_request) Successful in 56s

Details

1321d57354

Builds the ID <-> on-page-anchor map used by both the GenAIStep (to emit the
segment-tagged user message) and the provenance mapper (to resolve LLM-cited
IDs back to bbox/text/file_index).

Design notes:

- `build()` is a classmethod so the pipeline constructs the index in one
  place (OCRStep) and passes the constructed instance along in the internal
  context. No mutable global state; tests build indexes inline from fake
  OCR fixtures.

- Per-page metadata (file_index) arrives via a parallel `list[PageMetadata]`
  rather than being smuggled into OCRResult. Keeps segmentation decoupled
  from ingestion — the OCR engine legitimately doesn't know which file a
  page came from.

- Page-tag lines (`<page …>` / `</page>`) are filtered via a regex so the
  LLM can never cite them as provenance. `line_idx_in_page` increments only
  for real lines so the IDs stay dense (p1_l0, p1_l1, ...).

- Bounding-box normalisation divides x-coords by page width, y-coords by
  page height. Zero dimensions (defensive) pass through unchanged.

- `to_prompt_text(context_texts=[...])` appends paperless-style texts
  untagged, separated from the tagged body by a blank line (spec §7.2b).
  Deterministic for prompt caching.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

goldstein merged commit b2ff27c1ca into main

2026-04-18 08:54:03 +00:00

goldstein referenced this pull request from a commit

2026-04-18 08:54:04 +00:00

feat(segmentation): SegmentIndex + prompt-text formatter (spec §9.1) (#6)

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: goldstein/infoxtractor#6

No description provided.