|
|
290e51416f
|
feat(ingestion): fetch_file + MIME sniff + DocumentIngestor (spec §6.1)
tests / test (push) Successful in 57s
tests / test (pull_request) Successful in 1m12s
Three layered modules the SetupStep will wire together in Task 2.4.
- fetch.py: async httpx fetch with configurable timeouts + incremental
size cap (stream=True, accumulate bytes, raise IX_000_007 when
exceeded). file:// URLs read locally. Auth headers pass through. The
caller injects a FetchConfig — env reads happen in ix.config (Chunk 3).
- mime.py: python-magic byte-sniff + SUPPORTED_MIMES frozenset +
require_supported(mime) helper that raises IX_000_005.
- pages.py: DocumentIngestor.build_pages(files, texts) ->
(list[Page], list[PageMetadata]). PDFs via PyMuPDF (hard 100 pg/PDF
cap -> IX_000_006), images via Pillow (multi-frame TIFFs yield
multiple Pages), texts as zero-dim Pages so GenAIStep can still cite
them.
21 new unit tests (141 total) cover: fetch success with headers, 4xx/5xx
mapping, timeout -> IX_000_007, size cap enforced globally + per-file,
file:// happy path + missing file, MIME detection for PDF/PNG/JPEG/TIFF,
require_supported gate, PDF/TIFF/text page counts, 101-page PDF ->
IX_000_006, multi-file file_index assignment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
2026-04-18 11:12:00 +02:00 |
|
