goldstein/infoxtractor
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
Async on-prem LLM-powered structured information extraction microservice
46 commits 44 branches 0 tags 765 KiB
Python 98.2%
Shell 1.1%
Dockerfile 0.7%
Find a file
Dirk Riemann 141153ffa7
All checks were successful
tests / test (push) Successful in 1m10s
Details
tests / test (pull_request) Successful in 1m10s
Details
feat(store): JobsRepo CRUD over ix_jobs + integration fixtures (spec §4) 
JobsRepo covers the full job-lifecycle surface:

- insert_pending: idempotent on (client_id, request_id) via ON CONFLICT
  DO NOTHING + re-select; assigns a 16-hex ix_id.
- claim_next_pending: FOR UPDATE SKIP LOCKED so concurrent workers never
  double-dispatch a row.
- get / get_by_correlation: hydrates JSONB back through Pydantic.
- mark_done: done iff response.error is None, else error.
- mark_error: explicit convenience wrapper.
- update_callback_status: delivered | failed (no status transition).
- sweep_orphans: time-based rescue of stuck running rows; attempts++.

Integration fixtures (tests/integration/conftest.py):
- Skip cleanly when neither IX_TEST_DATABASE_URL nor IX_POSTGRES_URL is
  set (unit suite stays runnable on a bare laptop).
- Alembic upgrade/downgrade runs in a subprocess so its internal
  asyncio.run() doesn't collide with pytest-asyncio's loop.
- Per-test engine + truncate so loops never cross and tests start clean.

15 integration tests against a live postgres:16, including SKIP LOCKED
concurrency + orphan sweep.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 11:43:11 +02:00
.forgejo/workflows ci: run on every push (not just main) so feat branches also get CI 2026-04-18 10:40:44 +02:00
alembic feat(store): Alembic scaffolding + initial ix_jobs migration (spec §4) 2026-04-18 11:37:21 +02:00
docs Implementation plan for ix MVP 2026-04-18 10:34:30 +02:00
scripts test(pipeline): end-to-end hermetic test with fakes + synthetic fixture 2026-04-18 11:24:29 +02:00
src/ix feat(store): JobsRepo CRUD over ix_jobs + integration fixtures (spec §4) 2026-04-18 11:43:11 +02:00
tests feat(store): JobsRepo CRUD over ix_jobs + integration fixtures (spec §4) 2026-04-18 11:43:11 +02:00
.env.example feat(scaffold): project skeleton with uv + pytest + forgejo CI 2026-04-18 10:36:43 +02:00
.gitignore feat(scaffold): project skeleton with uv + pytest + forgejo CI 2026-04-18 10:36:43 +02:00
.python-version feat(scaffold): project skeleton with uv + pytest + forgejo CI 2026-04-18 10:36:43 +02:00
AGENTS.md Initial design: on-prem LLM extraction microservice MVP 2026-04-18 10:23:17 +02:00
alembic.ini feat(store): Alembic scaffolding + initial ix_jobs migration (spec §4) 2026-04-18 11:37:21 +02:00
pyproject.toml feat(scaffold): project skeleton with uv + pytest + forgejo CI 2026-04-18 10:36:43 +02:00
README.md Initial design: on-prem LLM extraction microservice MVP 2026-04-18 10:23:17 +02:00
uv.lock feat(scaffold): project skeleton with uv + pytest + forgejo CI 2026-04-18 10:36:43 +02:00

README.md

InfoXtractor (ix)

Async, on-prem, LLM-powered structured information extraction microservice.

Given a document (PDF, image, text) and a named use case, ix returns a structured JSON result whose shape matches the use-case schema — together with per-field provenance (OCR segment IDs, bounding boxes, cross-OCR agreement flags) that let the caller decide how much to trust each extracted value.

Status: design phase. Implementation about to start.

Principles

  • On-prem always. LLM = Ollama, OCR = local engines (Surya first). No OpenAI / Anthropic / Azure / AWS / cloud.
  • Grounded extraction, not DB truth. ix returns best-effort fields + provenance; the caller decides what to trust.
  • Transport-agnostic pipeline core. REST + Postgres-queue adapters in parallel on one job store.