feat(deploy): setup_server.sh + deployment runbook (#28)

Lands Task 5.2.

docs/deployment.md

@@ -0,0 +1,124 @@
+# Deployment
+
+On-prem deploy to `192.168.68.42`. Push-to-deploy via a bare git repo + `post-receive` hook that rebuilds the Docker Compose stack. Pattern mirrors mammon and unified_messaging.
+
+## Topology
+
+```
+Mac (dev)
+  │  git push server main
+  ▼
+192.168.68.42:/home/server/Public/infoxtractor/repos.git   (bare)
+  │  post-receive → GIT_WORK_TREE=/…/app git checkout -f main
+  │                 docker compose up -d --build
+  │                 curl /healthz (60 s gate)
+  ▼
+Docker container `infoxtractor` (port 8994)
+  ├─ host.docker.internal:11434  →  Ollama (gpt-oss:20b)
+  └─ host.docker.internal:5431   →  postgis (database `infoxtractor`)
+```
+
+## One-time server setup
+
+Run **once** from the Mac. Idempotent.
+
+```bash
+export IX_POSTGRES_PASSWORD=<generate-a-strong-one>
+./scripts/setup_server.sh
+```
+
+The script:
+1. Creates `/home/server/Public/infoxtractor/repos.git` (bare) + `/home/server/Public/infoxtractor/app/` (worktree).
+2. Installs the `post-receive` hook (see `scripts/setup_server.sh` for the template).
+3. Creates the `infoxtractor` Postgres role + database on the shared `postgis` container.
+4. Writes `/home/server/Public/infoxtractor/app/.env` (mode 0600) from `.env.example` with the password substituted in.
+5. Verifies `gpt-oss:20b` is pulled in Ollama.
+6. Prints a hint to open UFW for port 8994 on the LAN subnet if it's missing.
+
+After the script finishes, add the deploy remote to the local repo:
+
+```bash
+git remote add server ssh://server@192.168.68.42/home/server/Public/infoxtractor/repos.git
+```
+
+## Normal deploy workflow
+
+```bash
+# after merging a feat branch into main
+git push server main
+
+# tail the server's deploy log
+ssh server@192.168.68.42 "tail -f /tmp/infoxtractor-deploy.log"
+
+# healthz gate (the post-receive hook also waits up to 60 s for this)
+curl http://192.168.68.42:8994/healthz
+
+# end-to-end smoke — this IS the real acceptance test
+python scripts/e2e_smoke.py
+```
+
+If the post-receive hook exits non-zero (healthz never reaches 200), the deploy is considered failed. The previous container keeps running (the hook swaps via `docker compose up -d --build`, which first builds the new image and only swaps if the build succeeds; if the new container fails `/healthz`, it's still up but broken). Investigate with `docker compose logs --tail 200` in `${APP_DIR}` and either fix forward or revert (see below).
+
+## Rollback
+
+Never force-push `main`. Rollbacks happen as **forward commits** via `git revert`:
+
+```bash
+git revert HEAD     # creates a revert commit for the last change
+git push forgejo main
+git push server main
+```
+
+## First deploy
+
+_(fill in after running — timestamps, commit sha, e2e_smoke output)_
+
+- **Date:** TBD
+- **Commit:** TBD
+- **`/healthz` first-ok time:** TBD
+- **`e2e_smoke.py` status:** TBD
+- **Notes:** —
+
+## Operational checklists
+
+### After `ollama pull` on the host
+
+The `IX_DEFAULT_MODEL` env var on the server's `.env` must match something in `ollama list`. Changing the default means:
+
+1. Edit `/home/server/Public/infoxtractor/app/.env` → `IX_DEFAULT_MODEL=<new>`.
+2. `docker compose --project-directory /home/server/Public/infoxtractor/app restart`.
+3. `curl http://192.168.68.42:8994/healthz` → confirm `ollama: ok`.
+
+### If `/healthz` shows `ollama: degraded`
+
+`gpt-oss:20b` (or the configured default) is not pulled. On the host:
+```bash
+ssh server@192.168.68.42 "docker exec ollama ollama pull gpt-oss:20b"
+```
+
+### If `/healthz` shows `ocr: fail`
+
+Surya couldn't initialize (model missing, CUDA unavailable, OOM). First run can be slow — models download on first call. Check container logs:
+```bash
+ssh server@192.168.68.42 "docker logs infoxtractor --tail 200"
+```
+
+### If the container fails to start
+
+```bash
+ssh server@192.168.68.42 "tail -100 /tmp/infoxtractor-deploy.log"
+ssh server@192.168.68.42 "docker compose -f /home/server/Public/infoxtractor/app/docker-compose.yml logs --tail 200"
+```
+
+## Monitoring
+
+- Monitoring dashboard auto-discovers via the `infrastructure.web_url` label on the container: `http://192.168.68.42:8001` → "infoxtractor" card.
+- Backup opt-in via `backup.enable=true` + `backup.type=postgres` + `backup.name=infoxtractor` labels. The daily backup script picks up the `infoxtractor` Postgres database automatically.
+
+## Ports
+
+| Port | Direction | Source | Service |
+|------|-----------|--------|---------|
+| 8994/tcp | ALLOW | 192.168.68.0/24 | ix REST + healthz (LAN only; not publicly exposed) |
+
+No VPS Caddy entry; no `infrastructure.docs_url` label — this is an internal service.

scripts/setup_server.sh

@@ -0,0 +1,127 @@
+#!/usr/bin/env bash
+# One-shot server setup for InfoXtractor. Idempotent: safe to re-run.
+#
+# Run from the Mac:
+#   IX_POSTGRES_PASSWORD=<pw> ./scripts/setup_server.sh
+#
+# What it does on 192.168.68.42:
+#   1. Creates the bare git repo `/home/server/Public/infoxtractor/repos.git` if missing.
+#   2. Writes the post-receive hook (or updates it) and makes it executable.
+#   3. Creates the Postgres role + database on the shared `postgis` container.
+#   4. Writes `/home/server/Public/infoxtractor/app/.env` (0600) from .env.example.
+#   5. Verifies `gpt-oss:20b` is pulled in Ollama.
+
+set -euo pipefail
+
+SERVER="${IX_SERVER:-server@192.168.68.42}"
+APP_BASE="/home/server/Public/infoxtractor"
+REPOS_GIT="${APP_BASE}/repos.git"
+APP_DIR="${APP_BASE}/app"
+DB_NAME="infoxtractor"
+DB_USER="infoxtractor"
+
+if [ -z "${IX_POSTGRES_PASSWORD:-}" ]; then
+  read -r -s -p "Postgres password for role '${DB_USER}': " IX_POSTGRES_PASSWORD
+  echo
+fi
+
+if [ -z "${IX_POSTGRES_PASSWORD}" ]; then
+  echo "IX_POSTGRES_PASSWORD is required." >&2
+  exit 1
+fi
+
+echo "==> 1/5  Ensuring bare repo + post-receive hook on ${SERVER}"
+ssh "${SERVER}" bash -s <<EOF
+set -euo pipefail
+mkdir -p "${REPOS_GIT}" "${APP_DIR}"
+if [ ! -f "${REPOS_GIT}/HEAD" ]; then
+  git init --bare "${REPOS_GIT}"
+fi
+
+cat >"${REPOS_GIT}/hooks/post-receive" <<'HOOK'
+#!/usr/bin/env bash
+set -eo pipefail
+
+APP_DIR="${APP_DIR}"
+LOG="/tmp/infoxtractor-deploy.log"
+
+echo "[\$(date -u '+%FT%TZ')] post-receive start" >> "\$LOG"
+
+mkdir -p "\$APP_DIR"
+GIT_WORK_TREE="\$APP_DIR" git --git-dir="${REPOS_GIT}" checkout -f main >> "\$LOG" 2>&1
+
+cd "\$APP_DIR"
+docker compose up -d --build >> "\$LOG" 2>&1
+
+# Deploy gate: /healthz must return 200 within 60 s.
+for i in \$(seq 1 30); do
+  if curl -fsS http://localhost:8994/healthz > /dev/null 2>&1; then
+    echo "[\$(date -u '+%FT%TZ')] healthz OK" >> "\$LOG"
+    exit 0
+  fi
+  sleep 2
+done
+
+echo "[\$(date -u '+%FT%TZ')] healthz never reached OK" >> "\$LOG"
+docker compose logs --tail 100 >> "\$LOG" 2>&1 || true
+exit 1
+HOOK
+
+chmod +x "${REPOS_GIT}/hooks/post-receive"
+EOF
+
+echo "==> 2/5  Verifying Ollama has gpt-oss:20b pulled"
+if ! ssh "${SERVER}" "docker exec ollama ollama list | awk '{print \$1}' | grep -qx 'gpt-oss:20b'"; then
+  echo "FAIL: gpt-oss:20b not found in Ollama. Run: ssh ${SERVER} 'docker exec ollama ollama pull gpt-oss:20b'" >&2
+  exit 1
+fi
+
+echo "==> 3/5  Creating Postgres role '${DB_USER}' and database '${DB_NAME}' on postgis container"
+# Idempotent via DO blocks; uses docker exec to avoid needing psql on the host.
+ssh "${SERVER}" bash -s <<EOF
+set -euo pipefail
+docker exec -i postgis psql -U postgres <<SQL
+DO \\\$\\\$
+BEGIN
+  IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = '${DB_USER}') THEN
+    CREATE ROLE ${DB_USER} LOGIN PASSWORD '${IX_POSTGRES_PASSWORD}';
+  ELSE
+    ALTER ROLE ${DB_USER} WITH PASSWORD '${IX_POSTGRES_PASSWORD}';
+  END IF;
+END
+\\\$\\\$;
+SQL
+
+if ! docker exec -i postgis psql -U postgres -tc "SELECT 1 FROM pg_database WHERE datname = '${DB_NAME}'" | grep -q 1; then
+  docker exec -i postgis createdb -U postgres -O ${DB_USER} ${DB_NAME}
+fi
+EOF
+
+echo "==> 4/5  Writing ${APP_DIR}/.env on the server"
+# Render .env from the repo's .env.example, substituting the password placeholder.
+LOCAL_ENV_CONTENT="$(
+  sed "s#<password>#${IX_POSTGRES_PASSWORD}#g" \
+      "$(dirname "$0")/../.env.example"
+)"
+# Append the IX_TEST_MODE=production for safety (fake mode stays off).
+# .env is written atomically and permissioned 0600.
+ssh "${SERVER}" "install -d -m 0755 '${APP_DIR}' && cat > '${APP_DIR}/.env' <<'ENVEOF'
+${LOCAL_ENV_CONTENT}
+ENVEOF
+chmod 0600 '${APP_DIR}/.env'"
+
+echo "==> 5/5  Checking UFW rule for port 8994 (LAN only)"
+ssh "${SERVER}" "sudo ufw status numbered | grep -F 8994" >/dev/null 2>&1 || {
+  echo "NOTE: UFW doesn't yet allow 8994. Run on the server:"
+  echo "  sudo ufw allow from 192.168.68.0/24 to any port 8994 proto tcp"
+}
+
+echo
+echo "Done."
+echo
+echo "Next steps (on the Mac):"
+echo "  git remote add server ssh://server@192.168.68.42${REPOS_GIT}"
+echo "  git push server main"
+echo "  ssh ${SERVER} 'tail -f /tmp/infoxtractor-deploy.log'"
+echo "  curl http://192.168.68.42:8994/healthz"
+echo "  python scripts/e2e_smoke.py"