# Batch architecture — full-catalog build-out (design, Fable 5, 2026-07-07)

Design is final; Sonnet implements the TODOs in `pipeline/batch_run.py`
without redesigning. Contracts below are what the skeleton, `progress.py`,
`extract_local.py` and `qa_sample.py` already assume.

## Staging

| Stage | Scope | LLM | Where |
|---|---|---|---|
| 1 packages | all 3 039 ESCO occupations | none | local + throttled Gitea push |
| 2 evidence | all occupations, ~60 ads each | Ollama (extraction only) | local + RTX-3090 box |
| 3 depth tier 1 | top-200 by evidence volume | Claude (per templates/) | Claude session |
| 3 depth tier 3 | rest, night batch | Claude, resumable | Claude sessions |

Order within stage 2 follows `docs/batch-plan.md` (HR/IT -> finance ->
management), then the remaining catalog by ISCO group.

## Budgets & throttles

- **JSearch: hard cap 33 000 requests lifetime**, enforced in
  `progress.spend_request()` BEFORE each HTTP call — not advisory.
  ~6 requests/occupation x 3 039 = ~18 k expected, cap leaves headroom.
- Rate: <= 2 req/s (sleep 0.5 s), cache first — a cache hit costs nothing.
- Thin market (< 20 usable ads): ESCO-synonym fallback -> Adzuna (free) ->
  mark `evidence: failed / thin market`, move on. No endless retries.
- Gitea: >= 1 s between repo operations, check-before-create (resumable).
- Ollama: sequential, one ad per call (~6.5 s measured on gemma3:27b);
  ~180 k ads ≈ 14 days GPU time — run as detached script on the webserver
  (survives Claude sessions), NOT inside the Claude loop.

## Data contracts

- `data/raw/jobs/<slug>/jsearch_<query>_<country>_p<N>.json` — raw cache
  (existing recruiter files stay where they are; new fetches use subdirs)
- `data/raw/jobs/<slug>_ads.json` — deduped `[{job_id, title, employer,
  country, description}]` (same shape as recruiter_ads.json)
- `data/evidence/<slug>.jsonl` — one validated extraction per line
  (schema: `pipeline/prompts/extract_schema.json` + `job_id`)
- `data/progress.json` — see `pipeline/progress.py` docstring; the ONLY
  place run state lives. Deleting it = full re-scan, but no re-spend
  (caches + file existence still short-circuit).

## MSSQL change (prerequisite for stage 2)

```sql
ALTER TABLE evidence_job ADD occupation_slug NVARCHAR(200) NULL;
UPDATE evidence_job SET occupation_slug = 'recruitment-consultant'
  WHERE occupation_slug IS NULL;  -- backfill: all 350 existing rows are recruiter
CREATE INDEX ix_evidence_job_occ ON evidence_job(occupation_slug);
```

`p3b_store_evidence.py` becomes incremental: MERGE on
`(occupation_slug, job_id)` instead of drop-and-rebuild; `evidence_entity`
keyed by job_id as today. `p3c_aggregate.py` parameterized by slug.

## QA cadence

After each occupation's extraction: `qa_sample.py --rate 0.02` (min 1
record). Claude reviews the generated `docs/qa/<slug>-sample.md` in the next
session pass; two consecutive harness failures (exit 1) = stop the batch,
diagnose (systemic prompt/model drift, not noise). Depth files are checked
against `pipeline/templates/QUALITY_BAR.md` (mechanical part greppable).

## Failure policy

Any per-occupation failure is recorded (`error` field) and skipped — the
batch never blocks on a single occupation. `--slugs` re-runs individual
failures after a fix. BudgetExhausted stops the run with exit 2.