skillfactor-pipeline/docs/batch-architecture.md

Batch architecture — full-catalog build-out (design, Fable 5, 2026-07-07)

Design is final; Sonnet implements the TODOs in pipeline/batch_run.py without redesigning. Contracts below are what the skeleton, progress.py, extract_local.py and qa_sample.py already assume.

Staging

Stage	Scope	LLM	Where
1 packages	all 3 039 ESCO occupations	none	local + throttled Gitea push
2 evidence	all occupations, ~60 ads each	Ollama (extraction only)	local + RTX-3090 box
3 depth tier 1	top-200 by evidence volume	Claude (per templates/)	Claude session
3 depth tier 3	rest, night batch	Claude, resumable	Claude sessions

Order within stage 2 follows docs/batch-plan.md (HR/IT -> finance -> management), then the remaining catalog by ISCO group.

Budgets & throttles

• JSearch: hard cap 33 000 requests lifetime, enforced in progress.spend_request() BEFORE each HTTP call — not advisory. ~6 requests/occupation x 3 039 = ~18 k expected, cap leaves headroom.
• Rate: <= 2 req/s (sleep 0.5 s), cache first — a cache hit costs nothing.
• Thin market (< 20 usable ads): ESCO-synonym fallback -> Adzuna (free) -> mark evidence: failed / thin market, move on. No endless retries.
• Gitea: >= 1 s between repo operations, check-before-create (resumable).
• Ollama: sequential, one ad per call (~6.5 s measured on gemma3:27b); ~180 k ads ≈ 14 days GPU time — run as detached script on the webserver (survives Claude sessions), NOT inside the Claude loop.

Data contracts

• data/raw/jobs/<slug>/jsearch_<query>_<country>_p<N>.json — raw cache (existing recruiter files stay where they are; new fetches use subdirs)
• data/raw/jobs/<slug>_ads.json — deduped [{job_id, title, employer, country, description}] (same shape as recruiter_ads.json)
• data/evidence/<slug>.jsonl — one validated extraction per line (schema: pipeline/prompts/extract_schema.json + job_id)
• data/progress.json — see pipeline/progress.py docstring; the ONLY place run state lives. Deleting it = full re-scan, but no re-spend (caches + file existence still short-circuit).

MSSQL change (prerequisite for stage 2)

ALTER TABLE evidence_job ADD occupation_slug NVARCHAR(200) NULL;
UPDATE evidence_job SET occupation_slug = 'recruitment-consultant'
  WHERE occupation_slug IS NULL;  -- backfill: all 350 existing rows are recruiter
CREATE INDEX ix_evidence_job_occ ON evidence_job(occupation_slug);

p3b_store_evidence.py becomes incremental: MERGE on (occupation_slug, job_id) instead of drop-and-rebuild; evidence_entity keyed by job_id as today. p3c_aggregate.py parameterized by slug.

QA cadence

After each occupation's extraction: qa_sample.py --rate 0.02 (min 1 record). Claude reviews the generated docs/qa/<slug>-sample.md in the next session pass; two consecutive harness failures (exit 1) = stop the batch, diagnose (systemic prompt/model drift, not noise). Depth files are checked against pipeline/templates/QUALITY_BAR.md (mechanical part greppable).

Failure policy

Any per-occupation failure is recorded (error field) and skipped — the batch never blocks on a single occupation. --slugs re-runs individual failures after a fix. BudgetExhausted stops the run with exit 2.