3b54f996ca
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RCcxND1mMWu6Lt2c2PxexN
70 lines
3.3 KiB
Markdown
70 lines
3.3 KiB
Markdown
# Batch architecture — full-catalog build-out (design, Fable 5, 2026-07-07)
|
|
|
|
Design is final; Sonnet implements the TODOs in `pipeline/batch_run.py`
|
|
without redesigning. Contracts below are what the skeleton, `progress.py`,
|
|
`extract_local.py` and `qa_sample.py` already assume.
|
|
|
|
## Staging
|
|
|
|
| Stage | Scope | LLM | Where |
|
|
|---|---|---|---|
|
|
| 1 packages | all 3 039 ESCO occupations | none | local + throttled Gitea push |
|
|
| 2 evidence | all occupations, ~60 ads each | Ollama (extraction only) | local + RTX-3090 box |
|
|
| 3 depth tier 1 | top-200 by evidence volume | Claude (per templates/) | Claude session |
|
|
| 3 depth tier 3 | rest, night batch | Claude, resumable | Claude sessions |
|
|
|
|
Order within stage 2 follows `docs/batch-plan.md` (HR/IT -> finance ->
|
|
management), then the remaining catalog by ISCO group.
|
|
|
|
## Budgets & throttles
|
|
|
|
- **JSearch: hard cap 33 000 requests lifetime**, enforced in
|
|
`progress.spend_request()` BEFORE each HTTP call — not advisory.
|
|
~6 requests/occupation x 3 039 = ~18 k expected, cap leaves headroom.
|
|
- Rate: <= 2 req/s (sleep 0.5 s), cache first — a cache hit costs nothing.
|
|
- Thin market (< 20 usable ads): ESCO-synonym fallback -> Adzuna (free) ->
|
|
mark `evidence: failed / thin market`, move on. No endless retries.
|
|
- Gitea: >= 1 s between repo operations, check-before-create (resumable).
|
|
- Ollama: sequential, one ad per call (~6.5 s measured on gemma3:27b);
|
|
~180 k ads ≈ 14 days GPU time — run as detached script on the webserver
|
|
(survives Claude sessions), NOT inside the Claude loop.
|
|
|
|
## Data contracts
|
|
|
|
- `data/raw/jobs/<slug>/jsearch_<query>_<country>_p<N>.json` — raw cache
|
|
(existing recruiter files stay where they are; new fetches use subdirs)
|
|
- `data/raw/jobs/<slug>_ads.json` — deduped `[{job_id, title, employer,
|
|
country, description}]` (same shape as recruiter_ads.json)
|
|
- `data/evidence/<slug>.jsonl` — one validated extraction per line
|
|
(schema: `pipeline/prompts/extract_schema.json` + `job_id`)
|
|
- `data/progress.json` — see `pipeline/progress.py` docstring; the ONLY
|
|
place run state lives. Deleting it = full re-scan, but no re-spend
|
|
(caches + file existence still short-circuit).
|
|
|
|
## MSSQL change (prerequisite for stage 2)
|
|
|
|
```sql
|
|
ALTER TABLE evidence_job ADD occupation_slug NVARCHAR(200) NULL;
|
|
UPDATE evidence_job SET occupation_slug = 'recruitment-consultant'
|
|
WHERE occupation_slug IS NULL; -- backfill: all 350 existing rows are recruiter
|
|
CREATE INDEX ix_evidence_job_occ ON evidence_job(occupation_slug);
|
|
```
|
|
|
|
`p3b_store_evidence.py` becomes incremental: MERGE on
|
|
`(occupation_slug, job_id)` instead of drop-and-rebuild; `evidence_entity`
|
|
keyed by job_id as today. `p3c_aggregate.py` parameterized by slug.
|
|
|
|
## QA cadence
|
|
|
|
After each occupation's extraction: `qa_sample.py --rate 0.02` (min 1
|
|
record). Claude reviews the generated `docs/qa/<slug>-sample.md` in the next
|
|
session pass; two consecutive harness failures (exit 1) = stop the batch,
|
|
diagnose (systemic prompt/model drift, not noise). Depth files are checked
|
|
against `pipeline/templates/QUALITY_BAR.md` (mechanical part greppable).
|
|
|
|
## Failure policy
|
|
|
|
Any per-occupation failure is recorded (`error` field) and skipped — the
|
|
batch never blocks on a single occupation. `--slugs` re-runs individual
|
|
failures after a fix. BudgetExhausted stops the run with exit 2.
|