938ed54e15
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RCcxND1mMWu6Lt2c2PxexN
86 lines
4.5 KiB
Markdown
86 lines
4.5 KiB
Markdown
# HANDOVER — Fable 5 → Sonnet 4.6 (2026-07-07)
|
|
|
|
Quality foundation is done. Your job is the multi-day full-catalog run.
|
|
**Read `Sites(staging)\skillfactor\auftrag_sonnet.md`, follow
|
|
`pipeline/templates/` and `pipeline/prompts/` strictly — do not redesign
|
|
anything.** Design contracts: `docs/batch-architecture.md`.
|
|
|
|
## What Fable built today (all committed)
|
|
|
|
| Artefact | Where | State |
|
|
|---|---|---|
|
|
| Baseline inventory | `docs/inventory.md` | done |
|
|
| Extraction schema | `pipeline/prompts/extract_schema.json` | final |
|
|
| Extraction prompt (Ollama) | `pipeline/prompts/extract_posting.txt` | final, tested |
|
|
| Local extractor + validator | `pipeline/extract_local.py` | **done & tested: 20/20 clean, ø 6.5 s/ad on gemma3:27b** |
|
|
| Depth templates + style | `pipeline/templates/*.template.md`, `STYLE.md` | final |
|
|
| Acceptance bar | `pipeline/templates/QUALITY_BAR.md` | final |
|
|
| Gold standard | `skills/recruitment-consultant/` | reviewed & upgraded (Output lines, verified market %, TRAP eval) |
|
|
| Budget/resume module | `pipeline/progress.py` | done & tested (hard cap 33 000 JSearch req) |
|
|
| QA harness | `pipeline/qa_sample.py` | done & tested |
|
|
| Batch runner | `pipeline/batch_run.py` | **skeleton — you implement the TODOs** |
|
|
| Architecture design | `docs/batch-architecture.md` | final |
|
|
| `.env` | `OLLAMA_URL` / `OLLAMA_MODEL` added | works |
|
|
|
|
## Your exact start sequence
|
|
|
|
1. Read `auftrag_sonnet.md`, `docs/batch-architecture.md`,
|
|
`pipeline/batch_run.py` (the TODOs name their own acceptance criteria).
|
|
2. Apply the MSSQL change (batch-architecture.md §MSSQL) —
|
|
`occupation_slug` column + backfill + index.
|
|
3. Implement `stage_packages` (reuse `p2_generate.py`/`p4_publish.py`,
|
|
throttled ≥ 1 s/repo, check-before-create) and run Phase 1 for all
|
|
3 039 ESCO occupations.
|
|
4. Implement `stage_evidence` (generalize p3a per occupation, extraction
|
|
via `extract_local.py` as-is, incremental p3b, per-slug p3c).
|
|
Every JSearch HTTP call goes through `progress.spend_request()` FIRST.
|
|
5. After each occupation batch: `qa_sample.py --rate 0.02`; two consecutive
|
|
exit-1 = stop and diagnose. Depth files must pass QUALITY_BAR mechanical
|
|
checks before pushing.
|
|
6. Depth tier 1 (top-200) per templates; tier 3 as resumable night batch.
|
|
|
|
## Hard rules (from the finalize order — not negotiable)
|
|
|
|
- JSearch lifetime budget 33 000 requests — enforced in code, do not bypass.
|
|
- ≤ 2 req/s; cache first; thin market → synonyms → Adzuna → mark failed.
|
|
- Mass extraction (~180 k ads ≈ 14 GPU-days) runs as a **detached script**
|
|
(`run_in_background` / scheduled task), never inside your session loop.
|
|
- No secrets in code/logs. `.env` stays uncommitted.
|
|
- Commits small and conventional.
|
|
|
|
## Open points / caveats
|
|
|
|
1. **Git history was squashed today 11:21** into one commit (orphan branch
|
|
`clean-history` → renamed master) — not by Claude. Old history is kept
|
|
as tag `backup/pre-squash-history`. Ask Robert before deleting the tag;
|
|
if the squash was unintended, the old commits are all still there.
|
|
2. `ADZUNA_APP_ID/KEY` in `.env` are empty — the finalize order lists
|
|
values (§2); copy them in before the Adzuna fallback is needed.
|
|
3. Enrichment templates (curriculum/learning/literature/glossary/
|
|
capabilities) do NOT exist yet — Fable's day covered the four depth
|
|
files per auftrag_fable §3. Derive the enrichment ones from
|
|
STYLE.md + the recruiter examples when Phase 3 starts, same pattern.
|
|
4. `p2_generate.py` currently emits a fixed balanced set of 100 — needs the
|
|
per-occupation refactor (stage_packages TODO).
|
|
5. SKILL.md trigger descriptions come from crosswalk O*NET tasks and are
|
|
sometimes off-profile (recruiter shows HR-manager tasks). Known
|
|
generator limitation — fix in stage_packages if cheap, else log it.
|
|
6. Ollama box (91.184.179.239:11434) is unencrypted/unauthenticated —
|
|
COLOGNE2 firewall must restrict port 11434 to this webserver's IP
|
|
(flagged to Robert earlier, unverified).
|
|
7. `data/progress.json` is committed in its empty initial state; it is
|
|
run-state, commit it whenever you checkpoint a batch.
|
|
|
|
## Verification snapshot (what "working" looked like today)
|
|
|
|
```
|
|
.venv/Scripts/python.exe pipeline/extract_local.py \
|
|
--ads data/raw/jobs/recruiter_ads.json \
|
|
--out data/evidence/testset_fable.jsonl --limit 20
|
|
# -> done: 20 ok, 0 failed (0% fail)
|
|
.venv/Scripts/python.exe pipeline/qa_sample.py \
|
|
--extractions data/evidence/testset_fable.jsonl \
|
|
--ads data/raw/jobs/recruiter_ads.json --rate 0.10
|
|
# -> 2 sampled, 0 mechanical issues
|
|
```
|