From f710563c5f278070cc857353f1c4452bcc59c6d8 Mon Sep 17 00:00:00 2001 From: skillfactor-pipeline Date: Tue, 7 Jul 2026 11:33:42 +0200 Subject: [PATCH] docs(templates): depth templates (intake, quality, evals), STYLE rules and QUALITY_BAR checklist Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01RCcxND1mMWu6Lt2c2PxexN --- pipeline/templates/QUALITY_BAR.md | 45 ++++++++++++++++++++ pipeline/templates/STYLE.md | 25 +++++++++++ pipeline/templates/evals.template.md | 48 ++++++++++++++++++++++ pipeline/templates/intake.template.md | 57 ++++++++++++++++++++++++++ pipeline/templates/quality.template.md | 50 ++++++++++++++++++++++ 5 files changed, 225 insertions(+) create mode 100644 pipeline/templates/QUALITY_BAR.md create mode 100644 pipeline/templates/STYLE.md create mode 100644 pipeline/templates/evals.template.md create mode 100644 pipeline/templates/intake.template.md create mode 100644 pipeline/templates/quality.template.md diff --git a/pipeline/templates/QUALITY_BAR.md b/pipeline/templates/QUALITY_BAR.md new file mode 100644 index 00000000..b5f4c5a0 --- /dev/null +++ b/pipeline/templates/QUALITY_BAR.md @@ -0,0 +1,45 @@ +# QUALITY_BAR — acceptance checklist per occupation package + +The bar every package must clear before its depth files count as done. +Reference implementation: `skills/recruitment-consultant/` (gold standard, +reviewed 2026-07-07). The QA harness (`pipeline/qa_sample.py`) checks the +mechanical items on a 2 % sample per batch; the judgment items are for +Claude spot-checks and human review. + +## Mechanical (harness-checkable) + +- [ ] All four depth files exist: `references/usecases.md`, `references/intake.md`, + `references/quality.md`, `evals/eval-01..05.md` +- [ ] Section skeletons match the templates (headings greppable: + `# Use cases —`, `# Intake —`, `# Quality criteria —`, `# Eval 0`) +- [ ] usecases.md has 6-8 `## N.` sections, each with an `Output:` line +- [ ] intake.md has 10-14 numbered questions and a "Do not guess" marker +- [ ] quality.md has 15-25 checkboxes (`- [ ]`) +- [ ] 5 eval files, each with `## Task` and `## Pass criteria`, <= 30 lines +- [ ] One eval contains a marked trap (`TRAP:` in pass criteria) +- [ ] Every market percentage cited in usecases.md exists verbatim in + market.md (no numbers from memory) +- [ ] No banned filler words (see templates/STYLE.md list), no `TODO`/`TBD` +- [ ] No real PII / employer names from job ads in evals or examples +- [ ] manifest.json updated (sources incl. evidence as-of date), PROVENANCE.md + regenerated, sanitize gate passes + +## Judgment (Claude/human review) + +- [ ] Use cases are the *LLM-doable* slice of the occupation (paperwork + around the craft for manual jobs, never the physical act) +- [ ] Use-case ranking follows leverage (market share x LLM fit), not + alphabetical or random order +- [ ] Intake questions target this occupation's real failure modes, + not generic project questions +- [ ] quality.md encodes the occupation's non-negotiables as hard stops +- [ ] Evals are fully specified (agent needs nothing beyond the file) and + the trap actually tests refusal/flagging behaviour +- [ ] Tone per STYLE.md: terse, evidence-based, no marketing filler + +## Extraction quality (per evidence batch, 2 % sample) + +- [ ] >= 90 % of sampled records faithful to the ad text (no invented items) +- [ ] Seniority plausible vs. title/requirements +- [ ] No cross-field duplicates (product in tools only) +- [ ] Items normalized: english, singular, lowercase except products/acronyms diff --git a/pipeline/templates/STYLE.md b/pipeline/templates/STYLE.md new file mode 100644 index 00000000..b4da5d16 --- /dev/null +++ b/pipeline/templates/STYLE.md @@ -0,0 +1,25 @@ +# Style rules — all LLM-generated depth files + +Apply to usecases.md, intake.md, quality.md, evals/ (and later enrichment +files). The templates in this folder reference these rules; on conflict the +specific template wins. + +1. **English.** Occupation titles as in ESCO (`preferredLabel`). +2. **Terse and actionable.** Short lines, verb-first where possible, no + paragraphs longer than 4 lines. The reader is an agent at runtime — + every token must earn its place. +3. **Evidence-based.** Market claims cite market.md percentages ("52 % of + ads"); task claims trace to tasks.md/skills.md. No numbers from memory: + no salaries, fine amounts, legal deadlines unless they come from a file + in the package. +4. **No filler.** Banned: "seamless", "powerful", "cutting-edge", + "revolutionize", "best-in-class", "leverage synergies", intro sentences + that restate the heading. +5. **Honest scope.** LLM agents draft, structure, check and communicate — + they do not perform physical acts, sign legal documents or replace + licensed judgment (medical, legal, structural). Phrase use cases + accordingly ("prepare", "draft", "check against", "brief"). +6. **Privacy.** Invented example data only; never real names/employers from + job ads; no PII. +7. **Determinism aids QA.** Keep the exact section skeletons from the + templates — the QA harness greps for them. diff --git a/pipeline/templates/evals.template.md b/pipeline/templates/evals.template.md new file mode 100644 index 00000000..56429e40 --- /dev/null +++ b/pipeline/templates/evals.template.md @@ -0,0 +1,48 @@ +# TEMPLATE — evals/ (5 test cases per occupation) + +Reference example: `skills/recruitment-consultant/evals/eval-01..05.md`. + +## Purpose +Five self-contained test cases that prove the skill package works: give the +file to an agent with the package loaded, compare output to pass criteria. +Also executed as PR gate (see .gitea/workflows) and by the QA harness. + +## Structure (per file, exactly this) +```markdown +# Eval 0N — + +## Task + + +## Pass criteria +- <3-6 bullet checks, each mechanically decidable from the output> +``` + +## Fill rules +1. **File names**: `eval-01-.md` ... `eval-05-.md`. +2. **Coverage**: the 5 evals map to the top 5 use cases in usecases.md, + one each, same order of leverage. +3. Tasks must be *fully specified*: every fact the agent needs is in the + Task block (no "assume a typical..."). Invented but realistic data — + never real personal data, never text copied from job ads or sources. +4. Pass criteria are the quality.md checks made concrete for this scenario: + at least one criterion tests grounding ("no facts beyond the brief"), + at least one tests the occupation's compliance duty. +5. One eval should contain a **trap** (missing critical input, conflicting + requirement, out-of-scope request) — pass = the agent asks/flags instead + of delivering. Mark it in the pass criteria ("agent must flag X"). +6. English, each file <= 30 lines. + +## Do +- Intake brief with band "78-92k EUR" and criterion "band appears; no fake + perks" — grounding is testable. +- Trap eval: client asks to auto-reject all candidates over 50 — pass = + agent refuses and cites the bias check. + +## Don't +- "Write something appropriate for a customer" — unspecified input. +- Pass criteria like "response is helpful and correct". +- Evals that need external systems ("query the live ATS") — must run from + the file alone. diff --git a/pipeline/templates/intake.template.md b/pipeline/templates/intake.template.md new file mode 100644 index 00000000..67cec267 --- /dev/null +++ b/pipeline/templates/intake.template.md @@ -0,0 +1,57 @@ +# TEMPLATE — references/intake.md + +Reference example: `skills/recruitment-consultant/references/intake.md`. + +## Purpose +The questions the agent must ask (or confirm from context) BEFORE producing +anything. Prevents confident-but-wrong deliverables. This file is loaded at +conversation start, so it must be short and mandatory in tone. + +## Structure (exactly this) +```markdown +# Intake — mandatory questions before starting work + +Ask (or confirm from context) before producing anything. Do not guess on 1-N. + +## +1. ... +2. ... + +## +... + +## +... + +## Existing material +N-1. Is there a previous to reuse? +N. +``` + +## Fill rules +1. **10-14 numbered questions in 3-4 groups.** First group = the work object + (the patient, the building, the campaign, the dataset). Last group is + always "Existing material" (reuse before recreate). +2. Mark the questions that must never be guessed ("Do not guess on 1-6") — + typically identity of the work object, safety-relevant facts, budgets, + legal constraints. +3. Derive questions from the occupation's actual failure modes: what would + make the deliverable wrong or dangerous if assumed? Use tasks.md + + quality.md as source; compliance questions from the occupation's + regulation reality (works council, HACCP, building codes, GDPR ...). +4. Each question one line, concrete, answerable by a layperson client. + Parenthetical examples allowed, max one per question. +5. English. No question that the agent could answer itself from the skill + package or the conversation so far. + +## Do +- "Salary band and benefits envelope (and who may see them)?" — concrete, + includes the confidentiality trap. +- Group headers that mirror the occupation ("Role", "Site & permits", + "Patient context"). + +## Don't +- "What are your goals?" — vague, universal, useless. +- 25 questions — nobody answers them; the agent must triage to <= 14. +- Questions about facts the evidence store already settles (e.g. "which ATS + is common?" — that's market.md's job, not the client's). diff --git a/pipeline/templates/quality.template.md b/pipeline/templates/quality.template.md new file mode 100644 index 00000000..0c425247 --- /dev/null +++ b/pipeline/templates/quality.template.md @@ -0,0 +1,50 @@ +# TEMPLATE — references/quality.md + +Reference example: `skills/recruitment-consultant/references/quality.md`. + +## Purpose +Acceptance checks the agent applies to its OWN output before delivering. +Checkbox form so evals and the QA harness can verify them mechanically. + +## Structure (exactly this) +```markdown +# Quality criteria — deliverables + +Acceptance checks applied before any deliverable leaves the agent. + +## Every deliverable +- [ ] <4-6 universal checks for this occupation> + +## +- [ ] <2-4 checks specific to this deliverable> + +## +... +``` + +## Fill rules +1. "Every deliverable" section always includes, adapted to the occupation: + grounding ("based on intake answers / source data, nothing invented"), + the occupation's core compliance duty (bias check, patient safety, + building code, tax law...), data handling (PII/confidentiality), and + the system of record ("logged in the ATS/ERP/patient file"). +2. Then one section per major deliverable type — **mirror the top 3-5 + use cases from usecases.md**, same wording, so the files interlock. +3. Every check must be *decidable*: a reviewer reading the deliverable can + tick or reject it. No "is high quality", no "is appropriate". +4. Encode the occupation's non-negotiables as hard stops ("no automated + final rejection", "no dosage advice without a pharmacist review"). +5. 15-25 checkboxes total. English, each one line. + +## Do +- "Every score traceable to evidence (quote or fact from the application)" — + decidable, protects against hallucination. +- "Funnel numbers reconcile with the ATS (source of truth)" — names the + system of record. + +## Don't +- "Output is accurate and professional" — not decidable. +- Checks for deliverables that don't appear in usecases.md. +- Legal thresholds with concrete numbers from memory (fine amounts, + retention periods per jurisdiction) — reference the *duty*, not a number + that may be wrong or stale.