skillfactor-pipeline/docs/quality-audit-ai-engineer.md

Quality audit — skills-core/artificial-intelligence-engineer (v1)

Date: 2026-07-09 · Baseline: git tag v1 (af05383b) · Auditor: pipeline session Scope: SKILL.md, references/market.md, references/ai-skills.md + the generating pipeline code (p2_generate, p3a-c, p5_enrich_ai_skills).

Findings (measured against the v1 tag)

F1 — market.md contamination: CONFIRMED, 24% of corpus

data/raw/jobs/artificial-intelligence-engineer_ads.json holds 66 ads; 16 of 66 (24%) are designer/creative/marketing roles. Verbatim titles (kept as labeled near-misses for the relevance-gate eval set): "AI-Powered UI/UX Designer", "Senior AI UI/UX Designer", "Principal Product Designer, SeekrGuard", "Graphic Designer - AI Trainer", "AI Automation Design Packaging Lead", "UX and UI Designer for AI-Driven Software Development Application", "AI Performance Creative Designer (Meta/YouTube Ads)", "Staff AI Product Designer, Gemini Assistant", "Senior AI Product Designer, Gemini Assistant", "AI Web/UX Designer", "Lead Product Designer", "Senior AI Experience Designer" (+4 more). Effect on rankings: Figma 15%, Sketch 8%, interaction design 12%, user research 11%, product design 8% — none belong in an AI-engineer profile. Root cause: p3a fetches by title/altLabel keyword ("AI …") with no relevance gate; extraction ingests every fetched ad.

F2 — corpus thin and US-only: CONFIRMED

market.md header: "66 real job ads (JSearch API, countries: us 66)". No EU/DACH slice despite ESCO (EU) taxonomy base and DACH/EU target market. Root cause: p3a country list; no per-region reporting or minimum-corpus threshold in p3c.

F3 — ai-skills.md over-inclusion + count mismatch: CONFIRMED, worse than assumed

The file contains 427 table entries; the landing-page chip claims "83 agent skills" (hard-coded from an earlier run). Flooding sources: LambdaTest test-framework skills, NVIDIA vendor stack, n8n workflows, Google Mobile-Ads SDKs, marketing/PR skills. Root cause: p5 auto-domain keyword matching is pure recall (any keyword hit maps the whole skill), no per-source cap, no relevance tiering; all landing-page numbers are hard-coded in home.tmpl instead of generated.

F4 — SKILL.md quality defects: CONFIRMED

• Grammar: intro is unedited ESCO raw text ("Artificial intelligence engineer apply methods…", "normally requiring high level of human expertise").
• Competence list: p2_generate.py:255 cuts essential[:15] after an alphabetical sort → list ends mid-alphabet at "deliver visual presentation of data" (a–d only), silently.
• "Hot technologies": alphabetical ONET dump (Alteryx, Amazon EC2, …), not a relevance ranking; ONET data here is a proxy (15-2051.00 Data Scientists — no exact ONET occupation exists). The proxy caveat exists in tasks.md/tools.md/footer since 2026-07-08 but not yet as the standardized one-liner wherever ONET data is used.

Corrections to the briefing assumptions

• None material. One nuance: the "83" landing-page number was accurate for the 3-source enrichment run and became stale after the 16-source run — the defect is hard-coded counts, not a broken counter.

Phase-0 verification

• Raw ads persisted: 180 slugs under data/raw/jobs/*_ads.json (with title/employer/country/provenance) + 180 extraction outputs data/evidence/*.jsonl — fully re-processable without re-fetching. US-side re-classification can run 100% from disk.
• v1 immutable baseline: tag v1 (af05383b) on branch master; work continues on quality/reference-standard.
• v1 heatmap: pending the Phase-3 linter (by design).
• Eval harvest: the 16 contaminated titles above are the seed for evals/ad-relevance/ (real near-misses, no synthetic ones needed).