Files
skillfactor-pipeline/evals/skill-tiering/eval-set.json
skillfactor-pipeline 3b4a225594 feat(mapping): tiered relevance (core/adjacent) + source caps, eval 20/20
Deterministic tiers: core = token overlap with top-20 market hard skills or
essential ESCO competences; in-tier ranking by overlap score. Per-source cap
10 (overflow -> audit log, not the package). Source-domain allowlists stop
cross-contamination (marketing skill entering an engineering package via a
generic 'api' keyword). Flagship selection: 427 -> 73 entries; all 10 real
v1 flood negatives excluded, 10 must-have positives included.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDKeXvpT6tENSvyQGLV1Uq
2026-07-09 06:45:38 +02:00

28 lines
2.4 KiB
JSON

{
"target_slug": "artificial-intelligence-engineer",
"note": "20 real cases from the v1 ai-skills.md over-inclusion audit (427 entries for one occupation). Negatives are actual v1 floods (Google Mobile-Ads SDKs, marketing skills, generic test frameworks). Positives are must-have capabilities for an AI engineer, specified at SKILL granularity (v1 lesson: plugin-level names never match entry names). Ship criterion: >= 18/20 end-to-end (negative = must NOT be in the final capped selection; positive = must be in).",
"ship_threshold": 18,
"cases": [
{ "source": "wshobson", "skill_contains": "prompt-engineering", "expected": "include" },
{ "source": "wshobson", "skill_contains": "langchain", "expected": "include" },
{ "source": "wshobson", "skill_contains": "python", "expected": "include" },
{ "source": "anthropic", "skill": "mcp-builder", "expected": "include" },
{ "source": "superpowers", "skill": "test-driven-development", "expected": "include" },
{ "source": "superpowers", "skill": "systematic-debugging", "expected": "include" },
{ "source": "anthropic", "skill_contains": "claude-api", "expected": "include" },
{ "source": "nvidia", "skill_contains": "cuopt", "expected": "include" },
{ "source": "wshobson", "skill_contains": "debugging", "expected": "include" },
{ "source": "venice", "skill_contains": "embeddings", "expected": "include" },
{ "source": "google", "skill_contains": "mobile-ads-banner", "expected": "exclude" },
{ "source": "google", "skill_contains": "mobile-ads-interstitial", "expected": "exclude" },
{ "source": "google", "skill_contains": "mobile-ads-rewarded", "expected": "exclude" },
{ "source": "marketing", "skill_contains": "cold-email", "expected": "exclude" },
{ "source": "marketing", "skill_contains": "aso", "expected": "exclude" },
{ "source": "marketing", "skill_contains": "co-marketing", "expected": "exclude" },
{ "source": "advertising", "skill_contains": "ad-creative", "expected": "exclude" },
{ "source": "lambdatest", "skill_contains": "selenium", "expected": "exclude" },
{ "source": "lambdatest", "skill_contains": "appium", "expected": "exclude" },
{ "source": "resend", "skill_contains": "deliverability", "expected": "exclude" }
]
}