129 lines
5.6 KiB
Markdown
129 lines
5.6 KiB
Markdown
# skillfactor — Occupational Skill Library (MVP)
|
|
|
|
**skillfactor** ("KI mit Berufserfahrung") builds ready-to-use occupational skill
|
|
packages for AI agents: public occupation databases (ESCO, O*NET) + live job-ad
|
|
evidence (JSearch/Adzuna APIs) + expert knowledge (Wikipedia/Wikidata,
|
|
literature) are compiled into one git repo per occupation and published to a
|
|
git-native marketplace (Gitea).
|
|
|
|
## Architecture
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
A[ESCO / O*NET / Crosswalk] --> DB[(MSSQL 'skillfactor')]
|
|
J[Job ads via JSearch/Adzuna APIs] --> DB
|
|
W[Wikipedia / Wikidata / literature] --> G
|
|
DB --> G[Generator]
|
|
G --> S[skills/<slug>/ packages]
|
|
S --> M[Gitea marketplace<br/>orgs: skills-core / skills-community / tenant-acme]
|
|
```
|
|
|
|
- **Evidence store: Microsoft SQL Server** (database `skillfactor`, local
|
|
instance) — project requirement: the whole stack runs on MSSQL (Gitea itself
|
|
included). No SQLite.
|
|
- **Pipeline:** one script per phase in `pipeline/` (`p1_load.py`, …),
|
|
orchestrated by `pipeline/run_all.py`. Downloads are cached in `data/raw/`
|
|
(not committed).
|
|
- **Skill package layout** (per occupation, language: English):
|
|
|
|
```
|
|
skills/<slug>/
|
|
├── SKILL.md # frontmatter: name, description (synonyms + task
|
|
│ # triggers), core workflow; < 300 lines
|
|
├── references/
|
|
│ ├── profile.md # occupation profile (ESCO), ESCO URI, ISCO code
|
|
│ ├── tasks.md # O*NET task statements + detailed work activities
|
|
│ ├── skills.md # ESCO skills (essential/optional)
|
|
│ └── tools.md # O*NET tools & technology
|
|
└── manifest.json # ids, sources, version, layer:"core", attribution
|
|
```
|
|
|
|
## Setup
|
|
|
|
1. Python >= 3.11, `python -m venv .venv`, `pip install -r requirements.txt`
|
|
2. Copy `.env` (never committed): Gitea URL/token, JSearch/Adzuna keys, MSSQL
|
|
credentials (see `.env` keys in `pipeline/db.py`)
|
|
3. `python pipeline/run_all.py`
|
|
|
|
## Data sources & licensing
|
|
|
|
- **ESCO** (European Commission) — occupations/skills/relations. Usage per the
|
|
[ESCO conditions](https://esco.ec.europa.eu/en/use-esco/download).
|
|
- **O*NET** (U.S. Department of Labor, CC BY 4.0) — task statements, tools &
|
|
technology, detailed work activities. Attribution included in every
|
|
`manifest.json` and below.
|
|
- Job ads exclusively via official APIs (JSearch, Adzuna) — no HTML scraping.
|
|
- Wikipedia/Wikidata content is paraphrased, never copied; source URLs cited.
|
|
|
|
> This product includes information from the O*NET Database by the U.S.
|
|
> Department of Labor, Employment and Training Administration (USDOL/ETA),
|
|
> used under the CC BY 4.0 license. skillfactor is not endorsed by USDOL/ETA.
|
|
|
|
## Data load smoke test (Phase 1)
|
|
|
|
| Table | Rows |
|
|
|---|---|
|
|
| esco_occupation | 3,039 |
|
|
| esco_skill | 13,939 |
|
|
| esco_occ_skill | 126,051 |
|
|
| esco_isco_group | 619 |
|
|
| onet_occupation | 1,016 |
|
|
| onet_task | 18,796 |
|
|
| onet_software | 31,821 |
|
|
| onet_task_dwa | 23,850 |
|
|
| onet_dwa | 2,087 |
|
|
| crosswalk_esco_onet | 4,210 |
|
|
|
|
Note (O*NET 30.3): the former "Technology Skills" file is now called
|
|
**"Software Skills"**; tools data loads from there.
|
|
|
|
## Status / TODO
|
|
|
|
- [x] Phase 0 — setup (repo, venv, MSSQL evidence store, Gitea Actions + runner)
|
|
- [x] Phase 1 — load ESCO/O*NET/crosswalk into MSSQL
|
|
- [x] Phase 2 — generate ~100 core skill packages (exactly 100 published)
|
|
|
|
### Phase 2 spot check (5 samples read manually)
|
|
|
|
`recruitment-consultant`, `accountant`, `marketing-manager`,
|
|
`ict-help-desk-agent`, `market-research-analyst`:
|
|
- structure complete (SKILL.md < 300 lines, 4 reference files, manifest)
|
|
- ESCO description/competences and O*NET tasks/software are consistent with
|
|
the occupation; crosswalk matches are plausible (recruiter → 13-1071.00)
|
|
- known limitation: O*NET "hot technologies" for broad SOC codes contain
|
|
generic office software — the job-ad market evidence (phase 3) is the
|
|
corrective for that
|
|
- first selection draft was ISCO-11..13-heavy (alphabetical fill); fixed via
|
|
round-robin bucket selection across ISCO prefixes (11:7, 12:17, 13:11,
|
|
24:23, 25:13, 26:6, 33:24, 35:8, 42:6)
|
|
- [x] Phase 3 — recruiter deep enrichment (glossary, literature, usecases,
|
|
intake, quality, 5 evals)
|
|
- [x] Phase 4 — published to Gitea: orgs `skills-core`/`skills-community`
|
|
(public) + `tenant-acme` (private), 100 repos, marketplace, PR gates
|
|
green (sanitize + eval on self-hosted act_runner, host mode)
|
|
- [x] Phase 5 — screenshots in `docs/screenshots/`
|
|
|
|
### Market evidence (phase 3) — DONE
|
|
|
|
101 unique recruiter ads (us+gb) fetched via JSearch (11 of 200 monthly
|
|
requests), entities extracted (1,859) into the MSSQL evidence store,
|
|
aggregated at the ≥20 % threshold into the recruiter package:
|
|
**ATS 54 %**, LinkedIn/LinkedIn Recruiter 26 %, candidate sourcing 48 %,
|
|
full-cycle recruiting 33 %, candidate screening 27 %, talent pipelining 29 %.
|
|
Remaining occupations follow the staged plan in `docs/batch-plan.md`
|
|
(~30 occupations/month within the free tier).
|
|
|
|
### Open TODO
|
|
- Structured-interview article on Wikipedia describes the survey-research
|
|
method; the hiring context in `glossary.md` is our own framing.
|
|
|
|
### Environment notes (this deployment)
|
|
|
|
- Gitea 1.24.5 on **Microsoft SQL Server**, served at
|
|
`https://zeiterfassung.cloud/skillfactor/` (Caddy sub-path setup).
|
|
- Gitea Actions: `github.server_url` does not include the sub-path (Gitea
|
|
limitation) — Caddy additionally forwards the git smart-HTTP endpoints of
|
|
the three skill orgs without the prefix so `actions/checkout` can clone.
|
|
- act_runner 0.6.1, host mode, label `windows`; started via
|
|
`Tools\gitea\act_runner\start-actrunner.ps1` (no autostart).
|