Files

skillfactor-pipeline 7850baf6bb feat(knowledge): SE practitioner-QA layer scaffold + live progress hook

Dump stackexchange_20260331 (newest mirror), 6 sites, downloads running;
streamed ingest/filter/tags pipeline; status writer + homepage strip show
the layer's build progress every 5 minutes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDKeXvpT6tENSvyQGLV1Uq

2026-07-09 22:17:26 +02:00

1.8 KiB

Raw Blame History

SkillFactor knowledge layer — Stack Exchange practitioner Q&A

Curated practitioner knowledge from the Stack Exchange data dumps (CC-BY-SA 4.0), mapped onto ESCO skills / O*NET occupations and compiled into per-profession knowledge files.

Dump version: stackexchange_20260331 (archive.org community mirror — newest at build time, see DECISIONS.md D1). Sites: workplace, pm, law, money, softwareengineering, datascience (~929 MB compressed).

Setup

# venv of the parent project is reused
..\.venv\Scripts\pip install py7zr
powershell -File scripts\download.ps1        # resumable, skips existing

Pipeline (idempotent, in order)

python pipeline\se_pipeline.py ingest    # 7z -> SQLite (streamed iterparse)
python pipeline\se_pipeline.py filter    # quality gates -> qa_pairs
python pipeline\se_pipeline.py tags      # tag frequencies >= 20 -> CSV
#   -> LLM tag mapping produces mappings\tag_to_skill.csv (see DECISIONS D5)
python pipeline\se_pipeline.py compile   # professions\<slug>\knowledge.md + vocabulary.json
python pipeline\se_pipeline.py verify    # consistency checks (exit != 0 on failure)

Every step writes data/progress-se.json; the homepage status strip picks it up within 5 minutes (see DECISIONS D7).

Layout

data/raw/          downloaded 7z archives            (git-ignored)
data/extracted/    Posts/Tags/Users.xml per site     (git-ignored)
data/se.db         SQLite: questions/answers/users/qa_pairs (git-ignored)
mappings/          tag_to_skill.csv, unmapped.csv    (versioned)
professions/       <slug>/knowledge.md, vocabulary.json (versioned)

License

Source content: Stack Exchange Network, CC-BY-SA 4.0 — see ATTRIBUTION.md. Compiled knowledge files are derivative works and carry per-entry attribution (question link, author, license); published derived content must remain share-alike.

1.8 KiB Raw Blame History