feat(knowledge): SE practitioner-QA layer scaffold + live progress hook
Dump stackexchange_20260331 (newest mirror), 6 sites, downloads running; streamed ingest/filter/tags pipeline; status writer + homepage strip show the layer's build progress every 5 minutes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PDKeXvpT6tENSvyQGLV1Uq
This commit is contained in:
skillfactor-pipeline
6
.gitignore
vendored
6
.gitignore
vendored
@@ -15,3 +15,9 @@ __pycache__/
|
|||||||
|
|
||||||
# Local scratch
|
# Local scratch
|
||||||
tmp/
|
tmp/
|
||||||
|
|
||||||
|
# SE knowledge layer bulk data
|
||||||
|
knowledge/data/raw/
|
||||||
|
knowledge/data/extracted/
|
||||||
|
knowledge/data/se.db*
|
||||||
|
|
||||||
|
|||||||
@@ -207,6 +207,7 @@
|
|||||||
<span>market-evidence crawl started <b style="color:#25253d;" data-st="started">…</b></span>
|
<span>market-evidence crawl started <b style="color:#25253d;" data-st="started">…</b></span>
|
||||||
<span>running for <b style="color:#25253d;" data-st="elapsed">…</b></span>
|
<span>running for <b style="color:#25253d;" data-st="elapsed">…</b></span>
|
||||||
<span><b style="color:#25253d;" data-st="ollama">…</b> Ollama extraction requests</span>
|
<span><b style="color:#25253d;" data-st="ollama">…</b> Ollama extraction requests</span>
|
||||||
|
<span id="sf-se-status" style="display:none;">knowledge layer (Stack Exchange): <b style="color:#25253d;" data-st="se_step">…</b> <span style="color:#8a8a99;" data-st="se_detail"></span></span>
|
||||||
<span style="color:#8a8a99;">status as of <span data-st="updated">…</span> · computed in <span data-st="ms">…</span> ms · auto-refreshes every 5 min</span>
|
<span style="color:#8a8a99;">status as of <span data-st="updated">…</span> · computed in <span data-st="ms">…</span> ms · auto-refreshes every 5 min</span>
|
||||||
</div>
|
</div>
|
||||||
<div style="margin-top:8px;font-size:13px;color:#8a8a99;line-height:1.5;">
|
<div style="margin-top:8px;font-size:13px;color:#8a8a99;line-height:1.5;">
|
||||||
@@ -447,6 +448,12 @@
|
|||||||
setSt("started", s.started_at);
|
setSt("started", s.started_at);
|
||||||
setSt("elapsed", s.elapsed_h + " h " + s.elapsed_m + " min");
|
setSt("elapsed", s.elapsed_h + " h " + s.elapsed_m + " min");
|
||||||
setSt("ollama", Number(s.ollama_requests || 0).toLocaleString("en-US"));
|
setSt("ollama", Number(s.ollama_requests || 0).toLocaleString("en-US"));
|
||||||
|
if (s.se_step) {
|
||||||
|
var seEl = document.getElementById("sf-se-status");
|
||||||
|
if (seEl) seEl.style.display = "";
|
||||||
|
setSt("se_step", s.se_step);
|
||||||
|
setSt("se_detail", s.se_detail ? s.se_detail.slice(0, 90) : "");
|
||||||
|
}
|
||||||
setSt("updated", s.updated_at);
|
setSt("updated", s.updated_at);
|
||||||
setSt("ms", s.compute_ms);
|
setSt("ms", s.compute_ms);
|
||||||
}).catch(function () {});
|
}).catch(function () {});
|
||||||
|
|||||||
56
knowledge/DECISIONS.md
Normal file
56
knowledge/DECISIONS.md
Normal file
@@ -0,0 +1,56 @@
|
|||||||
|
# DECISIONS — Stack-Exchange-Wissensschicht
|
||||||
|
|
||||||
|
Autonome Entscheidungen dieser Pipeline, chronologisch.
|
||||||
|
|
||||||
|
## D1 — Dump-Version
|
||||||
|
`stackexchange_20260331` (archive.org-Suche am 2026-07-09 ergab einen neueren
|
||||||
|
Mirror als den im Briefing genannten `stackexchange_20251231`). Dateien liegen
|
||||||
|
im Unterordner `stackexchange_20260331/`; alle 6 Ziel-Archive vorhanden,
|
||||||
|
zusammen ~929 MB (Meta-Sites bewusst NICHT geladen).
|
||||||
|
|
||||||
|
## D2 — Projektort & Repo
|
||||||
|
Als Unterprojekt `knowledge/` im bestehenden Pipeline-Repo `C:\dev\skillfactor`
|
||||||
|
(Branch `quality/reference-standard`), nicht als separates Repo: die
|
||||||
|
Homepage-Quellen (`gitea-custom/templates/home.tmpl`), stats.json-Erzeugung
|
||||||
|
und die Provenance-Maschinerie (p3d) leben hier — die SE-Integration greift
|
||||||
|
direkt in diese Bausteine.
|
||||||
|
|
||||||
|
## D3 — Referenz-Taxonomien
|
||||||
|
KEIN Neu-Download: ESCO v1.2.1 und O*NET 30.3 liegen bereits vollständig als
|
||||||
|
Tabellen im lokalen MSSQL (`skillfactor`-DB: esco_occupation, esco_skill,
|
||||||
|
esco_occ_skill, onet_occupation, onet_task, onet_software, …), geladen von den
|
||||||
|
offiziellen Quellen in einer früheren Pipeline-Phase. Wiederverwendung ist
|
||||||
|
idempotent und spart Zeit/Bandbreite; Versionen sind in den Paket-Manifests
|
||||||
|
dokumentiert.
|
||||||
|
|
||||||
|
## D4 — 7z-Entpackung
|
||||||
|
`py7zr` (pip) statt System-7-Zip: keine Binary-Abhängigkeit, selektive
|
||||||
|
Extraktion nur von `Posts.xml` + `Tags.xml` je Site.
|
||||||
|
|
||||||
|
## D5 — LLM-Arbeitsteilung (Kosten/Kontext)
|
||||||
|
- Tag→Taxonomie-Mapping: Claude klassifiziert die frequenzstärksten Tags
|
||||||
|
direkt in Batches (Sprachverständnis, Konfidenz + Rationale); der Long-Tail
|
||||||
|
(niedrige Frequenz nahe der 20er-Schwelle) läuft über das lokale
|
||||||
|
gemma3:27b mit Claude-designtem Prompt und wird stichprobengeprüft.
|
||||||
|
Grund: mehrere tausend Tags × Rationale sprengen das Session-Kontextbudget.
|
||||||
|
- Q&A-Zusammenfassungen (Schritt 4, ~10×150 Einträge): lokales gemma3
|
||||||
|
(etabliertes Muster der Evidence-Pipeline: Ollama extrahiert, Claude
|
||||||
|
designt Prompt + validiert + macht Stichproben-QA). Läuft detached, GPU
|
||||||
|
wird mit dem laufenden Evidence-Crawl geteilt.
|
||||||
|
|
||||||
|
## D6 — Homepage-Integration
|
||||||
|
Der Homepage-Code LIEGT im Arbeitsverzeichnis (Live-Template
|
||||||
|
`Tools\gitea\custom\templates\home.tmpl`, Spiegel in
|
||||||
|
`gitea-custom/templates/`). Integration erfolgt daher direkt (kein
|
||||||
|
`homepage/`-Snippet-Ordner nötig): Stack Exchange wird 6. Provenance-Kategorie
|
||||||
|
(`stackx`) in p3d/p4/stats.json; der Donut-/Legenden-Anteil bemisst sich
|
||||||
|
datenbasiert aus den Content-Items der kompilierten Q&A-Einträge (gleiche
|
||||||
|
Zählmethode wie alle anderen Quellen: count_items über die generierten
|
||||||
|
Dateien). Für die 10 Eval-Berufe wird das kompilierte Wissen zusätzlich als
|
||||||
|
`references/practitioner-qa.md` in die zugehörigen Skill-Pakete gelegt —
|
||||||
|
damit ist der Pie-Anteil echt gemessen, nicht angenommen.
|
||||||
|
|
||||||
|
## D7 — Live-Fortschritt (Nutzerwunsch 2026-07-09)
|
||||||
|
Die Pipeline schreibt `knowledge/data/progress-se.json` nach jedem Schritt;
|
||||||
|
der bestehende 5-Minuten-Status-Writer merged das in `assets/status.json`,
|
||||||
|
die Homepage-Statusleiste zeigt den SE-Layer-Stand neben dem Crawl-Zähler.
|
||||||
48
knowledge/README.md
Normal file
48
knowledge/README.md
Normal file
@@ -0,0 +1,48 @@
|
|||||||
|
# SkillFactor knowledge layer — Stack Exchange practitioner Q&A
|
||||||
|
|
||||||
|
Curated practitioner knowledge from the Stack Exchange data dumps
|
||||||
|
(CC-BY-SA 4.0), mapped onto ESCO skills / O*NET occupations and compiled
|
||||||
|
into per-profession knowledge files.
|
||||||
|
|
||||||
|
**Dump version:** `stackexchange_20260331` (archive.org community mirror —
|
||||||
|
newest at build time, see DECISIONS.md D1). Sites: workplace, pm, law,
|
||||||
|
money, softwareengineering, datascience (~929 MB compressed).
|
||||||
|
|
||||||
|
## Setup
|
||||||
|
|
||||||
|
```
|
||||||
|
# venv of the parent project is reused
|
||||||
|
..\.venv\Scripts\pip install py7zr
|
||||||
|
powershell -File scripts\download.ps1 # resumable, skips existing
|
||||||
|
```
|
||||||
|
|
||||||
|
## Pipeline (idempotent, in order)
|
||||||
|
|
||||||
|
```
|
||||||
|
python pipeline\se_pipeline.py ingest # 7z -> SQLite (streamed iterparse)
|
||||||
|
python pipeline\se_pipeline.py filter # quality gates -> qa_pairs
|
||||||
|
python pipeline\se_pipeline.py tags # tag frequencies >= 20 -> CSV
|
||||||
|
# -> LLM tag mapping produces mappings\tag_to_skill.csv (see DECISIONS D5)
|
||||||
|
python pipeline\se_pipeline.py compile # professions\<slug>\knowledge.md + vocabulary.json
|
||||||
|
python pipeline\se_pipeline.py verify # consistency checks (exit != 0 on failure)
|
||||||
|
```
|
||||||
|
|
||||||
|
Every step writes `data/progress-se.json`; the homepage status strip picks
|
||||||
|
it up within 5 minutes (see DECISIONS D7).
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
data/raw/ downloaded 7z archives (git-ignored)
|
||||||
|
data/extracted/ Posts/Tags/Users.xml per site (git-ignored)
|
||||||
|
data/se.db SQLite: questions/answers/users/qa_pairs (git-ignored)
|
||||||
|
mappings/ tag_to_skill.csv, unmapped.csv (versioned)
|
||||||
|
professions/ <slug>/knowledge.md, vocabulary.json (versioned)
|
||||||
|
```
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Source content: Stack Exchange Network, **CC-BY-SA 4.0** — see
|
||||||
|
`ATTRIBUTION.md`. Compiled knowledge files are derivative works and carry
|
||||||
|
per-entry attribution (question link, author, license); published derived
|
||||||
|
content must remain share-alike.
|
||||||
277
knowledge/pipeline/se_pipeline.py
Normal file
277
knowledge/pipeline/se_pipeline.py
Normal file
@@ -0,0 +1,277 @@
|
|||||||
|
"""SkillFactor knowledge layer — Stack Exchange practitioner Q&A.
|
||||||
|
|
||||||
|
Reproducible pipeline over the archive.org data-dump mirror
|
||||||
|
(stackexchange_20260331, CC-BY-SA 4.0). See knowledge/README.md.
|
||||||
|
|
||||||
|
CLI:
|
||||||
|
python knowledge/pipeline/se_pipeline.py ingest 7z -> SQLite (streamed)
|
||||||
|
python knowledge/pipeline/se_pipeline.py filter quality gates
|
||||||
|
python knowledge/pipeline/se_pipeline.py tags tag frequency export
|
||||||
|
python knowledge/pipeline/se_pipeline.py compile profession knowledge files
|
||||||
|
python knowledge/pipeline/se_pipeline.py verify consistency checks
|
||||||
|
|
||||||
|
Idempotent: existing archives are not re-downloaded (scripts/download.ps1),
|
||||||
|
extraction/ingest skip completed sites, summaries are cached per post id.
|
||||||
|
Every step updates data/progress-se.json for the homepage live status.
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import html
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sqlite3
|
||||||
|
import sys
|
||||||
|
from datetime import datetime
|
||||||
|
from xml.etree import ElementTree as ET
|
||||||
|
|
||||||
|
K = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) # knowledge/
|
||||||
|
RAW = os.path.join(K, "data", "raw")
|
||||||
|
EXTRACTED = os.path.join(K, "data", "extracted")
|
||||||
|
DB = os.path.join(K, "data", "se.db")
|
||||||
|
PROGRESS = os.path.join(K, "data", "progress-se.json")
|
||||||
|
|
||||||
|
DUMP_VERSION = "stackexchange_20260331"
|
||||||
|
DUMP_URL = "https://archive.org/download/stackexchange_20260331/"
|
||||||
|
|
||||||
|
SITES = {
|
||||||
|
"workplace": "workplace.stackexchange.com",
|
||||||
|
"pm": "pm.stackexchange.com",
|
||||||
|
"law": "law.stackexchange.com",
|
||||||
|
"money": "money.stackexchange.com",
|
||||||
|
"softwareengineering": "softwareengineering.stackexchange.com",
|
||||||
|
"datascience": "datascience.stackexchange.com",
|
||||||
|
}
|
||||||
|
|
||||||
|
MIN_Q_SCORE = 5
|
||||||
|
MIN_A_SCORE = 5
|
||||||
|
MIN_ANSWER_CHARS = 300
|
||||||
|
TAG_MIN_FREQ = 20
|
||||||
|
|
||||||
|
|
||||||
|
def progress(step, **info):
|
||||||
|
data = {}
|
||||||
|
if os.path.exists(PROGRESS):
|
||||||
|
try:
|
||||||
|
data = json.load(open(PROGRESS, encoding="utf-8"))
|
||||||
|
except ValueError:
|
||||||
|
data = {}
|
||||||
|
data["step"] = step
|
||||||
|
data["updated_at"] = datetime.now().isoformat(timespec="seconds")
|
||||||
|
data.setdefault("steps", {})[step] = {
|
||||||
|
"at": data["updated_at"], **info}
|
||||||
|
json.dump(data, open(PROGRESS, "w", encoding="utf-8"), indent=1)
|
||||||
|
|
||||||
|
|
||||||
|
def strip_html(s):
|
||||||
|
if not s:
|
||||||
|
return ""
|
||||||
|
s = re.sub(r"<(pre|code)[^>]*>(.*?)</\1>",
|
||||||
|
lambda m: "\n```\n" + m.group(2) + "\n```\n", s, flags=re.S)
|
||||||
|
s = re.sub(r"<br\s*/?>|</p>|</li>", "\n", s)
|
||||||
|
s = re.sub(r"<[^>]+>", "", s)
|
||||||
|
return html.unescape(s).strip()
|
||||||
|
|
||||||
|
|
||||||
|
def db():
|
||||||
|
cn = sqlite3.connect(DB)
|
||||||
|
cn.execute("PRAGMA journal_mode=WAL")
|
||||||
|
return cn
|
||||||
|
|
||||||
|
|
||||||
|
def ensure_schema(cn):
|
||||||
|
cn.executescript("""
|
||||||
|
CREATE TABLE IF NOT EXISTS questions(
|
||||||
|
site TEXT, id INTEGER, title TEXT, body TEXT, tags TEXT,
|
||||||
|
score INTEGER, accepted_answer_id INTEGER, view_count INTEGER,
|
||||||
|
closed INTEGER, owner_id INTEGER, PRIMARY KEY(site, id));
|
||||||
|
CREATE TABLE IF NOT EXISTS answers(
|
||||||
|
site TEXT, id INTEGER, parent_id INTEGER, body TEXT,
|
||||||
|
score INTEGER, owner_id INTEGER, PRIMARY KEY(site, id));
|
||||||
|
CREATE TABLE IF NOT EXISTS users(
|
||||||
|
site TEXT, id INTEGER, display_name TEXT, PRIMARY KEY(site, id));
|
||||||
|
CREATE TABLE IF NOT EXISTS ingest_log(
|
||||||
|
site TEXT PRIMARY KEY, questions INTEGER, answers INTEGER,
|
||||||
|
users INTEGER, at TEXT);
|
||||||
|
CREATE INDEX IF NOT EXISTS ix_answers_parent ON answers(site, parent_id);
|
||||||
|
""")
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_ingest():
|
||||||
|
import py7zr
|
||||||
|
cn = db()
|
||||||
|
ensure_schema(cn)
|
||||||
|
done = {r[0] for r in cn.execute("SELECT site FROM ingest_log")}
|
||||||
|
for key, host in SITES.items():
|
||||||
|
if key in done:
|
||||||
|
print(f"{key}: already ingested — skip")
|
||||||
|
continue
|
||||||
|
arc = os.path.join(RAW, f"{host}.7z")
|
||||||
|
if not os.path.exists(arc):
|
||||||
|
print(f"{key}: archive missing ({arc}) — skip")
|
||||||
|
continue
|
||||||
|
xdir = os.path.join(EXTRACTED, key)
|
||||||
|
os.makedirs(xdir, exist_ok=True)
|
||||||
|
if not os.path.exists(os.path.join(xdir, "Posts.xml")):
|
||||||
|
print(f"{key}: extracting Posts/Tags/Users ...")
|
||||||
|
with py7zr.SevenZipFile(arc) as z:
|
||||||
|
names = [n for n in z.getnames()
|
||||||
|
if n in ("Posts.xml", "Tags.xml", "Users.xml")]
|
||||||
|
z.extract(path=xdir, targets=names)
|
||||||
|
|
||||||
|
nq = na = nu = 0
|
||||||
|
qb, ab = [], []
|
||||||
|
for _ev, el in ET.iterparse(os.path.join(xdir, "Posts.xml")):
|
||||||
|
if el.tag != "row":
|
||||||
|
continue
|
||||||
|
pt = el.get("PostTypeId")
|
||||||
|
if pt == "1":
|
||||||
|
qb.append((key, int(el.get("Id")), el.get("Title", ""),
|
||||||
|
strip_html(el.get("Body", "")),
|
||||||
|
el.get("Tags", ""), int(el.get("Score", 0)),
|
||||||
|
int(el.get("AcceptedAnswerId", 0) or 0),
|
||||||
|
int(el.get("ViewCount", 0) or 0),
|
||||||
|
1 if el.get("ClosedDate") else 0,
|
||||||
|
int(el.get("OwnerUserId", 0) or 0)))
|
||||||
|
nq += 1
|
||||||
|
elif pt == "2":
|
||||||
|
ab.append((key, int(el.get("Id")),
|
||||||
|
int(el.get("ParentId", 0) or 0),
|
||||||
|
strip_html(el.get("Body", "")),
|
||||||
|
int(el.get("Score", 0)),
|
||||||
|
int(el.get("OwnerUserId", 0) or 0)))
|
||||||
|
na += 1
|
||||||
|
el.clear()
|
||||||
|
if len(qb) >= 2000:
|
||||||
|
cn.executemany("INSERT OR REPLACE INTO questions VALUES(?,?,?,?,?,?,?,?,?,?)", qb)
|
||||||
|
qb = []
|
||||||
|
if len(ab) >= 2000:
|
||||||
|
cn.executemany("INSERT OR REPLACE INTO answers VALUES(?,?,?,?,?,?)", ab)
|
||||||
|
ab = []
|
||||||
|
if qb:
|
||||||
|
cn.executemany("INSERT OR REPLACE INTO questions VALUES(?,?,?,?,?,?,?,?,?,?)", qb)
|
||||||
|
if ab:
|
||||||
|
cn.executemany("INSERT OR REPLACE INTO answers VALUES(?,?,?,?,?,?)", ab)
|
||||||
|
|
||||||
|
upath = os.path.join(xdir, "Users.xml")
|
||||||
|
if os.path.exists(upath):
|
||||||
|
ub = []
|
||||||
|
for _ev, el in ET.iterparse(upath):
|
||||||
|
if el.tag == "row":
|
||||||
|
ub.append((key, int(el.get("Id")),
|
||||||
|
el.get("DisplayName", "")))
|
||||||
|
nu += 1
|
||||||
|
el.clear()
|
||||||
|
if len(ub) >= 5000:
|
||||||
|
cn.executemany("INSERT OR REPLACE INTO users VALUES(?,?,?)", ub)
|
||||||
|
ub = []
|
||||||
|
if ub:
|
||||||
|
cn.executemany("INSERT OR REPLACE INTO users VALUES(?,?,?)", ub)
|
||||||
|
|
||||||
|
cn.execute("INSERT OR REPLACE INTO ingest_log VALUES(?,?,?,?,?)",
|
||||||
|
(key, nq, na, nu, datetime.now().isoformat()))
|
||||||
|
cn.commit()
|
||||||
|
print(f"{key}: {nq} questions, {na} answers, {nu} users")
|
||||||
|
progress("ingest", site=key, questions=nq, answers=na)
|
||||||
|
cn.close()
|
||||||
|
progress("ingest-done")
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_filter():
|
||||||
|
cn = db()
|
||||||
|
cn.executescript("""
|
||||||
|
DROP TABLE IF EXISTS qa_pairs;
|
||||||
|
CREATE TABLE qa_pairs(
|
||||||
|
site TEXT, q_id INTEGER, title TEXT, q_body TEXT, tags TEXT,
|
||||||
|
q_score INTEGER, view_count INTEGER,
|
||||||
|
a_id INTEGER, a_body TEXT, a_score INTEGER, a_accepted INTEGER,
|
||||||
|
q_owner INTEGER, a_owner INTEGER,
|
||||||
|
PRIMARY KEY(site, q_id, a_id));
|
||||||
|
""")
|
||||||
|
stats = {}
|
||||||
|
for key in SITES:
|
||||||
|
before = cn.execute(
|
||||||
|
"SELECT COUNT(*) FROM questions WHERE site=?", (key,)).fetchone()[0]
|
||||||
|
# accepted + top-scored answer per qualifying question
|
||||||
|
cn.execute("""
|
||||||
|
INSERT OR REPLACE INTO qa_pairs
|
||||||
|
SELECT q.site, q.id, q.title, q.body, q.tags, q.score, q.view_count,
|
||||||
|
a.id, a.body, a.score,
|
||||||
|
CASE WHEN a.id = q.accepted_answer_id THEN 1 ELSE 0 END,
|
||||||
|
q.owner_id, a.owner_id
|
||||||
|
FROM questions q
|
||||||
|
JOIN answers a ON a.site = q.site AND a.parent_id = q.id
|
||||||
|
WHERE q.site = ?
|
||||||
|
AND q.score >= ?
|
||||||
|
AND q.closed = 0
|
||||||
|
AND LENGTH(a.body) >= ?
|
||||||
|
AND (a.id = q.accepted_answer_id OR
|
||||||
|
a.id = (SELECT a2.id FROM answers a2
|
||||||
|
WHERE a2.site = q.site AND a2.parent_id = q.id
|
||||||
|
ORDER BY a2.score DESC, a2.id LIMIT 1))
|
||||||
|
AND EXISTS (SELECT 1 FROM answers ax
|
||||||
|
WHERE ax.site = q.site AND ax.parent_id = q.id
|
||||||
|
AND (ax.score >= ? OR ax.id = q.accepted_answer_id))
|
||||||
|
""", (key, MIN_Q_SCORE, MIN_ANSWER_CHARS, MIN_A_SCORE))
|
||||||
|
after_q = cn.execute(
|
||||||
|
"SELECT COUNT(DISTINCT q_id) FROM qa_pairs WHERE site=?",
|
||||||
|
(key,)).fetchone()[0]
|
||||||
|
after_p = cn.execute(
|
||||||
|
"SELECT COUNT(*) FROM qa_pairs WHERE site=?", (key,)).fetchone()[0]
|
||||||
|
stats[key] = {"questions_raw": before, "questions_kept": after_q,
|
||||||
|
"pairs_kept": after_p}
|
||||||
|
print(f"{key}: {before} raw questions -> {after_q} kept "
|
||||||
|
f"({after_p} Q&A pairs)")
|
||||||
|
cn.commit()
|
||||||
|
cn.close()
|
||||||
|
json.dump(stats, open(os.path.join(K, "data", "filter-stats.json"), "w",
|
||||||
|
encoding="utf-8"), indent=1)
|
||||||
|
progress("filter-done", **{k: v["questions_kept"] for k, v in stats.items()})
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_tags():
|
||||||
|
cn = db()
|
||||||
|
out = os.path.join(K, "data", "tags_freq.csv")
|
||||||
|
rows = []
|
||||||
|
for key in SITES:
|
||||||
|
freq = {}
|
||||||
|
for (tags,) in cn.execute(
|
||||||
|
"SELECT tags FROM qa_pairs WHERE site=?", (key,)):
|
||||||
|
for t in re.findall(r"[<|]([^><|]+)[>|]", tags or ""):
|
||||||
|
freq[t] = freq.get(t, 0) + 1
|
||||||
|
for t, n in sorted(freq.items(), key=lambda kv: -kv[1]):
|
||||||
|
if n >= TAG_MIN_FREQ:
|
||||||
|
rows.append((key, t, n))
|
||||||
|
with open(out, "w", encoding="utf-8", newline="\n") as f:
|
||||||
|
f.write("site,tag,count\n")
|
||||||
|
for r in rows:
|
||||||
|
f.write(f"{r[0]},{r[1]},{r[2]}\n")
|
||||||
|
print(f"{len(rows)} tags with freq >= {TAG_MIN_FREQ} -> {out}")
|
||||||
|
per_site = {}
|
||||||
|
for s, _t, _n in rows:
|
||||||
|
per_site[s] = per_site.get(s, 0) + 1
|
||||||
|
print(per_site)
|
||||||
|
progress("tags-done", tags_total=len(rows), **per_site)
|
||||||
|
cn.close()
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("cmd", choices=["ingest", "filter", "tags",
|
||||||
|
"compile", "verify"])
|
||||||
|
args = ap.parse_args()
|
||||||
|
if args.cmd == "ingest":
|
||||||
|
cmd_ingest()
|
||||||
|
elif args.cmd == "filter":
|
||||||
|
cmd_filter()
|
||||||
|
elif args.cmd == "tags":
|
||||||
|
cmd_tags()
|
||||||
|
elif args.cmd == "compile":
|
||||||
|
from se_compile import cmd_compile
|
||||||
|
cmd_compile()
|
||||||
|
elif args.cmd == "verify":
|
||||||
|
from se_verify import cmd_verify
|
||||||
|
cmd_verify()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
22
knowledge/scripts/download.ps1
Normal file
22
knowledge/scripts/download.ps1
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
# Lädt die 6 Stack-Exchange-Site-Archive vom archive.org-Mirror (resumable).
|
||||||
|
# Version: stackexchange_20260331 (neuester Mirror, geprüft 2026-07-09).
|
||||||
|
$base = "https://archive.org/download/stackexchange_20260331/stackexchange_20260331"
|
||||||
|
$dest = "C:\dev\skillfactor\knowledge\data\raw"
|
||||||
|
New-Item -ItemType Directory -Force $dest | Out-Null
|
||||||
|
$files = @(
|
||||||
|
"workplace.stackexchange.com.7z",
|
||||||
|
"pm.stackexchange.com.7z",
|
||||||
|
"law.stackexchange.com.7z",
|
||||||
|
"money.stackexchange.com.7z",
|
||||||
|
"softwareengineering.stackexchange.com.7z",
|
||||||
|
"datascience.stackexchange.com.7z"
|
||||||
|
)
|
||||||
|
$log = "C:\dev\skillfactor\docs\logs\se_download.log"
|
||||||
|
foreach ($f in $files) {
|
||||||
|
"$(Get-Date -Format 'HH:mm:ss') start $f" | Out-File $log -Append -Encoding utf8
|
||||||
|
# curl.exe: resumable (-C -), folgt Redirects (-L), still (-sS)
|
||||||
|
& curl.exe -L -sS -C - -o "$dest\$f" "$base/$f" 2>&1 | Out-File $log -Append -Encoding utf8
|
||||||
|
$size = [Math]::Round((Get-Item "$dest\$f").Length / 1MB)
|
||||||
|
"$(Get-Date -Format 'HH:mm:ss') done $f ($size MB)" | Out-File $log -Append -Encoding utf8
|
||||||
|
}
|
||||||
|
"$(Get-Date -Format 'HH:mm:ss') ALL DOWNLOADS COMPLETE" | Out-File $log -Append -Encoding utf8
|
||||||
@@ -54,6 +54,12 @@ while ($true) {
|
|||||||
$startDt = [datetime]::ParseExact($started, "yyyy-MM-dd HH:mm", $null)
|
$startDt = [datetime]::ParseExact($started, "yyyy-MM-dd HH:mm", $null)
|
||||||
$elapsed = (Get-Date) - $startDt
|
$elapsed = (Get-Date) - $startDt
|
||||||
$sw.Stop()
|
$sw.Stop()
|
||||||
|
# Stack-Exchange-Wissensschicht: Fortschritt der knowledge-Pipeline
|
||||||
|
$se = $null
|
||||||
|
$seFile = "$dev\knowledge\data\progress-se.json"
|
||||||
|
if (Test-Path $seFile) {
|
||||||
|
try { $se = Get-Content $seFile -Raw -Encoding utf8 | ConvertFrom-Json } catch {}
|
||||||
|
}
|
||||||
$obj = [ordered]@{
|
$obj = [ordered]@{
|
||||||
done = $done
|
done = $done
|
||||||
total = $total
|
total = $total
|
||||||
@@ -62,6 +68,11 @@ while ($true) {
|
|||||||
elapsed_h = [Math]::Floor($elapsed.TotalHours)
|
elapsed_h = [Math]::Floor($elapsed.TotalHours)
|
||||||
elapsed_m = $elapsed.Minutes
|
elapsed_m = $elapsed.Minutes
|
||||||
ollama_requests = $ollama
|
ollama_requests = $ollama
|
||||||
|
se_step = $(if ($se) { $se.step } else { $null })
|
||||||
|
se_updated = $(if ($se) { $se.updated_at } else { $null })
|
||||||
|
se_detail = $(if ($se -and $se.steps -and $se.step -and $se.steps.($se.step)) {
|
||||||
|
($se.steps.($se.step) | ConvertTo-Json -Compress)
|
||||||
|
} else { $null })
|
||||||
updated_at = (Get-Date -Format "yyyy-MM-dd HH:mm:ss")
|
updated_at = (Get-Date -Format "yyyy-MM-dd HH:mm:ss")
|
||||||
compute_ms = $sw.ElapsedMilliseconds
|
compute_ms = $sw.ElapsedMilliseconds
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user