skills-core/data-engineer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: data-engineer skill package v0.1.0

Browse Source
This commit is contained in:
skillfactor-pipeline
2026-08-14 17:28:40 +02:00
commit 85d470904c
9 changed files with 703 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

166
references/ai-skills.md Normal file
View File

@@ -0,0 +1,166 @@
# External AI agent skills — data-engineer
Proven, publicly available AI agent skills mapped to this occupation.
Nothing is copied from the sources: every entry is a name, a one-line
summary and a link to the upstream skill package. Each section names
its source repository, commit, license and retrieval date.
**Tiers:** `core` = the skill directly exercises a top market hard
skill, tool or method (from gated job-ad evidence) or an essential
ESCO competence of this occupation; `adjacent` =
plausibly useful, secondary. Entries are capped at 12 per source
and 80 in total per occupation (core first,
strongest matches survive); everything beyond the caps is excluded
and logged in the pipeline audit trail, not in this package.
_Matched deterministically (ISCO group + title/competence keywords,
tiered against market evidence + ESCO essentials) by
`pipeline/p5_enrich_ai_skills.py` on 2026-07-14._
## Source: anthropics/skills
- Repository: [https://github.com/anthropics/skills](https://github.com/anthropics/skills) (commit `f6656c1`, retrieved 2026-07-14)
- License: Apache-2.0; the document skills (docx/pdf/pptx/xlsx) are source-available — see the LICENSE.txt in the upstream skill folder
| Skill | Tier | What it adds | Upstream |
|---|---|---|---|
| `mcp-builder` | core | Guide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python … | [source](https://github.com/anthropics/skills/tree/main/skills/mcp-builder) |
| `docx` | adjacent | Use this skill whenever the user wants to create, read, edit, or manipulate Word documents (.docx files) or Word templates (.dotx files). Triggers include: any mention of 'Word doc', 'word document', '.docx', '.dotx', or requests to … | [source](https://github.com/anthropics/skills/tree/main/skills/docx) |
| `skill-creator` | adjacent | Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with … | [source](https://github.com/anthropics/skills/tree/main/skills/skill-creator) |
| `webapp-testing` | adjacent | Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs. | [source](https://github.com/anthropics/skills/tree/main/skills/webapp-testing) |
| `claude-api` | adjacent | Reference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration. TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a … | [source](https://github.com/anthropics/skills/tree/main/skills/claude-api) |
| `pdf` | adjacent | Use this skill whenever the user wants to do anything with PDF files. This includes reading or extracting text/tables from PDFs, combining or merging multiple PDFs into one, splitting PDFs apart, rotating pages, adding watermarks, creating … | [source](https://github.com/anthropics/skills/tree/main/skills/pdf) |
## Source: obra/superpowers
- Repository: [https://github.com/obra/superpowers](https://github.com/obra/superpowers) (commit `d884ae0`, retrieved 2026-07-14)
- License: MIT (c) Jesse Vincent
| Skill | Tier | What it adds | Upstream |
|---|---|---|---|
| `using-git-worktrees` | core | Use when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallback | [source](https://github.com/obra/superpowers/tree/main/skills/using-git-worktrees) |
| `brainstorming` | adjacent | You MUST use this before any creative work - creating features, building components, adding functionality, or modifying behavior. Explores user intent, requirements and design before implementation. | [source](https://github.com/obra/superpowers/tree/main/skills/brainstorming) |
| `finishing-a-development-branch` | adjacent | Use when implementation is complete, all tests pass, and you need to decide how to integrate the work - guides completion of development work by presenting structured options for merge, PR, or cleanup | [source](https://github.com/obra/superpowers/tree/main/skills/finishing-a-development-branch) |
| `subagent-driven-development` | adjacent | Use when executing implementation plans with independent tasks in the current session | [source](https://github.com/obra/superpowers/tree/main/skills/subagent-driven-development) |
| `test-driven-development` | adjacent | Use when implementing any feature or bugfix, before writing implementation code | [source](https://github.com/obra/superpowers/tree/main/skills/test-driven-development) |
| `dispatching-parallel-agents` | adjacent | Use when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | [source](https://github.com/obra/superpowers/tree/main/skills/dispatching-parallel-agents) |
| `executing-plans` | adjacent | Use when you have a written implementation plan to execute in a separate session with review checkpoints | [source](https://github.com/obra/superpowers/tree/main/skills/executing-plans) |
| `receiving-code-review` | adjacent | Use when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementation | [source](https://github.com/obra/superpowers/tree/main/skills/receiving-code-review) |
| `requesting-code-review` | adjacent | Use when completing tasks, implementing major features, or before merging to verify work meets requirements | [source](https://github.com/obra/superpowers/tree/main/skills/requesting-code-review) |
| `systematic-debugging` | adjacent | Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes | [source](https://github.com/obra/superpowers/tree/main/skills/systematic-debugging) |
| `verification-before-completion` | adjacent | Use when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always | [source](https://github.com/obra/superpowers/tree/main/skills/verification-before-completion) |
| `writing-plans` | adjacent | Use when you have a spec or requirements for a multi-step task, before touching code | [source](https://github.com/obra/superpowers/tree/main/skills/writing-plans) |
## Source: wshobson/agents
- Repository: [https://github.com/wshobson/agents](https://github.com/wshobson/agents) (commit `6fd3247`, retrieved 2026-07-14)
- License: MIT (c) Seth Hobson
| Skill | Tier | What it adds | Upstream |
|---|---|---|---|
| `langchain-architecture` | core | Design LLM applications using LangChain 1.x and LangGraph for agents, memory, and tool integration. Use when building LangChain applications, implementing AI agents, or creating complex LLM workflows. | [source](https://github.com/wshobson/agents/tree/main/plugins/llm-application-dev/skills/langchain-architecture) |
| `python-background-jobs` | core | Python background job patterns including task queues, workers, and event-driven architecture. Use when implementing async task processing, job queues, long-running operations, or decoupling work from request/response cycles. | [source](https://github.com/wshobson/agents/tree/main/plugins/python-development/skills/python-background-jobs) |
| `python-project-structure` | core | Python project organization, module architecture, and public API design. Use when setting up new projects, organizing modules, defining public interfaces with __all__, or planning directory layouts. | [source](https://github.com/wshobson/agents/tree/main/plugins/python-development/skills/python-project-structure) |
| `spark-optimization` | core | Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines. | [source](https://github.com/wshobson/agents/tree/main/plugins/data-engineering/skills/spark-optimization) |
| `code-documentation-docs-architect (agent)` | core | Creates comprehensive technical documentation from existing codebases. Analyzes architecture, design patterns, and implementation details to produce long-form technical manuals and ebooks. Use PROACTIVELY for system documentation, … | [source](https://github.com/wshobson/agents/tree/main/plugins/code-documentation/agents/docs-architect.md) |
| `recsys-pipeline-architect` | core | Design composable recommendation, ranking, and feed pipelines using the six-stage Source→Hydrator→Filter→Scorer→Selector→SideEffect framework popularized by xAI's open-sourced X For You algorithm. Use when building any system that picks … | [source](https://github.com/wshobson/agents/tree/main/plugins/machine-learning-ops/skills/recsys-pipeline-architect) |
| `sql-optimization-patterns` | core | Master SQL query optimization, indexing strategies, and EXPLAIN analysis to dramatically improve database performance and eliminate slow queries. Use when debugging slow queries, designing database schemas, or optimizing application … | [source](https://github.com/wshobson/agents/tree/main/plugins/developer-essentials/skills/sql-optimization-patterns) |
| `uv-package-manager` | core | Master the uv package manager for fast Python dependency management, virtual environments, and modern Python project workflows. Use when setting up Python projects, managing dependencies, or optimizing Python development workflows with uv. | [source](https://github.com/wshobson/agents/tree/main/plugins/python-development/skills/uv-package-manager) |
| `auth-implementation-patterns` | core | Master authentication and authorization patterns including JWT, OAuth2, session management, and RBAC to build secure, scalable access control systems. Use when implementing auth systems, securing APIs, or debugging security issues. | [source](https://github.com/wshobson/agents/tree/main/plugins/developer-essentials/skills/auth-implementation-patterns) |
| `comprehensive-review-architect-review (agent)` | core | Master software architect specializing in modern architecture patterns, clean architecture, microservices, event-driven systems, and DDD. Reviews system designs and code changes for architectural integrity, scalability, and … | [source](https://github.com/wshobson/agents/tree/main/plugins/comprehensive-review/agents/architect-review.md) |
| `python-testing-patterns` | core | Implement comprehensive testing strategies with pytest, fixtures, mocking, and test-driven development. Use when writing Python tests, setting up test suites, or implementing testing best practices. | [source](https://github.com/wshobson/agents/tree/main/plugins/python-development/skills/python-testing-patterns) |
| `async-python-patterns` | core | Master Python asyncio, concurrent programming, and async/await patterns for high-performance applications. Use when building async APIs, concurrent systems, or I/O-bound applications requiring non-blocking operations. | [source](https://github.com/wshobson/agents/tree/main/plugins/python-development/skills/async-python-patterns) |
## Source: google/skills
- Repository: [https://github.com/google/skills](https://github.com/google/skills) (commit `b15f327`, retrieved 2026-07-14)
- License: Apache-2.0
| Skill | Tier | What it adds | Upstream |
|---|---|---|---|
| `bigquery-basics` | core | Manages datasets, tables, and jobs in BigQuery. Use when you need to interact with BigQuery, run SQL queries, manage BigQuery resources (datasets, tables, views), or perform basic data ingestion and analysis. | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/bigquery-basics) |
| `cloud-sql-basics` | core | This file generates or explains Cloud SQL resources. Use this file when the user asks to create a Cloud SQL instance or database for MySQL, PostgreSQL, or SQL Server. Cloud SQL manages third-party MySQL, PostgreSQL, and SQL Server … | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/cloud-sql-basics) |
| `agent-platform-eval-flywheel` | core | Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology. Use when evaluating an agent or model, building an eval dataset, picking or writing evaluation metrics, analyzing … | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/agent-platform-eval-flywheel) |
| `alloydb-basics` | core | Manages clusters, instances, and backups for AlloyDB for PostgreSQL, and integrates with AlloyDB model context protocol (MCP) tools for automated database operations. | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/alloydb-basics) |
| `bigquery-ai-ml` | core | Leverages BigQuery's built-in machine learning and GenAI capabilities for advanced data analytics. Use when you need to write SQL queries that perform time-series forecasting, detect outliers, find key drivers, or leverage generative AI … | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/bigquery-ai-ml) |
| `datalineage-bigquery-asset-impact-analysis` | core | Analyzes the downstream impact (blast radius) when a BigQuery table or view is broken, stale, or modified. Identifies all downstream tables, dashboards, and processes that will be affected. Use when: - Performing a blast radius or impact … | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/datalineage-bigquery-asset-impact-analysis) |
| `google-analytics-admin-api-basics` | core | Manages Google Analytics account and property settings, enables the Analytics Admin API via the Cloud CLI, lists accounts and properties, and manages data streams, custom dimensions, conversion events, and integrations. Use when you need … | [source](https://github.com/google/skills/tree/b15f327/skills/analytics/google-analytics-admin-api-basics) |
| `google-analytics-data-api-basics` | core | Manages Google Analytics reporting data, enables the Analytics Data API via the Cloud CLI, and creates reports using the Google Analytics Data API (v1beta). Use when you need to interact with Google Analytics properties, run customized … | [source](https://github.com/google/skills/tree/b15f327/skills/analytics/google-analytics-data-api-basics) |
| `bigtable-basics` | core | Assists in provisioning instances/tables, designing performant schemas, and querying data in Bigtable. Use when designing Bigtable row keys, configuring column families, writing SQL queries or client library code (Java, Go, Python) for … | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/bigtable-basics) |
| `agent-platform-alert-configuration` | adjacent | Configures best-practice alerting policies for Google Cloud Vertex AI / Agent Platform agents on Agent Runtime. Use when analyzing, writing, or deploying alerting policies to monitor agent latency, error rates, and quality metrics … | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/agent-platform-alert-configuration) |
| `gke-observability` | adjacent | Configures GKE observability, including Cloud Logging, Cloud Monitoring, and managed Prometheus. Use when configuring GKE monitoring, setting up GKE logging, or configuring Prometheus metrics collection. Don't use to configure local … | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/gke-observability) |
| `google-cloud-networking-observability` | adjacent | Investigates Google Cloud networking issues by analyzing logs, metrics, and diagnostics. Use when investigating VPC Flow Logs (including cost estimation), NAT, firewall, or threat logs, querying latency and throughput metrics, or running … | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/google-cloud-networking-observability) |
## Source: czlonkowski/n8n-skills
- Repository: [https://github.com/czlonkowski/n8n-skills](https://github.com/czlonkowski/n8n-skills) (commit `9ea3aa5`, retrieved 2026-07-14)
- License: MIT
| Skill | Tier | What it adds | Upstream |
|---|---|---|---|
| `n8n-workflow-patterns` | core | Proven workflow architectural patterns from real n8n workflows. Use when building new workflows, designing workflow structure, choosing workflow patterns, planning workflow architecture, or asking about webhook processing, HTTP API … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-workflow-patterns) |
| `n8n-code-python` | core | Write Python code in n8n Code nodes. Use when writing Python in n8n, using _input/_json/_node syntax, working with standard library, or need to understand Python limitations in n8n Code nodes. Use this skill when the user specifically … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-code-python) |
| `n8n-code-tool` | core | Write JavaScript or Python for the n8n Custom Code Tool (@n8n/n8n-nodes-langchain.toolCode) — the AI-agent-callable tool, NOT the workflow Code node. Use when building a Code Tool attached to an AI Agent, writing code that an LLM will … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-code-tool) |
| `n8n-mcp-tools-expert` | core | Expert guide for using n8n-mcp MCP tools effectively. Use when searching for nodes, validating configurations, accessing templates, managing workflows, managing credentials, auditing instance security, or using any n8n-mcp tool. Provides … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-mcp-tools-expert) |
| `n8n-agents` | adjacent | Design n8n AI agents the right way. Use when building or editing any @n8n/n8n-nodes-langchain.* AI node — an AI Agent, LLM chain, Text Classifier, or Information Extractor — and whenever the user mentions AI agents, LLM with tools, tool … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-agents) |
| `n8n-binary-and-data` | adjacent | Handle files and binary data in n8n correctly. Use when working with files, images, PDFs, attachments, uploads or downloads, base64, vision/multimodal input, or when an AI agent needs a file as tool input or output — and whenever the user … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-binary-and-data) |
| `n8n-code-javascript` | adjacent | Write JavaScript code in n8n Code nodes. Use when writing JavaScript in n8n, using $input/$json/$node syntax, making HTTP requests with this.helpers / the $helpers global, working with dates using DateTime, troubleshooting Code node … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-code-javascript) |
| `n8n-error-handling` | adjacent | Wire n8n error handling so failures are loud, structured, and recoverable. Use when building any webhook/API workflow, a scheduled or unattended workflow, or any path where a silent failure would drop user-visible work — and whenever the … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-error-handling) |
| `n8n-expression-syntax` | adjacent | Validate n8n expression syntax and fix common errors. Use when writing n8n expressions, using {{}} syntax, accessing $json/$node variables, troubleshooting expression errors, mapping data between nodes, or referencing webhook data in … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-expression-syntax) |
| `n8n-multi-instance` | adjacent | Use when an n8n-mcp account targets more than one n8n instance — i.e. the `n8n_instances` tool is available, the user mentions multiple n8n instances or environments (prod vs staging, several teams or clients), a workflow / datatable / … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-multi-instance) |
| `n8n-node-configuration` | adjacent | Operation-aware node configuration guidance. Use when configuring nodes, understanding property dependencies, determining required fields, choosing between get_node detail levels, or learning common configuration patterns by node type. … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-node-configuration) |
| `n8n-self-hosting` | adjacent | Deploy a production self-hosted n8n end-to-end to a fresh Linux VM over SSH, using Docker Compose behind a Caddy reverse proxy with automatic HTTPS. Use whenever the user wants to self-host, install, set up, provision, or deploy n8n on … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-self-hosting) |
## Source: NVIDIA/skills
- Repository: [https://github.com/NVIDIA/skills](https://github.com/NVIDIA/skills) (commit `153b14b`, retrieved 2026-07-14)
- License: CC-BY-4.0 (skills/docs), Apache-2.0 (code)
| Skill | Tier | What it adds | Upstream |
|---|---|---|---|
| `deepstream-dev` | core | NVIDIA DeepStream SDK 9.0 development with Python pyservicemaker API. Use when building video analytics pipelines, GStreamer-based video processing, TensorRT inference integration, object detection/tracking, or Kafka/message broker … | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/deepstream-dev) |
| `vss-manage-video-io-storage` | core | Use to call the VIOS REST API (sensor list, timelines, clip extraction, snapshots, add/delete sensors and streams). Not for VLM inference or search. | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/vss-manage-video-io-storage) |
| `vss-manage-alerts` | core | Use for VSS alert workflows — real-time monitoring, Alert-Bridge subscriptions, Slack notifications, incident queries, camera onboarding. Not for non-alert analytics. | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/vss-manage-alerts) |
| `nemo-automodel-model-onboarding` | core | Guide for onboarding new model architectures into NeMo AutoModel, including architecture discovery, implementation patterns, registration, and validation. | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/nemo-automodel-model-onboarding) |
| `nemo-rl-session-memory` | core | Manage durable working-session memory for coding agents. Use when a user asks to preserve or recover agent context across disconnects, VS Code restarts, long-running work, handoffs, or any session where important state should be written … | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/nemo-rl-session-memory) |
| `rag-blueprint` | core | NVIDIA RAG Blueprint — deploy, configure, troubleshoot, and manage. Handles any RAG action: deploy, install, start, enable, disable, toggle, change, configure, troubleshoot, debug, fix, shutdown, stop, or tear down any RAG feature or … | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/rag-blueprint) |
| `tao-train-segformer` | core | SegFormer for semantic segmentation. Lightweight transformer-based architecture with hierarchical feature extraction, efficient for real-time segmentation tasks. Use when training, evaluating, exporting, quantizing, or running inference … | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/tao-train-segformer) |
| `dali-dynamic-mode` | core | DALI imperative dynamic mode (`nvidia.dali.experimental.dynamic`, ndd): use when working on ndd code or migrating pipelines; skip pipeline-only tasks. | [source](https://github.com/NVIDIA/skills/tree/153b14b/plugins/nvidia-skills/skills/dali-dynamic-mode) |
| `deepstream-generate-pipeline` | core | Build DeepStream GStreamer pipelines interactively. Use when the user asks about pipelines for video/image inference, detection, tracking, or streaming — including natural phrases like 'pipeline to infer on image', 'run inference on … | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/deepstream-generate-pipeline) |
| `hsb-app` | core | Discover and run Holoscan Sensor Bridge example applications on a connected devkit. Filters available apps by the user's platform, HSB software version, board type, and sensors. Supports timed execution, failure analysis, code-edit … | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/hsb-app) |
| `tao-train-oneformer` | core | OneFormer for universal image segmentation. Unifies panoptic, instance, and semantic segmentation with a single architecture using task-conditioned queries. Use when training, evaluating, exporting, quantizing, or running inference for a … | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/tao-train-oneformer) |
| `vss-generate-video-report` | core | Use this skill when producing a VSS analysis report — Mode A per-clip VLM, Mode B incident-range via video-analytics. Not for standalone video summarization, real-time alerts or ad-hoc Q&A. | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/vss-generate-video-report) |
## Source: phuryn/pm-skills
- Repository: [https://github.com/phuryn/pm-skills](https://github.com/phuryn/pm-skills) (commit `18468a9`, retrieved 2026-07-14)
- License: MIT
| Skill | Tier | What it adds | Upstream |
|---|---|---|---|
| `dummy-dataset` | core | Generate realistic dummy datasets for testing with customizable columns, constraints, and output formats (CSV, JSON, SQL, Python script). Use when creating test data, building mock datasets, or generating sample data for development and … | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-execution/skills/dummy-dataset) |
| `sql-queries` | core | Generate SQL queries from natural language descriptions. Supports BigQuery, PostgreSQL, MySQL, and other dialects. Reads database schemas from uploaded diagrams or documentation. Use when writing SQL, building data reports, exploring … | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-data-analytics/skills/sql-queries) |
| `metrics-dashboard` | core | Define and design a product metrics dashboard with key metrics, data sources, visualization types, and alert thresholds. Use when creating a metrics dashboard, defining KPIs, setting up product analytics, or building a data monitoring plan. | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-product-discovery/skills/metrics-dashboard) |
| `ab-test-analysis` | core | Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split … | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-data-analytics/skills/ab-test-analysis) |
| `cohort-analysis` | core | Perform cohort analysis on user engagement data — retention curves, feature adoption trends, and segment-level insights. Use when analyzing user retention by cohort, studying feature adoption over time, investigating churn patterns, or … | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-data-analytics/skills/cohort-analysis) |
| `lean-canvas` | core | Generate a Lean Canvas with problem, solution, metrics, cost structure, UVP, unfair advantage, channels, segments, and revenue. Use when exploring a lean startup canvas, testing a business hypothesis, or modeling a new venture. | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-product-strategy/skills/lean-canvas) |
| `gtm-strategy` | adjacent | Create a go-to-market strategy covering marketing channels, messaging, success metrics, and launch timeline. Use when planning a product launch, creating a GTM plan from scratch, or defining a launch strategy for a new market. | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-go-to-market/skills/gtm-strategy) |
| `product-strategy` | adjacent | Create a comprehensive product strategy using the 9-section Product Strategy Canvas — vision, segments, costs, value propositions, trade-offs, metrics, growth, capabilities, and defensibility. Use when building a product strategy, creating … | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-product-strategy/skills/product-strategy) |
| `north-star-metric` | adjacent | Define a North Star Metric and 3-5 supporting input metrics that form a metrics constellation. Classify the business game (Attention, Transaction, Productivity) and validate against 7 criteria for an effective North Star. Use when choosing … | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-marketing-growth/skills/north-star-metric) |
## Source: veniceai/skills
- Repository: [https://github.com/veniceai/skills](https://github.com/veniceai/skills) (commit `de089fa`, retrieved 2026-07-14)
- License: MIT
| Skill | Tier | What it adds | Upstream |
|---|---|---|---|
| `venice-api-keys` | core | Manage Venice API keys. Covers GET/POST/PATCH/DELETE /api_keys, GET /api_keys/{id}, GET /api_keys/rate_limits, GET /api_keys/rate_limits/log, the two-step /api_keys/generate_web3_key wallet flow, INFERENCE vs ADMIN key types, and per-key … | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-api-keys) |
| `venice-api-overview` | core | High-level map of the Venice.ai API - base URL, authentication modes, endpoint categories, response headers, pricing model, error shape, and versioning. Load this first when starting any Venice integration. | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-api-overview) |
| `venice-auth` | adjacent | Authenticate to the Venice API with a Bearer API key or with an x402 / SIWE wallet. Covers header formats, the SIWE message fields, TTL and nonce rules, the venice-x402-client SDK, and how to choose between the two modes. | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-auth) |
| `venice-characters` | adjacent | Discover and use Venice public characters (persona-driven system prompts with a bound model). Covers GET /characters (search/filter/sort), /characters/{slug}, /characters/{slug}/reviews, the Character schema, and how to apply a character … | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-characters) |
| `venice-embeddings` | adjacent | Call POST /embeddings on Venice. Covers request shape (input, model, encoding_format, dimensions, user), OpenAI compatibility, response compression (gzip/br), and practical usage for retrieval, clustering, and RAG. | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-embeddings) |
| `venice-errors` | adjacent | Handle Venice API errors correctly. Covers the StandardError / DetailedError / ContentViolationError / X402InferencePaymentRequired body shapes, every meaningful status code (400, 401, 402, 403, 415, 422, 429, 500, 503, 504), the 402 … | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-errors) |
| `venice-image-edit` | adjacent | Transform existing images with Venice. Covers POST /image/edit (prompt-driven single-image edit), /image/multi-edit (compose 1-3 images), /image/upscale (2-4x upscale + enhance), and /image/background-remove. Accepts base64, file upload, … | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-image-edit) |
| `venice-image-generate` | adjacent | Generate images with Venice. Covers POST /image/generate (Venice-native), POST /images/generations (OpenAI-compatible), GET /image/styles (style presets), request fields (prompt, dimensions, cfg_scale, seed, variants, style_preset, … | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-image-generate) |
| `venice-responses` | adjacent | Use Venice's Alpha POST /responses endpoint - an OpenAI-compatible Responses API with typed output blocks (reasoning, message, function_call, web_search_call). Covers request shape, streaming, differences from /chat/completions, supported … | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-responses) |

100
references/market.md Normal file
View File

@@ -0,0 +1,100 @@
# Market evidence report — data-engineer
Source: **22 real job ads** (JSearch API, countries: us 22), extracted into the MSSQL evidence store; as of 2026-07-11.
This report contains extracted, aggregated facts only — no ad text is
reproduced (copyright / platform terms).
## Seniority distribution
| Seniority | Ads | Share |
|---|---|---|
| mid | 14 | 64 % |
| senior | 6 | 27 % |
| lead | 1 | 5 % |
| n/a | 1 | 5 % |
## Tools — full market ranking
| # | Item | Ads | Share |
|---|---|---|---|
| 1 | Databricks | 9 | 41 % |
| 2 | Python | 9 | 41 % |
| 3 | Apache Spark | 5 | 23 % |
| 4 | AWS | 5 | 23 % |
| 5 | Spark | 5 | 23 % |
| 6 | SQL | 5 | 23 % |
| 7 | informatica | 4 | 18 % |
| 8 | Snowflake | 4 | 18 % |
| 9 | Azure Data Factory | 3 | 14 % |
## Hard skills — full market ranking
| # | Item | Ads | Share |
|---|---|---|---|
| 1 | data modeling | 11 | 50 % |
| 2 | data integration | 6 | 27 % |
| 3 | data pipeline design | 6 | 27 % |
| 4 | data analysis | 5 | 23 % |
| 5 | data engineering | 5 | 23 % |
| 6 | data transformation | 5 | 23 % |
| 7 | data warehousing | 4 | 18 % |
| 8 | statistical analysis | 4 | 18 % |
| 9 | data architecture | 3 | 14 % |
| 10 | database design | 3 | 14 % |
## Methods — full market ranking
| # | Item | Ads | Share |
|---|---|---|---|
| 1 | agile development | 4 | 18 % |
## Responsibilities — full market ranking
| # | Item | Ads | Share |
|---|---|---|---|
| 1 | data pipeline maintenance | 4 | 18 % |
## Regional breakdown
> **Corpus note:** 22 relevant ads in total — below the 100-ad target for a fully reliable ranking. Percentages above should be read as indicative.
### US (us)
**Insufficient evidence** — 22 ads (minimum for a regional ranking: 30). No ranking is reported for this region.
### UK (gb)
**Insufficient evidence** — 0 ads (minimum for a regional ranking: 30). No ranking is reported for this region.
### EU/DACH (de, at, ch, nl)
**Insufficient evidence** — 0 ads (minimum for a regional ranking: 30). No ranking is reported for this region.
## Job title variants in the market
| Title | Ads |
|---|---|
| Senior Full-Stack Data Engineer – TS/SCI Clearance | 2 |
| Cloud Data Engineer: Pipelines, Spark & Databricks | 1 |
| Combat Data Systems Engineer 🏆 | 1 |
| Data Engineer (Databricks) | 1 |
| Data Engineer / Analytics Programmer (Healthcare Data) | 1 |
| Data Engineer / Data Architect III | 1 |
| Data Engineer All Levels Crystal City VA | 1 |
| Data Engineer III | 1 |
| Data Engineer- up to $260k - FS Poly McLean, VA | 1 |
| Data Engineer(AWS/DataBricks experience REQUIRED) | 1 |
| Data Engineer, Mid | 1 |
| Data Engineer: ETL, Databricks & Big Data (Remote) | 1 |
| Entry Level Data Engineer | 1 |
| ETL Data Engineer | 1 |
| Kearney Activate Senior Consultant Data Engineer | 1 |
| Mid Data Engineer | 1 |
| Remote Data Engineer | 1 |
| Senior Data Engineer | 1 |
| Senior Data Engineer (Remote) | 1 |
| Senior Data Engineer: Build Scalable Data Pipelines | 1 |
| Senior Data Engineer: Mission-Critical Pipelines (Secret) | 1 |
Methodology: entities extracted per ad ({hard_skills, tools, methods, responsibilities, seniority}), normalized, counted as DISTINCT ads per entity; report threshold ≥ 3 ads. Headline sections in skills.md/tools.md use the stricter ≥ 20 % threshold.

18
references/profile.md Normal file
View File

@@ -0,0 +1,18 @@
# Occupation profile — data engineer
- **ESCO URI:** http://data.europa.eu/esco/occupation/2079755f-d809-49e6-8037-4de6180e54c0
- **ESCO code:** 2511.20
- **ISCO-08 group:** 2511 — Systems analysts
## Description (ESCO)
Data engineers develop the architecture needed to process, manage, and store large amounts of data which will be used by data scientists for analysis. They design the infrastructure and maintain data pipelines and warehouses to leverage data for strategic advantage.
## Definition
nan
## Alternative labels
- data engineer expert
- research data engineer

64
references/skills.md Normal file
View File

@@ -0,0 +1,64 @@
# Competences — data engineer
Source: ESCO v1.2.1 occupation-skill relations (http://data.europa.eu/esco/occupation/2079755f-d809-49e6-8037-4de6180e54c0).
## Essential
- **cloud technologies** (knowledge)
- **computer science** (knowledge)
- **create data sets** (skill/competence)
- **data analytics** (knowledge)
- **data models** (knowledge)
- **data storage** (knowledge)
- **data warehouse** (knowledge)
- **database management systems** (knowledge)
- **design database in the cloud** (skill/competence)
- **develop data processing applications** (skill/competence)
- **digital data processing** (nan)
- **establish data processes** (skill/competence)
- **implement data warehousing techniques** (skill/competence)
- **manage data** (skill/competence)
- **manage ICT data architecture** (skill/competence)
- **manage quantitative data** (skill/competence)
- **manage research data** (skill/competence)
- **perform dimensionality reduction** (skill/competence)
- **process data** (skill/competence)
- **store digital data and systems** (skill/competence)
- **unstructured data** (knowledge)
- **use data processing techniques** (skill/competence)
- **use databases** (skill/competence)
## Optional
- analyse pipeline database information (skill/competence)
- create data models (skill/competence)
- SAS Data Management (knowledge)
- statistics (knowledge)
- Teradata Database (knowledge)
<!-- market-evidence -->
## Market evidence (job-ad analysis, 22 ads, as of 2026-07-11)
Share of analyzed job ads mentioning the item (threshold ≥ 20 %). Source: JSearch/Adzuna APIs.
### Hard skills
- data modeling — **50 %**
- data pipeline design — **27 %**
- data integration — **27 %**
- data analysis — **23 %**
- data engineering — **23 %**
- data transformation — **23 %**
- data warehousing — **18 %**
- statistical analysis — **18 %**
### Methods
- agile development — **18 %**
### Responsibilities
- data pipeline maintenance — **18 %**
<!-- market-evidence -->

41
references/tasks.md Normal file
View File

@@ -0,0 +1,41 @@
# Tasks & work activities — data engineer
Source: O*NET 30.3, occupation 15-2051.00 (Data Scientists) — manual nearest-occupation mapping via ISCO group 2511; the official ESCO crosswalk has no entry for this ESCO occupation.
## Task statements
- **[nan]** Analyze, manipulate, or process large sets of data using statistical software.
- **[nan]** Apply feature selection algorithms to models predicting outcomes of interest, such as sales, attrition, and healthcare use.
- **[nan]** Apply sampling techniques to determine groups to be surveyed or use complete enumeration methods.
- **[nan]** Clean and manipulate raw data using statistical software.
- **[nan]** Compare models using statistical performance metrics, such as loss functions or proportion of explained variance.
- **[nan]** Create graphs, charts, or other visualizations to convey the results of data analysis using specialized software.
- **[nan]** Deliver oral or written presentations of the results of mathematical modeling and data analysis to management or other end users.
- **[nan]** Design surveys, opinion polls, or other instruments to collect data.
- **[nan]** Identify business problems or management objectives that can be addressed through data analysis.
- **[nan]** Identify relationships and trends or any factors that could affect the results of research.
- **[nan]** Identify solutions to business problems, such as budgeting, staffing, and marketing decisions, using the results of data analysis.
- **[nan]** Propose solutions in engineering, the sciences, and other fields using mathematical theories and techniques.
- **[nan]** Read scientific articles, conference papers, or other sources of research to identify emerging analytic trends and technologies.
- **[nan]** Recommend data-driven solutions to key stakeholders.
- **[nan]** Test, validate, and reformulate models to ensure accurate prediction of outcomes of interest.
- **[nan]** Write new functions or applications in programming languages to conduct analyses.
## Detailed work activities
- Advise others on analytical techniques.
- Analyze business or financial data.
- Analyze data to identify or resolve operational problems.
- Analyze data to identify trends or relationships among variables.
- Analyze data to inform operational decisions or activities.
- Apply mathematical principles or statistical approaches to solve problems in scientific or applied fields.
- Determine appropriate methods for data analysis.
- Develop procedures to evaluate organizational activities.
- Develop scientific or mathematical models.
- Prepare analytical reports.
- Prepare data for analysis.
- Prepare graphics or other visual representations of information.
- Present research results to others.
- Select resources needed to accomplish tasks.
- Update technical knowledge.
- Write computer programming code.

112
references/tools.md Normal file
View File

@@ -0,0 +1,112 @@
# Tools & technology — data engineer
Source: O*NET 30.3, occupation 15-2051.00 (Data Scientists) — manual nearest-occupation mapping via ISCO group 2511; the official ESCO crosswalk has no entry for this ESCO occupation.
| Software | Category | Hot technology |
|---|---|---|
| Alteryx software | Business intelligence and data analysis software | yes |
| Amazon Elastic Compute Cloud EC2 | Data base user interface and query software | yes |
| Amazon Redshift | Data base user interface and query software | yes |
| Amazon Web Services AWS software | Data base user interface and query software | yes |
| Apache Airflow | Procedure management software | yes |
| Apache Cassandra | Data base management system software | yes |
| Apache Hadoop | Data base management system software | yes |
| Apache Hive | Data base management system software | yes |
| Apache Kafka | Development environment software | yes |
| Apache Spark | Business intelligence and data analysis software | yes |
| Atlassian Confluence | Project management software | yes |
| Atlassian JIRA | Content workflow software | yes |
| Bash | Operating system software | yes |
| C | Development environment software | yes |
| C# | Object or component oriented development software | yes |
| C++ | Object or component oriented development software | yes |
| Docker | Application server software | yes |
| Elasticsearch | Data base management system software | yes |
| Git | File versioning software | yes |
| GitHub | Application server software | yes |
| Go | Development environment software | yes |
| Google Looker Analytics | Business intelligence and data analysis software | yes |
| IBM SPSS Statistics | Analytical or scientific software | yes |
| JavaScript | Web platform development software | yes |
| JavaScript Object Notation JSON | Web platform development software | yes |
| Jenkins CI | Enterprise application integration software | yes |
| Kubernetes | Application server software | yes |
| Linux | Operating system software | yes |
| Microsoft Access | Data base user interface and query software | yes |
| Microsoft Azure software | Development environment software | yes |
| Microsoft Excel | Spreadsheet software | yes |
| Microsoft Office software | Office suite software | yes |
| Microsoft Power BI | Business intelligence and data analysis software | yes |
| Microsoft PowerPoint | Presentation software | yes |
| Microsoft SQL Server | Data base user interface and query software | yes |
| MongoDB | Data base management system software | yes |
| NoSQL | Data base management system software | yes |
| Oracle Java | Object or component oriented development software | yes |
| Perl | Object or component oriented development software | yes |
| PostgreSQL | Object oriented data base management software | yes |
| Python | Object or component oriented development software | yes |
| PyTorch | Data base user interface and query software | yes |
| R | Object or component oriented development software | yes |
| Ruby | Development environment software | yes |
| SAS | Analytical or scientific software | yes |
| Scala | Object or component oriented development software | yes |
| Shell script | Operating system software | yes |
| Snowflake | Data mining software | yes |
| Splunk Enterprise | Enterprise system management software | yes |
| Structured query language SQL | Data base user interface and query software | yes |
| Tableau | Business intelligence and data analysis software | yes |
| TensorFlow | Analytical or scientific software | yes |
| Teradata Database | Data base management system software | yes |
| The MathWorks MATLAB | Analytical or scientific software | yes |
| UNIX | Operating system software | yes |
| Amazon Simple Storage Service S3 | Storage networking software | |
| Amazon Web Services AWS SageMaker | Cloud-based management software | |
| Apache MXNet | Industrial control software | |
| Apache Pig | Data base management system software | |
| BigQuery | Data base user interface and query software | |
| Business intelligence software | Business intelligence and data analysis software | |
| Flask | Development environment software | |
| Geographic information system GIS systems | Geographic information system | |
| Google Cloud software | Cloud-based management software | |
| Julia | Development environment software | |
| Jupyter software | Object or component oriented development software | |
| Keras | Operating system software | |
| Kubeflow | Analytical or scientific software | |
| Management information systems MIS | Enterprise resource planning ERP software | |
| MapReduce big data software | Business intelligence and data analysis software | |
| Mathematical software | Analytical or scientific software | |
| Mlflow | Analytical or scientific software | |
| Neo4j | Data base user interface and query software | |
| NumPy | Data base user interface and query software | |
| OpenAI ChatGPT | Development environment software | |
| pandas | Data base user interface and query software | |
| PySpark | Data base user interface and query software | |
| Qlik Tech QlikView | Business intelligence and data analysis software | |
| Reporting software | Data base reporting software | |
| RESTful API | Web platform development software | |
| Scikit-learn | Development environment software | |
| SciPy | Object or component oriented development software | |
| Shiny | Object or component oriented development software | |
| spaCy | Object or component oriented development software | |
| StataCorp Stata | Analytical or scientific software | |
| Statistical software | Analytical or scientific software | |
| XGBoost | Development environment software | |
<!-- market-evidence -->
## Market evidence (job-ad analysis, 22 ads, as of 2026-07-11)
Share of analyzed job ads mentioning the item (threshold ≥ 20 %). Source: JSearch/Adzuna APIs.
### Tools
- Python — **41 %**
- Databricks — **41 %**
- Apache Spark — **23 %**
- AWS — **23 %**
- Spark — **23 %**
- SQL — **23 %**
- Snowflake — **18 %**
- informatica — **18 %**
<!-- market-evidence -->