skills-core/data-scientist
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: data-scientist skill package v0.1.0

Browse Source
This commit is contained in:
skillfactor-pipeline
2026-08-14 17:28:35 +02:00
commit 85e55a396b
11 changed files with 1978 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

166
references/ai-skills.md Normal file
View File

@@ -0,0 +1,166 @@
# External AI agent skills — data-scientist
Proven, publicly available AI agent skills mapped to this occupation.
Nothing is copied from the sources: every entry is a name, a one-line
summary and a link to the upstream skill package. Each section names
its source repository, commit, license and retrieval date.
**Tiers:** `core` = the skill directly exercises a top market hard
skill, tool or method (from gated job-ad evidence) or an essential
ESCO competence of this occupation; `adjacent` =
plausibly useful, secondary. Entries are capped at 12 per source
and 80 in total per occupation (core first,
strongest matches survive); everything beyond the caps is excluded
and logged in the pipeline audit trail, not in this package.
_Matched deterministically (ISCO group + title/competence keywords,
tiered against market evidence + ESCO essentials) by
`pipeline/p5_enrich_ai_skills.py` on 2026-07-14._
## Source: anthropics/skills
- Repository: [https://github.com/anthropics/skills](https://github.com/anthropics/skills) (commit `f6656c1`, retrieved 2026-07-14)
- License: Apache-2.0; the document skills (docx/pdf/pptx/xlsx) are source-available — see the LICENSE.txt in the upstream skill folder
| Skill | Tier | What it adds | Upstream |
|---|---|---|---|
| `mcp-builder` | core | Guide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python … | [source](https://github.com/anthropics/skills/tree/main/skills/mcp-builder) |
| `claude-api` | adjacent | Reference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration. TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a … | [source](https://github.com/anthropics/skills/tree/main/skills/claude-api) |
| `webapp-testing` | adjacent | Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs. | [source](https://github.com/anthropics/skills/tree/main/skills/webapp-testing) |
| `docx` | adjacent | Use this skill whenever the user wants to create, read, edit, or manipulate Word documents (.docx files) or Word templates (.dotx files). Triggers include: any mention of 'Word doc', 'word document', '.docx', '.dotx', or requests to … | [source](https://github.com/anthropics/skills/tree/main/skills/docx) |
| `pdf` | adjacent | Use this skill whenever the user wants to do anything with PDF files. This includes reading or extracting text/tables from PDFs, combining or merging multiple PDFs into one, splitting PDFs apart, rotating pages, adding watermarks, creating … | [source](https://github.com/anthropics/skills/tree/main/skills/pdf) |
| `skill-creator` | adjacent | Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with … | [source](https://github.com/anthropics/skills/tree/main/skills/skill-creator) |
## Source: obra/superpowers
- Repository: [https://github.com/obra/superpowers](https://github.com/obra/superpowers) (commit `d884ae0`, retrieved 2026-07-14)
- License: MIT (c) Jesse Vincent
| Skill | Tier | What it adds | Upstream |
|---|---|---|---|
| `finishing-a-development-branch` | adjacent | Use when implementation is complete, all tests pass, and you need to decide how to integrate the work - guides completion of development work by presenting structured options for merge, PR, or cleanup | [source](https://github.com/obra/superpowers/tree/main/skills/finishing-a-development-branch) |
| `subagent-driven-development` | adjacent | Use when executing implementation plans with independent tasks in the current session | [source](https://github.com/obra/superpowers/tree/main/skills/subagent-driven-development) |
| `test-driven-development` | adjacent | Use when implementing any feature or bugfix, before writing implementation code | [source](https://github.com/obra/superpowers/tree/main/skills/test-driven-development) |
| `using-git-worktrees` | adjacent | Use when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallback | [source](https://github.com/obra/superpowers/tree/main/skills/using-git-worktrees) |
| `brainstorming` | adjacent | You MUST use this before any creative work - creating features, building components, adding functionality, or modifying behavior. Explores user intent, requirements and design before implementation. | [source](https://github.com/obra/superpowers/tree/main/skills/brainstorming) |
| `executing-plans` | adjacent | Use when you have a written implementation plan to execute in a separate session with review checkpoints | [source](https://github.com/obra/superpowers/tree/main/skills/executing-plans) |
| `receiving-code-review` | adjacent | Use when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementation | [source](https://github.com/obra/superpowers/tree/main/skills/receiving-code-review) |
| `dispatching-parallel-agents` | adjacent | Use when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | [source](https://github.com/obra/superpowers/tree/main/skills/dispatching-parallel-agents) |
| `requesting-code-review` | adjacent | Use when completing tasks, implementing major features, or before merging to verify work meets requirements | [source](https://github.com/obra/superpowers/tree/main/skills/requesting-code-review) |
| `systematic-debugging` | adjacent | Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes | [source](https://github.com/obra/superpowers/tree/main/skills/systematic-debugging) |
| `verification-before-completion` | adjacent | Use when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always | [source](https://github.com/obra/superpowers/tree/main/skills/verification-before-completion) |
| `writing-plans` | adjacent | Use when you have a spec or requirements for a multi-step task, before touching code | [source](https://github.com/obra/superpowers/tree/main/skills/writing-plans) |
## Source: wshobson/agents
- Repository: [https://github.com/wshobson/agents](https://github.com/wshobson/agents) (commit `6fd3247`, retrieved 2026-07-14)
- License: MIT (c) Seth Hobson
| Skill | Tier | What it adds | Upstream |
|---|---|---|---|
| `debugging-strategies` | core | Master systematic debugging techniques, profiling tools, and root cause analysis to efficiently track down bugs across any codebase or technology stack. Use when investigating bugs, performance issues, or unexpected behavior. | [source](https://github.com/wshobson/agents/tree/main/plugins/developer-essentials/skills/debugging-strategies) |
| `sql-optimization-patterns` | core | Master SQL query optimization, indexing strategies, and EXPLAIN analysis to dramatically improve database performance and eliminate slow queries. Use when debugging slow queries, designing database schemas, or optimizing application … | [source](https://github.com/wshobson/agents/tree/main/plugins/developer-essentials/skills/sql-optimization-patterns) |
| `code-documentation-code-reviewer (agent)` | core | Elite code review expert specializing in modern AI-powered code analysis, security vulnerabilities, performance optimization, and production reliability. Masters static analysis tools, security scanning, and configuration review with … | [source](https://github.com/wshobson/agents/tree/main/plugins/code-documentation/agents/code-reviewer.md) |
| `shellcheck-configuration` | core | Master ShellCheck static analysis configuration and usage for shell script quality. Use when setting up linting infrastructure, fixing code issues, or ensuring script portability. | [source](https://github.com/wshobson/agents/tree/main/plugins/shell-scripting/skills/shellcheck-configuration) |
| `code-refactoring-code-reviewer (agent)` | core | Elite code review expert specializing in modern AI-powered code analysis, security vulnerabilities, performance optimization, and production reliability. Masters static analysis tools, security scanning, and configuration review with … | [source](https://github.com/wshobson/agents/tree/main/plugins/code-refactoring/agents/code-reviewer.md) |
| `codebase-cleanup-code-reviewer (agent)` | core | Elite code review expert specializing in modern AI-powered code analysis, security vulnerabilities, performance optimization, and production reliability. Masters static analysis tools, security scanning, and configuration review with … | [source](https://github.com/wshobson/agents/tree/main/plugins/codebase-cleanup/agents/code-reviewer.md) |
| `comprehensive-review-code-reviewer (agent)` | core | Elite code review expert specializing in modern AI-powered code analysis, security vulnerabilities, performance optimization, and production reliability. Masters static analysis tools, security scanning, and configuration review with … | [source](https://github.com/wshobson/agents/tree/main/plugins/comprehensive-review/agents/code-reviewer.md) |
| `git-pr-workflows-code-reviewer (agent)` | core | Elite code review expert specializing in modern AI-powered code analysis, security vulnerabilities, performance optimization, and production reliability. Masters static analysis tools, security scanning, and configuration review with … | [source](https://github.com/wshobson/agents/tree/main/plugins/git-pr-workflows/agents/code-reviewer.md) |
| `tdd-workflows-code-reviewer (agent)` | core | Elite code review expert specializing in modern AI-powered code analysis, security vulnerabilities, performance optimization, and production reliability. Masters static analysis tools, security scanning, and configuration review with … | [source](https://github.com/wshobson/agents/tree/main/plugins/tdd-workflows/agents/code-reviewer.md) |
| `python-error-handling` | core | Python error handling patterns including input validation, exception hierarchies, and partial failure handling. Use when implementing validation logic, designing exception strategies, handling batch processing failures, or building robust … | [source](https://github.com/wshobson/agents/tree/main/plugins/python-development/skills/python-error-handling) |
| `uv-package-manager` | core | Master the uv package manager for fast Python dependency management, virtual environments, and modern Python project workflows. Use when setting up Python projects, managing dependencies, or optimizing Python development workflows with uv. | [source](https://github.com/wshobson/agents/tree/main/plugins/python-development/skills/uv-package-manager) |
| `dbt-transformation-patterns` | core | Master dbt (data build tool) for analytics engineering with model organization, testing, documentation, and incremental strategies. Use when building data transformations, creating data models, or implementing analytics engineering best … | [source](https://github.com/wshobson/agents/tree/main/plugins/data-engineering/skills/dbt-transformation-patterns) |
## Source: google/skills
- Repository: [https://github.com/google/skills](https://github.com/google/skills) (commit `b15f327`, retrieved 2026-07-14)
- License: Apache-2.0
| Skill | Tier | What it adds | Upstream |
|---|---|---|---|
| `bigquery-basics` | core | Manages datasets, tables, and jobs in BigQuery. Use when you need to interact with BigQuery, run SQL queries, manage BigQuery resources (datasets, tables, views), or perform basic data ingestion and analysis. | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/bigquery-basics) |
| `bigquery-ai-ml` | core | Leverages BigQuery's built-in machine learning and GenAI capabilities for advanced data analytics. Use when you need to write SQL queries that perform time-series forecasting, detect outliers, find key drivers, or leverage generative AI … | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/bigquery-ai-ml) |
| `datalineage-bigquery-asset-impact-analysis` | core | Analyzes the downstream impact (blast radius) when a BigQuery table or view is broken, stale, or modified. Identifies all downstream tables, dashboards, and processes that will be affected. Use when: - Performing a blast radius or impact … | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/datalineage-bigquery-asset-impact-analysis) |
| `agent-platform-eval-flywheel` | core | Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology. Use when evaluating an agent or model, building an eval dataset, picking or writing evaluation metrics, analyzing … | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/agent-platform-eval-flywheel) |
| `bigtable-basics` | core | Assists in provisioning instances/tables, designing performant schemas, and querying data in Bigtable. Use when designing Bigtable row keys, configuring column families, writing SQL queries or client library code (Java, Go, Python) for … | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/bigtable-basics) |
| `cloud-sql-basics` | core | This file generates or explains Cloud SQL resources. Use this file when the user asks to create a Cloud SQL instance or database for MySQL, PostgreSQL, or SQL Server. Cloud SQL manages third-party MySQL, PostgreSQL, and SQL Server … | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/cloud-sql-basics) |
| `alloydb-basics` | adjacent | Manages clusters, instances, and backups for AlloyDB for PostgreSQL, and integrates with AlloyDB model context protocol (MCP) tools for automated database operations. | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/alloydb-basics) |
| `agent-platform-alert-configuration` | adjacent | Configures best-practice alerting policies for Google Cloud Vertex AI / Agent Platform agents on Agent Runtime. Use when analyzing, writing, or deploying alerting policies to monitor agent latency, error rates, and quality metrics … | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/agent-platform-alert-configuration) |
| `gke-observability` | adjacent | Configures GKE observability, including Cloud Logging, Cloud Monitoring, and managed Prometheus. Use when configuring GKE monitoring, setting up GKE logging, or configuring Prometheus metrics collection. Don't use to configure local … | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/gke-observability) |
| `google-analytics-admin-api-basics` | adjacent | Manages Google Analytics account and property settings, enables the Analytics Admin API via the Cloud CLI, lists accounts and properties, and manages data streams, custom dimensions, conversion events, and integrations. Use when you need … | [source](https://github.com/google/skills/tree/b15f327/skills/analytics/google-analytics-admin-api-basics) |
| `google-analytics-data-api-basics` | adjacent | Manages Google Analytics reporting data, enables the Analytics Data API via the Cloud CLI, and creates reports using the Google Analytics Data API (v1beta). Use when you need to interact with Google Analytics properties, run customized … | [source](https://github.com/google/skills/tree/b15f327/skills/analytics/google-analytics-data-api-basics) |
| `google-cloud-networking-observability` | adjacent | Investigates Google Cloud networking issues by analyzing logs, metrics, and diagnostics. Use when investigating VPC Flow Logs (including cost estimation), NAT, firewall, or threat logs, querying latency and throughput metrics, or running … | [source](https://github.com/google/skills/tree/b15f327/skills/cloud/google-cloud-networking-observability) |
## Source: czlonkowski/n8n-skills
- Repository: [https://github.com/czlonkowski/n8n-skills](https://github.com/czlonkowski/n8n-skills) (commit `9ea3aa5`, retrieved 2026-07-14)
- License: MIT
| Skill | Tier | What it adds | Upstream |
|---|---|---|---|
| `n8n-code-python` | core | Write Python code in n8n Code nodes. Use when writing Python in n8n, using _input/_json/_node syntax, working with standard library, or need to understand Python limitations in n8n Code nodes. Use this skill when the user specifically … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-code-python) |
| `n8n-code-tool` | core | Write JavaScript or Python for the n8n Custom Code Tool (@n8n/n8n-nodes-langchain.toolCode) — the AI-agent-callable tool, NOT the workflow Code node. Use when building a Code Tool attached to an AI Agent, writing code that an LLM will … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-code-tool) |
| `n8n-node-configuration` | adjacent | Operation-aware node configuration guidance. Use when configuring nodes, understanding property dependencies, determining required fields, choosing between get_node detail levels, or learning common configuration patterns by node type. … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-node-configuration) |
| `n8n-workflow-patterns` | adjacent | Proven workflow architectural patterns from real n8n workflows. Use when building new workflows, designing workflow structure, choosing workflow patterns, planning workflow architecture, or asking about webhook processing, HTTP API … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-workflow-patterns) |
| `n8n-agents` | adjacent | Design n8n AI agents the right way. Use when building or editing any @n8n/n8n-nodes-langchain.* AI node — an AI Agent, LLM chain, Text Classifier, or Information Extractor — and whenever the user mentions AI agents, LLM with tools, tool … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-agents) |
| `n8n-subworkflows` | adjacent | Build reusable, composable n8n sub-workflows. Use when extracting shared logic, building anything multi-step or reused across workflows, or any workflow over ~10 nodes — and whenever the user mentions sub-workflows, Execute Workflow, … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-subworkflows) |
| `n8n-validation-expert` | adjacent | Interpret validation errors and guide fixing them. Use when encountering validation errors, validation warnings, false positives, operator structure issues, or need help understanding validation results. Also use when asking about … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-validation-expert) |
| `n8n-binary-and-data` | adjacent | Handle files and binary data in n8n correctly. Use when working with files, images, PDFs, attachments, uploads or downloads, base64, vision/multimodal input, or when an AI agent needs a file as tool input or output — and whenever the user … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-binary-and-data) |
| `n8n-code-javascript` | adjacent | Write JavaScript code in n8n Code nodes. Use when writing JavaScript in n8n, using $input/$json/$node syntax, making HTTP requests with this.helpers / the $helpers global, working with dates using DateTime, troubleshooting Code node … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-code-javascript) |
| `n8n-multi-instance` | adjacent | Use when an n8n-mcp account targets more than one n8n instance — i.e. the `n8n_instances` tool is available, the user mentions multiple n8n instances or environments (prod vs staging, several teams or clients), a workflow / datatable / … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-multi-instance) |
| `n8n-error-handling` | adjacent | Wire n8n error handling so failures are loud, structured, and recoverable. Use when building any webhook/API workflow, a scheduled or unattended workflow, or any path where a silent failure would drop user-visible work — and whenever the … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-error-handling) |
| `n8n-expression-syntax` | adjacent | Validate n8n expression syntax and fix common errors. Use when writing n8n expressions, using {{}} syntax, accessing $json/$node variables, troubleshooting expression errors, mapping data between nodes, or referencing webhook data in … | [source](https://github.com/czlonkowski/n8n-skills/tree/9ea3aa5/skills/n8n-expression-syntax) |
## Source: NVIDIA/skills
- Repository: [https://github.com/NVIDIA/skills](https://github.com/NVIDIA/skills) (commit `153b14b`, retrieved 2026-07-14)
- License: CC-BY-4.0 (skills/docs), Apache-2.0 (code)
| Skill | Tier | What it adds | Upstream |
|---|---|---|---|
| `nemo-rl-auto-research` | core | Autonomous NeMo-RL research agent workflow for directed hypothesis testing and open-ended discovery. Guides agents through the full experiment lifecycle: understanding recipes and environments, wiring RL or NeMo-gym runs, launching … | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/nemo-rl-auto-research) |
| `deepstream-dev` | core | NVIDIA DeepStream SDK 9.0 development with Python pyservicemaker API. Use when building video analytics pipelines, GStreamer-based video processing, TensorRT inference integration, object detection/tracking, or Kafka/message broker … | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/deepstream-dev) |
| `hsb-app` | core | Discover and run Holoscan Sensor Bridge example applications on a connected devkit. Filters available apps by the user's platform, HSB software version, board type, and sensors. Supports timed execution, failure analysis, code-edit … | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/hsb-app) |
| `vss-generate-video-report` | core | Use this skill when producing a VSS analysis report — Mode A per-clip VLM, Mode B incident-range via video-analytics. Not for standalone video summarization, real-time alerts or ad-hoc Q&A. | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/vss-generate-video-report) |
| `tao-finetune-clip` | core | CLIP vision-language model for image-text retrieval, zero-shot classification, embedding extraction, ONNX export, and TensorRT deployment. Use when fine-tuning or training CLIP, running zero-shot classification, computing image embeddings, … | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/tao-finetune-clip) |
| `tilegym-cutile-python` | core | Expert cuTile programming assistant. Write high-performance GPU kernels using cuTile's tile-based programming model with proper validation and optimization. Supports deep agent orchestration for complex multi-kernel tasks. | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/tilegym-cutile-python) |
| `tao-train-visual-changenet` | core | Visual ChangeNet for binary image classification and segmentation in AOI defect detection. Use when training, evaluating, exporting, or running inference for PCB defect detection or visual inspection, comparing image pairs for PASS/NO_PASS … | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/tao-train-visual-changenet) |
| `cuopt-install` | core | Install cuOpt for Python, C, or server via pip, conda, or Docker; verify the install. For building cuOpt from source, see cuopt-developer. | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/cuopt-install) |
| `jetson-diagnostic` | core | Read-only Jetson health snapshot for identity, memory, GPU, thermal, power, storage, services, and top processes. | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/jetson-diagnostic) |
| `tao-port-huggingface-model` | core | Integrate a HuggingFace Computer Vision model into the NVIDIA TAO Toolkit ecosystem (tao-core config, tao-pytorch trainer, tao-deploy TensorRT pipeline). Use when the user asks to "integrate a HuggingFace model into TAO", "add an HF model … | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/tao-port-huggingface-model) |
| `cuopt-developer` | core | Modify, build, test, debug, and contribute to NVIDIA cuOpt (C++/CUDA, Python, server, CI). Use for solver internals, PRs, DCO, and code conventions. | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/cuopt-developer) |
| `cupynumeric-install` | core | Install and verify cuPyNumeric for Python — requirements, commands, verification. Source builds are out of scope. | [source](https://github.com/NVIDIA/skills/tree/153b14b/skills/cupynumeric-install) |
## Source: phuryn/pm-skills
- Repository: [https://github.com/phuryn/pm-skills](https://github.com/phuryn/pm-skills) (commit `18468a9`, retrieved 2026-07-14)
- License: MIT
| Skill | Tier | What it adds | Upstream |
|---|---|---|---|
| `ab-test-analysis` | core | Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split … | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-data-analytics/skills/ab-test-analysis) |
| `cohort-analysis` | core | Perform cohort analysis on user engagement data — retention curves, feature adoption trends, and segment-level insights. Use when analyzing user retention by cohort, studying feature adoption over time, investigating churn patterns, or … | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-data-analytics/skills/cohort-analysis) |
| `sql-queries` | core | Generate SQL queries from natural language descriptions. Supports BigQuery, PostgreSQL, MySQL, and other dialects. Reads database schemas from uploaded diagrams or documentation. Use when writing SQL, building data reports, exploring … | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-data-analytics/skills/sql-queries) |
| `dummy-dataset` | core | Generate realistic dummy datasets for testing with customizable columns, constraints, and output formats (CSV, JSON, SQL, Python script). Use when creating test data, building mock datasets, or generating sample data for development and … | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-execution/skills/dummy-dataset) |
| `lean-canvas` | core | Generate a Lean Canvas with problem, solution, metrics, cost structure, UVP, unfair advantage, channels, segments, and revenue. Use when exploring a lean startup canvas, testing a business hypothesis, or modeling a new venture. | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-product-strategy/skills/lean-canvas) |
| `metrics-dashboard` | core | Define and design a product metrics dashboard with key metrics, data sources, visualization types, and alert thresholds. Use when creating a metrics dashboard, defining KPIs, setting up product analytics, or building a data monitoring plan. | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-product-discovery/skills/metrics-dashboard) |
| `gtm-strategy` | adjacent | Create a go-to-market strategy covering marketing channels, messaging, success metrics, and launch timeline. Use when planning a product launch, creating a GTM plan from scratch, or defining a launch strategy for a new market. | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-go-to-market/skills/gtm-strategy) |
| `north-star-metric` | adjacent | Define a North Star Metric and 3-5 supporting input metrics that form a metrics constellation. Classify the business game (Attention, Transaction, Productivity) and validate against 7 criteria for an effective North Star. Use when choosing … | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-marketing-growth/skills/north-star-metric) |
| `product-strategy` | adjacent | Create a comprehensive product strategy using the 9-section Product Strategy Canvas — vision, segments, costs, value propositions, trade-offs, metrics, growth, capabilities, and defensibility. Use when building a product strategy, creating … | [source](https://github.com/phuryn/pm-skills/tree/18468a9/pm-product-strategy/skills/product-strategy) |
## Source: veniceai/skills
- Repository: [https://github.com/veniceai/skills](https://github.com/veniceai/skills) (commit `de089fa`, retrieved 2026-07-14)
- License: MIT
| Skill | Tier | What it adds | Upstream |
|---|---|---|---|
| `venice-embeddings` | core | Call POST /embeddings on Venice. Covers request shape (input, model, encoding_format, dimensions, user), OpenAI compatibility, response compression (gzip/br), and practical usage for retrieval, clustering, and RAG. | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-embeddings) |
| `venice-api-keys` | adjacent | Manage Venice API keys. Covers GET/POST/PATCH/DELETE /api_keys, GET /api_keys/{id}, GET /api_keys/rate_limits, GET /api_keys/rate_limits/log, the two-step /api_keys/generate_web3_key wallet flow, INFERENCE vs ADMIN key types, and per-key … | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-api-keys) |
| `venice-characters` | adjacent | Discover and use Venice public characters (persona-driven system prompts with a bound model). Covers GET /characters (search/filter/sort), /characters/{slug}, /characters/{slug}/reviews, the Character schema, and how to apply a character … | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-characters) |
| `venice-api-overview` | adjacent | High-level map of the Venice.ai API - base URL, authentication modes, endpoint categories, response headers, pricing model, error shape, and versioning. Load this first when starting any Venice integration. | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-api-overview) |
| `venice-errors` | adjacent | Handle Venice API errors correctly. Covers the StandardError / DetailedError / ContentViolationError / X402InferencePaymentRequired body shapes, every meaningful status code (400, 401, 402, 403, 415, 422, 429, 500, 503, 504), the 402 … | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-errors) |
| `venice-auth` | adjacent | Authenticate to the Venice API with a Bearer API key or with an x402 / SIWE wallet. Covers header formats, the SIWE message fields, TTL and nonce rules, the venice-x402-client SDK, and how to choose between the two modes. | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-auth) |
| `venice-image-edit` | adjacent | Transform existing images with Venice. Covers POST /image/edit (prompt-driven single-image edit), /image/multi-edit (compose 1-3 images), /image/upscale (2-4x upscale + enhance), and /image/background-remove. Accepts base64, file upload, … | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-image-edit) |
| `venice-image-generate` | adjacent | Generate images with Venice. Covers POST /image/generate (Venice-native), POST /images/generations (OpenAI-compatible), GET /image/styles (style presets), request fields (prompt, dimensions, cfg_scale, seed, variants, style_preset, … | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-image-generate) |
| `venice-responses` | adjacent | Use Venice's Alpha POST /responses endpoint - an OpenAI-compatible Responses API with typed output blocks (reasoning, message, function_call, web_search_call). Covers request shape, streaming, differences from /chat/completions, supported … | [source](https://github.com/veniceai/skills/tree/de089fa/skills/venice-responses) |

79
references/berufsbild-de.md Normal file
View File

@@ -0,0 +1,79 @@
# Berufsbild (Deutschland) — Data Scientist
> **Quelle:** BERUFENET, Bundesagentur für Arbeit — Beruf-ID 129987,
> KldB 2010: B 43104 · via öffentliche BERUFENET-API,
> abgerufen 2026-07-12. Automatisches Label-Matching
> (Score 1.00); Inhalte paraphrasiert/gekürzt, Volltext beim
> Original: https://web.arbeitsagentur.de/berufenet/beruf/129987
## Aufgaben und Tätigkeiten kompakt
Data Scientists analysieren große, teils unstrukturierte Datenmengen aus Quellen wie Social Media, Bereitstellung von Computerressourcen über das Internet. Infrastruktur wie Server oder Datenspeicher, aber auch Software, Plattformen und Funktionen müssen nicht auf lokalen Computern vorgehalten werden, sondern stehen als Dienstleistung bereit. IT-Dienstleistungen können so dynamisch an den Bedarf angepasst angeboten, genutzt und abgerechnet werden.
" name="Cloud-Computing">Cloud Computing Services oder mobilen Anwendungen. Sie nutzen Methoden der Informatik, Mathematik und Statistik, um aus den Daten Informationen zu gewinnen. Dafür entwickeln sie technische Verfahren, Mit "Big Data" bezeichnet man große Mengen strukturierter oder unstrukturierter Daten, deren Umfang zudem schnell wächst beziehungsweise die sehr rasch ausgetauscht werden. Die Daten können in unterschiedlichsten Formaten, etwa als Text-, Bild- oder Videodateien vorliegen. Es können zum Beispiel Produktions- und Transportdaten von Unternehmen sein, Daten aus Finanztransaktionen oder zum Online-Suchverhalten. Mit manuellen und herkömmlichen Methoden der Datenverarbeitung kann diese Datenflut nicht mehr ausgewertet werden. Es bedarf einer speziellen Big-Data-Analyse, basierend auf Algorithmen und intelligenten Mustererkennungstechnologien. s. auch "Data Mining".
" name="Big Data">Big-Data-Architekturen sowie automatisierte Prozesse. Ihre Ergebnisse tragen sie zu aussagekräftigen Präsentationen zusammen und unterstützen die Firmenleitung und Kundschaft bei Entscheidungsprozessen. Sie erstellen webbasierte Informations- und Wissensmanagementsysteme und modellieren Datenschemata zur Integration und Analyse. Bei Bedarf beraten sie Kundschaft und übernehmen Schulungs-, Vertriebs- und Marketingaufgaben.
## Tätigkeitsbezeichnungen
Data Scientist
Auch übliche Berufsbezeichnung/SynonymDatenwissenschaftler/in
## Die Tätigkeit im Überblick
Data Scientists analysieren und verarbeiten mit Methoden der Mathematik, Informatik und Statistik große Datenmengen in Echtzeit aus verschiedenen Quellen ("Mit "Big Data" bezeichnet man große Mengen strukturierter oder unstrukturierter Daten, deren Umfang zudem schnell wächst beziehungsweise die sehr rasch ausgetauscht werden. Die Daten können in unterschiedlichsten Formaten, etwa als Text-, Bild- oder Videodateien vorliegen. Es können zum Beispiel Produktions- und Transportdaten von Unternehmen sein, Daten aus Finanztransaktionen oder zum Online-Suchverhalten. Mit manuellen und herkömmlichen Methoden der Datenverarbeitung kann diese Datenflut nicht mehr ausgewertet werden. Es bedarf einer speziellen Big-Data-Analyse, basierend auf Algorithmen und intelligenten Mustererkennungstechnologien. s. auch "Data Mining".
" name="Big Data">Big Data"), um z.B. über potenzielle Kunden oder Markttrends nutzbare Informationen zu erlangen.
## Trends
AIOps: Künstliche Intelligenz für den IT-BetriebAIOps (Artificial Intelligence for IT Operations) verbindet separate IT-Tools mithilfe von Künstlicher Intelligenz (KI) zu einer automatisierten IT-Betriebsplattform. Auch: Maschinelles Lernen, Deep Learning
Bestandteil künstlicher Intelligenz und Oberbegriff für alle Verfahren, die es Maschinen beziehungsweise Computerprogrammen ermöglichen, Wissen aus Erfahrung zu generieren, also zu lernen.
Eine Sonderform des Machine Learning ist das Deep Learning. Es eignet sich für alle Anwendungen, bei denen große Datenbestände zur Verfügung stehen, aus denen sich Muster und Modelle ableiten lassen. Als technische Grundlage des Deep Learning dienen künstliche neuronale Netze, die während des Lernvorgangs immer wieder neu verknüpft werden.
" name="Machine Learning (ML)">Machine Learning und Mit "Big Data" bezeichnet man große Mengen strukturierter oder unstrukturierter Daten, deren Umfang zudem schnell wächst beziehungsweise die sehr rasch ausgetauscht werden. Die Daten können in unterschiedlichsten Formaten, etwa als Text-, Bild- oder Videodateien vorliegen. Es können zum Beispiel Produktions- und Transportdaten von Unternehmen sein, Daten aus Finanztransaktionen oder zum Online-Suchverhalten. Mit manuellen und herkömmlichen Methoden der Datenverarbeitung kann diese Datenflut nicht mehr ausgewertet werden. Es bedarf einer speziellen Big-Data-Analyse, basierend auf Algorithmen und intelligenten Mustererkennungstechnologien. s. auch "Data Mining".
" name="Big Data">Big Data automatisieren und optimieren die Überwachung von IT-Infrastrukturen. Echtzeit-Datenanalysen identifizieren, überprüfen und beheben Probleme und Störungsfälle automatisch. AIOps-Plattformen entlasten dadurch IT-Personal und überwachen Systeme rund um die Uhr. Fach- und Führungskräfte in der IT-Administration werden sich zukünftig mit AIOps-Technologien auseinandersetzen und den IT-Betrieb mithilfe von Künstlicher Intelligenz optimieren.
Edge Computing: Dezentrale Datenverarbeitung am NetzwerkrandAnwendungen für Industrie 4.0, Smart Cities und das Digitale Plattformen, die verschiedenartige Geräte und Applikationen über das Internet der Dinge vernetzen. Die Plattformen sorgen für den Informationsaustausch, indem sie systemübergreifende Verknüpfungen herstellen und Funktionen zur Steuerung und zur Analyse von Daten bieten.
" name="IoT-Plattformen">Internet of Things erzeugen riesige Mengen an Speziell in der Automatisierungs-, Mess-, Steuerungs- und R
## Aufgaben und Tätigkeiten (Beschreibung)
Worum geht es?Data Scientists analysieren und verarbeiten mit Methoden der Mathematik, Informatik und Statistik große Datenmengen in Echtzeit aus verschiedenen Quellen ("Mit "Big Data" bezeichnet man große Mengen strukturierter oder unstrukturierter Daten, deren Umfang zudem schnell wächst beziehungsweise die sehr rasch ausgetauscht werden. Die Daten können in unterschiedlichsten Formaten, etwa als Text-, Bild- oder Videodateien vorliegen. Es können zum Beispiel Produktions- und Transportdaten von Unternehmen sein, Daten aus Finanztransaktionen oder zum Online-Suchverhalten. Mit manuellen und herkömmlichen Methoden der Datenverarbeitung kann diese Datenflut nicht mehr ausgewertet werden. Es bedarf einer speziellen Big-Data-Analyse, basierend auf Algorithmen und intelligenten Mustererkennungstechnologien. s. auch "Data Mining".
" name="Big Data">Big Data"), um z.B. über potenzielle Kunden oder Markttrends nutzbare Informationen zu erlangen.
Big Data beherrschenGroße Datenmengen fallen mittlerweile in fast allen Bereichen des Lebens an: bei Banken und Versicherungen, Telekommunikationsunternehmen, in Sozialen Netzwerken, bei Internetdienstleistern, in der Industrie oder der Marktforschung. Prognosen gehen davon aus, dass sich das Datenvolumen alle zwei Jahre verdoppelt. Immer mehr Unternehmen versprechen sich von der Nutzung dieser Daten unter anderem strategische Wettbewerbsvorteile, Umsatzsteigerungen und eine höhere Produktivität.
Das Problem dabei: Die komplexen Datenmengen sind oft unstrukturiert, stammen aus verschiedenen Quellen und müssen in Echtzeit verarbeitet werden. Um Ordnung in das scheinbare Datenchaos zu bringen, sind Data Scientists gefragte Spezialisten. Sie wissen Konzepte und Techniken aus Informatik, Statistik und Mathematik so zu nutzen, dass aus den Daten verwertbare Informationen und Wissen für Prognosen, Entscheidungen und Maßnahmen in den unterschiedlichen Geschäftsbereichen gewonnen werden. Darüber hinaus nutzen sie Verfahren aus den Bereichen Auch: Maschinelles Lernen, Deep Learning
Bestandteil künstlicher Intelligenz und Oberbegriff für alle Verfahren, die es Maschinen beziehungsweise Computerprogrammen ermöglichen, Wissen aus Erfahrung zu generieren, also zu lernen.
Eine Sonderform des Machine Learning ist das Deep Learning. Es eignet sich für alle Anwendungen, bei denen große Datenbestände zur Verfügung stehen, aus denen sich Muster und Modelle ableiten lassen. Als technische Grundlage des Deep Learning dienen künstliche neu
## Aufgaben und Tätigkeiten im Einzelnen
Datenerhebung und -analyseautomatisierte Systeme zur Datengewinnung und komplexe Analysemethoden entwickeln
technische Infrastruktur analysieren und Änderungs- bzw. Optimierungsbedarf feststellen
Studien planen und auswerten
DatenmanagementDesign und Implementierung von Mit "Big Data" bezeichnet man große Mengen strukturierter oder unstrukturierter Daten, deren Umfang zudem schnell wächst beziehungsweise die sehr rasch ausgetauscht werden. Die Daten können in unterschiedlichsten Formaten, etwa als Text-, Bild- oder Videodateien vorliegen. Es können zum Beispiel Produktions- und Transportdaten von Unternehmen sein, Daten aus Finanztransaktionen oder zum Online-Suchverhalten. Mit manuellen und herkömmlichen Methoden der Datenverarbeitung kann diese Datenflut nicht mehr ausgewertet werden. Es bedarf einer speziellen Big-Data-Analyse, basierend auf Algorithmen und intelligenten Mustererkennungstechnologien. s. auch "Data Mining".
" name="Big Data">Big-Data-Analytics-Lösungen entwickeln und Aufwände abschätzen
Systematische Anwendung statistischer Methoden auf große Datenbestände (insbesondere Big Data beziehungsweise Massendaten) mit dem Ziel, neue Querverbindungen und Trends zu erkennen.
Siehe auch
Big Data
" name="Datenanalyse">Data-Mining-Lösungen mit analytischen Tools und Big-Data-Technologien anwenden
Zentrale Datensammlung zur Informationsintegration und Datenanalyse zum Beispiel als Grundlage für betriebswirtschaftliche Entscheidungen. Mithilfe eines Data-Warehouse-Systems können Daten aus verschiedenen Quellen zusammengeführt und vereinheitlicht werden.
" name="Data Warehouse">Data-Warehouse-Architekturen evaluieren und ein Data-Warehouse-System aufbauen
Methodisches Rechenverfahren zum Lösen einer vorgegebenen Aufgabe oder eines Problems
" name="Algorithmus">Algorithmen für das maschinelle Lernen sowie für Data-Mining-Verfahren entwickeln
Optionen zum Aufbau einer Business-Intelligence- und Big-Data-Architektur, Datenbankkonzepte und Datenintegrations-Lösungen prüfen
Daten mit entsprechender Software aufbereiten, interpretieren und darstellen (auch grafisch)
vorhandene Modellierungsstandards und Reportings weiterentwickeln
Projektleitungtechnische Projektleitung oder Gesamtprojektleitung von interdisziplinären Big-Data-/Business-Analytics-Projekten übernehmen
Projektlösungsansätze erstellen und in einer Projektstruktur umsetzen
bei Erstellung von Angeboten und Angebotspräsentationen mitwirken
Beratung und Schulung Kundschaft system-

180
references/market.md Normal file
View File

@@ -0,0 +1,180 @@
# Market evidence report — data-scientist
Source: **46 real job ads** (JSearch API, countries: us 46), extracted into the MSSQL evidence store; as of 2026-07-11.
This report contains extracted, aggregated facts only — no ad text is
reproduced (copyright / platform terms).
## Seniority distribution
| Seniority | Ads | Share |
|---|---|---|
| mid | 25 | 54 % |
| lead | 10 | 22 % |
| senior | 8 | 17 % |
| junior | 3 | 7 % |
## Tools — full market ranking
| # | Item | Ads | Share |
|---|---|---|---|
| 1 | Python | 36 | 78 % |
| 2 | SQL | 21 | 46 % |
| 3 | Spark | 11 | 24 % |
| 4 | AWS | 10 | 22 % |
| 5 | EMR | 5 | 11 % |
| 6 | Hadoop | 5 | 11 % |
| 7 | scikit-learn | 5 | 11 % |
| 8 | Azure | 4 | 9 % |
| 9 | Conda | 4 | 9 % |
| 10 | H2O | 4 | 9 % |
| 11 | Hive | 4 | 9 % |
| 12 | Kafka | 4 | 9 % |
| 13 | PyTorch | 4 | 9 % |
| 14 | Tableau | 4 | 9 % |
| 15 | TensorFlow | 4 | 9 % |
| 16 | Java | 3 | 7 % |
| 17 | MATLAB | 3 | 7 % |
| 18 | NoSQL | 3 | 7 % |
| 19 | pandas | 3 | 7 % |
| 20 | Power BI | 3 | 7 % |
| 21 | R programming language | 3 | 7 % |
## Hard skills — full market ranking
| # | Item | Ads | Share |
|---|---|---|---|
| 1 | machine learning | 31 | 67 % |
| 2 | data analysis | 26 | 57 % |
| 3 | statistical modeling | 17 | 37 % |
| 4 | statistical analysis | 15 | 33 % |
| 5 | predictive modeling | 13 | 28 % |
| 6 | data mining | 12 | 26 % |
| 7 | data visualization | 9 | 20 % |
| 8 | natural language processing | 9 | 20 % |
| 9 | data modeling | 8 | 17 % |
| 10 | deep learning | 8 | 17 % |
| 11 | quantitative analysis | 8 | 17 % |
| 12 | time series analysis | 7 | 15 % |
| 13 | feature engineering | 6 | 13 % |
| 14 | anomaly detection | 5 | 11 % |
| 15 | classification | 5 | 11 % |
| 16 | clustering | 5 | 11 % |
| 17 | data cleaning | 5 | 11 % |
| 18 | data exploration | 5 | 11 % |
| 19 | model validation | 5 | 11 % |
| 20 | sentiment analysis | 5 | 11 % |
| 21 | algorithm development | 4 | 9 % |
| 22 | data integration | 4 | 9 % |
| 23 | data science | 4 | 9 % |
| 24 | risk assessment | 4 | 9 % |
| 25 | backtesting | 3 | 7 % |
| 26 | data analytics | 3 | 7 % |
| 27 | data cleansing | 3 | 7 % |
| 28 | data engineering | 3 | 7 % |
| 29 | data management | 3 | 7 % |
| 30 | data normalization | 3 | 7 % |
| 31 | predictive analytics | 3 | 7 % |
## Methods — full market ranking
| # | Item | Ads | Share |
|---|---|---|---|
| 1 | machine learning | 4 | 9 % |
| 2 | backtesting | 3 | 7 % |
| 3 | mlops | 3 | 7 % |
## Responsibilities — full market ranking
| # | Item | Ads | Share |
|---|---|---|---|
| 1 | data analysis | 16 | 35 % |
| 2 | model development | 11 | 24 % |
| 3 | solution development | 5 | 11 % |
| 4 | anomaly detection | 4 | 9 % |
| 5 | data retrieval | 4 | 9 % |
| 6 | client communication | 3 | 7 % |
| 7 | data combination | 3 | 7 % |
| 8 | data exploration | 3 | 7 % |
| 9 | data visualization | 3 | 7 % |
| 10 | documentation maintenance | 3 | 7 % |
| 11 | insight revelation | 3 | 7 % |
| 12 | model validation | 3 | 7 % |
| 13 | solution deployment | 3 | 7 % |
| 14 | trend identification | 3 | 7 % |
## Regional breakdown
> **Corpus note:** 46 relevant ads in total — below the 100-ad target for a fully reliable ranking. Percentages above should be read as indicative.
### US (us)
46 ads.
**Top hard skills:**
- machine learning — 67 % (31 ads)
- data analysis — 57 % (26 ads)
- statistical modeling — 37 % (17 ads)
- statistical analysis — 33 % (15 ads)
- predictive modeling — 28 % (13 ads)
- data mining — 26 % (12 ads)
- data visualization — 20 % (9 ads)
- natural language processing — 20 % (9 ads)
- data modeling — 17 % (8 ads)
- deep learning — 17 % (8 ads)
**Top tools:**
- Python — 78 % (36 ads)
- SQL — 46 % (21 ads)
- Spark — 24 % (11 ads)
- AWS — 22 % (10 ads)
- EMR — 11 % (5 ads)
- Hadoop — 11 % (5 ads)
- scikit-learn — 11 % (5 ads)
- Azure — 9 % (4 ads)
- Conda — 9 % (4 ads)
- H2O — 9 % (4 ads)
**Seniority:** mid 54 % · lead 22 % · senior 17 % · junior 7 %
### UK (gb)
**Insufficient evidence** — 0 ads (minimum for a regional ranking: 30). No ranking is reported for this region.
### EU/DACH (de, at, ch, nl)
**Insufficient evidence** — 0 ads (minimum for a regional ranking: 30). No ranking is reported for this region.
## Job title variants in the market
| Title | Ads |
|---|---|
| Data Scientist | 2 |
| Data Scientist - Experienced to Expert Level (Maryland) | 2 |
| Data Scientist, AWS Security | 2 |
| Data Scientist, Mid | 2 |
| Senior Associate, Data Scientist - Corporate Strategy | 2 |
| AI and ML Data Scientist Jobs | 1 |
| Data Science Lead, Personnel Health Research & Data Analytics | 1 |
| Data Scientist - Advanced Manufacturing Technologies (AMT) | 1 |
| Data Scientist - Expert | 1 |
| Data Scientist - Mid | 1 |
| Data Scientist - Senior | 1 |
| Data Scientist - Sr | 1 |
| Data Scientist - Supply Chain Risk Management with Security Clearance | 1 |
| Data Scientist - Traditional and Generative AI | 1 |
| Data Scientist (U.S. Citizen, Secret/Top Secret), Washington D.C. | 1 |
| Data Scientist II | 1 |
| Data Scientist III | 1 |
| Data Scientist Jobs | 1 |
| Data Scientist Lead ID | 1 |
| Data Scientist SETA (TS/SCI #26-099) | 1 |
| Data Scientist, Level 4 with Security Clearance | 1 |
| Data Scientist, Mid level | 1 |
| Data Scientist, Product Analytics | 1 |
| Experienced Data Scientist | 1 |
| Junior DoD Data Scientist - Onsite in Alexandria | 1 |
Methodology: entities extracted per ad ({hard_skills, tools, methods, responsibilities, seniority}), normalized, counted as DISTINCT ads per entity; report threshold ≥ 3 ads. Headline sections in skills.md/tools.md use the stricter ≥ 20 % threshold.

998
references/practitioner-qa.md Normal file
View File

@@ -0,0 +1,998 @@
# Practitioner knowledge — Data Scientist
> **Source & license:** Curated from the Stack Exchange data dump
> `stackexchange_20260331` (community mirror on archive.org). Original questions and
> answers are © their authors, licensed **CC-BY-SA 4.0**; per-entry
> attribution below links each original post and names its author. Summaries
> are SkillFactor's own wording; this compilation is share-alike
> (CC-BY-SA 4.0). Compiled 2026-07-11.
150 curated Q&A insights, grouped by theme, highest community score first.
## Machine Learning
**The user needs to split their dataset into three subsets – training, testing, and validation – using scikit-learn, but is aware that the standard `train_test_split` function only handles two splits.**
When a library function doesn't directly support a desired operation (like splitting data into more than two sets), you can often achieve the result by applying the function sequentially. This approach breaks down a complex task into smaller, manageable steps using available tools. It’s important to consider how each split affects the final proportions of your datasets.
*Source: [Train/Test/Validation Set Splitting in Sklearn](https://datascience.stackexchange.com/q/15135) — answer by hh32, CC-BY-SA 4.0*
**The questioner is trying to understand the "dying ReLU" problem described in neural network training materials – specifically, why some ReLU neurons can become permanently inactive during learning.**
The 'dying ReLU' issue occurs when a neuron learns weights that consistently result in it outputting zero for all inputs, effectively removing it from contributing to the network’s decision-making process. This happens because once a neuron is stuck outputting zero, gradient descent can't adjust its weights to reactivate it. Alternatives like 'Leaky ReLU' introduce a small slope for negative inputs, providing a pathway for recovery and preventing complete inactivity.
*Source: [What is the "dying ReLU" problem in neural networks?](https://datascience.stackexchange.com/q/5706) — answer by Neil Slater, CC-BY-SA 4.0*
**The user has successfully built a deep learning model and now needs to create a clear, professional-quality diagram of its architecture for inclusion in a research paper.**
When needing to visually represent complex technical systems like neural networks, leveraging existing tools can significantly streamline the process. Rather than manually creating diagrams, utilizing specialized software designed for this purpose ensures clarity, consistency, and ease of integration with document preparation systems. This allows professionals to focus on analysis and communication rather than tedious visual creation.
*Source: [How to draw Deep learning network architecture diagrams?](https://datascience.stackexchange.com/q/14899) — answer by Pablo Rivas, CC-BY-SA 4.0*
**The user is seeking updated Python libraries for neural networks because their current choice, pybrain, is no longer actively maintained.**
The field of deep learning frameworks evolves rapidly, so it's important to be aware that tools can become outdated quickly. Rather than committing to a single library, explore several options – TensorFlow, Keras, Lasagne, etc. – and choose the one best suited for your specific needs and project requirements, prioritizing active community support and ongoing development.
*Source: [Best python library for neural networks](https://datascience.stackexchange.com/q/694) — answer by Madison May, CC-BY-SA 4.0*
**The asker questions the behavior of the cross-entropy loss function when predicted probabilities approach zero, and asks for clarification on different formulations of the equation they've encountered.**
Cross-entropy can be understood as a measure of how 'surprised' a model is by observed data. It’s derived from maximizing the log-likelihood of observing the true labels given the predicted probabilities; a zero probability prediction for an actual class results in infinite loss because it represents a completely unexpected event. Framing it this way highlights that cross-entropy isn't just a mathematical formula, but reflects how well a model's predictions align with reality and can be interpreted as encoding length or 'surprise'.
*Source: [The cross-entropy error function in neural networks](https://datascience.stackexchange.com/q/9302) — answer by KT., CC-BY-SA 4.0*
**The questioner is learning machine learning and currently uses R, but sees many examples using Python and wants to know which language is preferred in academic and industry settings.**
While both languages are capable for the initial 'model building' phase of a machine learning project, Python excels at deployment into production systems due to its status as a general-purpose programming language already used in many organizations’ infrastructure. R remains strong in statistical analysis and visualization, particularly within academia, but often requires integration with other languages for full lifecycle implementation. Ultimately, the choice depends on where the model will be *used* – Python is favored when seamless production integration is key.
*Source: [Python vs R for machine learning](https://datascience.stackexchange.com/q/326) — answer by binga, CC-BY-SA 4.0*
**The asker is struggling to determine an appropriate learning rate when implementing Stochastic Gradient Descent (SGD) for training neural networks, and wants to understand how gradient shape influences this choice.**
Basic SGD uses a single, global learning rate independent of the error gradient; however, more advanced algorithms like Adagrad and Adadelta adapt the learning rate per parameter based on historical gradients. Finding a good starting value (around 0.01) typically requires experimentation with cross-validation. A common strategy is to start with a relatively high, stable learning rate for initial exploration and then gradually decrease it over time to refine the solution.
*Source: [Choosing a learning rate](https://datascience.stackexchange.com/q/410) — answer by indico, CC-BY-SA 4.0*
**A user is asking about the practical differences between using Gini Impurity and Information Gain (Entropy) when building decision trees, specifically which one performs better in different situations.**
While both metrics generally yield similar results for decision tree construction, Gini impurity offers a slight computational advantage. It avoids logarithmic calculations, making it faster to compute, especially with large datasets. For most practical applications within the CART framework, either metric is acceptable, but Gini impurity provides a minor efficiency benefit.
*Source: [When should I use Gini Impurity as opposed to Information Gain (Entropy)?](https://datascience.stackexchange.com/q/10228) — answer by Dawny33, CC-BY-SA 4.0*
**The asker is learning linear regression and questions why cost functions use squared errors instead of simply summing the absolute differences between predicted and actual values.**
Using squared error (or other even powers) in a cost function addresses a fundamental problem: minimizing raw error can lead to unstable or boundary-condition solutions. Squaring ensures the cost is always positive, creating a well-behaved optimization landscape. Furthermore, this choice aligns with statistical assumptions about noise – specifically that errors are often normally distributed due to the Central Limit Theorem – allowing for more robust model fitting when perfect accuracy isn't achievable.
*Source: [Why do cost functions use the square error?](https://datascience.stackexchange.com/q/10188) — answer by Harsh, CC-BY-SA 4.0*
**The questioner is unsure if AUC is truly valuable for model validation, especially given claims that its benefit over standard accuracy is minimal and primarily useful for identifying models that perform well due to data imbalance rather than genuine predictive power.**
AUC excels when dealing with imbalanced datasets where a high accuracy score can be misleading. By evaluating performance across various classification thresholds (represented by the ROC curve), AUC provides a more robust measure of a model's ability to distinguish between classes, regardless of class distribution. Focusing on metrics like True Positive Rate and False Positive Rate offers a clearer picture than relying solely on overall accuracy in skewed scenarios.
*Source: [Advantages of AUC vs standard accuracy](https://datascience.stackexchange.com/q/806) — answer by indico, CC-BY-SA 4.0*
**The user is asking how to calculate cross-entropy loss for a single training example in a five-class classification problem, given the true label and the network's predicted probabilities.**
Cross-entropy loss measures the difference between predicted probability distributions and the actual distribution of labels. It focuses solely on the probability assigned to the *correct* class; how the remaining probability is distributed among incorrect classes doesn’t affect the loss for that example. This makes it a useful metric because it directly penalizes confidence in wrong answers, while rewarding higher probabilities for correct ones.
*Source: [Cross-entropy loss explanation](https://datascience.stackexchange.com/q/20296) — answer by Neil Slater, CC-BY-SA 4.0*
**The questioner is asking for a clear explanation of the difference between Gradient Descent and Stochastic Gradient Descent, particularly how they differ in their approach to updating parameters during training.**
Both methods aim to minimize error by iteratively adjusting parameters, but they differ in *how much* data is used per update. Gradient Descent calculates updates using the entire dataset, which can be slow for large datasets, while Stochastic Gradient Descent uses only one (or a small batch of) data points per update, making it faster initially. Although SGD may not converge to the absolute minimum like GD, its quicker progress and tendency to oscillate near the optimum are often sufficient for practical applications.
*Source: [What is the difference between Gradient Descent and Stochastic Gradient Descent?](https://datascience.stackexchange.com/q/36450) — answer by Sociopath, CC-BY-SA 4.0*
**The user is working with a decision tree/random forest model in scikit-learn and needs to incorporate string features (like country names) alongside numerical data, but scikit-learn requires purely numeric inputs.**
Directly converting strings to numbers (e.g., hashing) can mislead the algorithm by imposing an artificial order on categorical data that doesn't exist. The preferred approach is one-hot encoding: creating separate binary columns for each unique string value, indicating presence or absence. While this increases dimensionality, it accurately represents the categorical information without introducing unintended relationships.
*Source: [strings as features in decision tree/random forest](https://datascience.stackexchange.com/q/5226) — answer by rapaio, CC-BY-SA 4.0*
**The questioner understands that correlated features provide redundant information but struggles to grasp *why* this redundancy can negatively impact machine learning models.**
While not always detrimental, highly correlated features often hinder model performance and stability. Linear models are particularly sensitive, potentially producing unreliable results due to multicollinearity. Even more complex algorithms like random forests may struggle to identify true feature interactions when faced with redundant information, and generally, a simpler model—one that avoids unnecessary redundancy—is preferable for better generalization.
*Source: [In supervised learning, why is it bad to have correlated features?](https://datascience.stackexchange.com/q/24452) — answer by Ami Tavory, CC-BY-SA 4.0*
**The user is seeking a clear explanation of the differences between Gradient Boosting Machines (GBM) and XGBoost, particularly why XGBoost often outperforms GBM.**
While both algorithms are based on gradient boosting principles, XGBoost incorporates regularization techniques to prevent overfitting, leading to improved predictive performance. Beyond algorithmic refinements, XGBoost was specifically engineered for computational efficiency and scalability, allowing it to handle larger datasets and complex models effectively. Essentially, XGBoost can be considered a more robust and optimized implementation of the core gradient boosting idea.
*Source: [GBM vs XGBOOST? Key differences?](https://datascience.stackexchange.com/q/16904) — answer by Icyblade, CC-BY-SA 4.0*
**The questioner is seeking a theoretical justification for using the exponential function within the softmax normalization, beyond just its numerical benefits. They understand cross-entropy loss but want to know *why* softmax specifically pairs with it, and why not other normalization methods.**
Softmax isn't strictly required for cross-entropy loss, but it provides a crucial differentiable approximation of the 'argmax' function which identifies the most likely class. The exponential transformation dramatically amplifies differences in input scores, making the highest score overwhelmingly probable after normalization – effectively creating a 'soft' version of the hard max operation. This non-linearity is key for effective gradient-based training and allows the model to confidently predict one class over others.
*Source: [In softmax classifier, why use exp function to do normalization?](https://datascience.stackexchange.com/q/23159) — answer by vega, CC-BY-SA 4.0*
**The questioner observes a rise in 'machine learning engineer' job postings, particularly in areas where 'data scientist' originated, and wonders if this signals a shift in job market terminology similar to how 'data scientist' superseded 'statistician'.**
The core difference lies in the application of knowledge: data scientists focus on exploration, theory, and model *creation*, while machine learning/data engineers concentrate on taking those models and reliably deploying them into practical systems. Many roles currently labeled 'data scientist' actually require engineering skills focused on productionizing solutions. Job titles are often misaligned with actual responsibilities; a true 'machine learning engineer' posting indicates the company understands this distinction and needs someone specifically for implementation.
*Source: [Data scientist vs machine learning engineer](https://datascience.stackexchange.com/q/28006) — answer by Vincenzo Lavorini, CC-BY-SA 4.0*
**The questioner is confused why deep learning models utilize small batches (like 32 or 64 examples) for training instead of processing all data in a single large batch, given that it seems less efficient.**
Using mini-batches strikes a balance between the drawbacks of full-batch and single-instance gradient descent. Full-batch can get stuck in suboptimal solutions, while single-instance is too noisy to converge efficiently. Mini-batches introduce enough noise to escape bad local minima but still provide a stable enough signal for relatively fast learning; recent research suggests smaller batch sizes (2-32) often yield better generalization performance despite hardware limitations.
*Source: [Why mini batch size is better than one single "batch" with all training data?](https://datascience.stackexchange.com/q/16807) — answer by horaceT, CC-BY-SA 4.0*
**The asker is confused about what 'logits' represent in machine learning and why applying non-linearities directly to them can be problematic.**
Logits are the raw, unnormalized output scores from a model – essentially its initial predictions before being converted into probabilities. While these values indicate relative confidence, they aren’t easily interpretable on their own. Normalizing logits (e.g., using softmax) transforms them into probabilities that provide meaningful insight into the model's certainty and allow for comparison between different predictions.
*Source: [What does Logits in machine learning mean?](https://datascience.stackexchange.com/q/31041) — answer by n1k31t4, CC-BY-SA 4.0*
**The user is questioning whether using K-means clustering with Euclidean distance on latitude/longitude coordinates is appropriate for geolocation data.**
K-means isn't ideal because it minimizes variance based on straight-line distances, which doesn’t account for the curvature of the Earth. Geolocation data requires algorithms that can utilize more accurate geodetic distance functions like Haversine. Algorithms such as hierarchical clustering or DBSCAN are better suited since they aren't limited to linear distance metrics and can handle non-linear spaces.
*Source: [Clustering geo location coordinates (lat,long pairs)](https://datascience.stackexchange.com/q/761) — answer by Has QUIT--Anony-Mousse, CC-BY-SA 4.0*
**The questioner noticed that Keras' implementations of Leaky ReLU and PReLU appear mathematically identical, despite being presented as distinct activation functions. They are seeking clarification on whether their understanding is incorrect.**
Both Leaky ReLU and PReLU address the 'dying ReLU' problem by introducing a small slope for negative inputs, allowing gradients to flow even when neurons aren’t activated. The key difference lies in how that slope is determined: Leaky ReLU uses a fixed coefficient, while PReLU learns this coefficient as a trainable parameter during network optimization. This learnability gives PReLU greater flexibility and potentially better performance, but at the cost of increased model complexity.
*Source: [LeakyReLU vs PReLU](https://datascience.stackexchange.com/q/18583) — answer by Thomas Wagenaar, CC-BY-SA 4.0*
**The user is trying to calculate precision, recall, and F1-score for a binary classification Keras model but is finding it difficult after an update to the Keras library.**
Recent versions of Keras have removed built-in functions for these metrics, requiring manual calculation. The solution involves defining custom metric functions using Keras' backend operations to compute precision, recall, and F1-score within each batch during training and evaluation. These custom metrics can then be included when compiling the model, allowing them to be tracked alongside loss and accuracy.
*Source: [How to get accuracy, F1, precision and recall, for a keras model?](https://datascience.stackexchange.com/q/45165) — answer by Tasos, CC-BY-SA 4.0*
**The questioner observes that the V-function (state value) seems redundant given the Q-function (state-action value), and is seeking clarification on the distinct roles each plays in reinforcement learning.**
Both V and Q functions estimate expected future rewards, but differ in their scope. The V-function evaluates the *value of being in a state* assuming optimal behavior from that point forward, while the Q-function assesses the *value of taking a specific action within a state*. The V-function can be derived by averaging the Q-values across all possible actions for a given state, weighted by their probabilities under the current policy.
*Source: [What is the Q function and what is the V function in reinforcement learning?](https://datascience.stackexchange.com/q/9832) — answer by aerin, CC-BY-SA 4.0*
**The user is unsure how to incorporate static or non-time series features (like weather) into an LSTM time series model alongside existing time-dependent data.**
LSTMs can handle multiple features at each timestep, so adding a feature like weather *is* possible. However, consider whether the added feature truly evolves over time; if not, directly feeding it to the LSTM might be inefficient. A better approach is often to process non-time series data separately and then merge its output with the LSTM's output using additional layers, effectively creating a multi-input model.
*Source: [Adding Features To Time Series Model LSTM](https://datascience.stackexchange.com/q/17099) — answer by Adam Sypniewski, CC-BY-SA 4.0*
**The questioner asks why machine learning batch sizes are often recommended to be powers of 2, questioning if maximizing GPU/RAM usage is more important.**
GPU architecture relies on data parallelism where many processing units work simultaneously on different parts of the same task. Aligning the number of virtual processors (batch size) with the power-of-2 structure of physical processors maximizes efficiency by ensuring all processors have work to do at each step. This is a low-level optimization related to GPU programming, not the learning process itself; it's about how computations are organized for parallel execution.
*Source: [What is the advantage of keeping batch size a power of 2?](https://datascience.stackexchange.com/q/20179) — answer by jcm69, CC-BY-SA 4.0*
**The questioner understands data normalization in machine learning but struggles to grasp why data shuffling is necessary, particularly when used with mini-batch gradient descent algorithms like Adam or SGD.**
Shuffling data introduces variability into the training process by effectively changing the 'shape' of the loss function with each iteration. This prevents optimization algorithms from getting trapped in local minima – points where improvement seems impossible because the algorithm only sees a limited, static view of the problem space. By presenting different subsets of data (mini-batches) in random order, shuffling helps the model explore more broadly and potentially find better solutions.
*Source: [Why should the data be shuffled for machine learning tasks](https://datascience.stackexchange.com/q/24511) — answer by Josh, CC-BY-SA 4.0*
**The questioner normalized training data for an author identification SVM and is unsure if they should also normalize the test data, reasoning that the model has already 'learned' feature importance.**
Machine learning models operate on the specific numerical representation of input data, including its scale. Normalization isn’t about teaching the algorithm what’s important; it’s about consistently presenting information in a format the model expects and was trained with. Therefore, test data *must* be normalized using the same parameters (scale and offset) derived from the training data to ensure consistent interpretation and accurate predictions.
*Source: [Should we apply normalization to test data as well?](https://datascience.stackexchange.com/q/27615) — answer by Neil Slater, CC-BY-SA 4.0*
**The question asks about effective methods for feature engineering from date/time data in machine learning applications, specifically how to extract useful features from timestamp columns.**
Effective time-based feature engineering begins with visualizing the time variable against other variables to identify patterns. Beyond standard cyclical features like day of week or hour of day, consider broader trends occurring over days, months, or years. Crucially, don't rely solely on automated feature creation; human visual inspection is vital for uncovering unusual but significant temporal relationships and, importantly, detecting data quality issues that can skew results.
*Source: [Machine learning - features engineering from date/time data](https://datascience.stackexchange.com/q/2368) — answer by Ben Haley, CC-BY-SA 4.0*
**The user is confused about how to interpret the feature importance metrics (Gain, Cover, Frequency, Split, RealCover, and RealCover%) output by XGBoost. They are seeking clarification on what these values represent and how to use them to understand which features are most important in their model.**
XGBoost's feature importance metrics provide a relative understanding of each feature’s contribution to the model's predictive power. 'Gain' is generally the most useful metric, representing the total reduction in error achieved by splitting on that feature across all trees. 'Cover' indicates how many data points are influenced by a feature, and 'Frequency' shows how often it's used for splits. These metrics are relative – they sum to one – allowing you to compare features and identify those with the greatest impact on model predictions.
*Source: [How to interpret the output of XGBoost importance?](https://datascience.stackexchange.com/q/12318) — answer by Sandeep S. Sandhu, CC-BY-SA 4.0*
**The user, new to Keras and CNNs, is unsure how to appropriately configure `batch_size`, `steps_per_epoch`, and `validation_steps` given training and test dataset sizes of 240,000 and 80,000 samples respectively.**
Choosing a `batch_size` involves balancing accuracy and speed; larger batches provide more stable gradient estimates but require more memory, while smaller batches are faster but noisier. `steps_per_epoch` and `validation_steps` control how much of the training/validation data is used *per epoch*, becoming important when dealing with very large or dynamically generated datasets. If you can process your entire dataset within a reasonable timeframe, it's generally best to omit these parameters and let Keras use the full dataset for each epoch.
*Source: [How to set batch_size, steps_per epoch, and validation steps?](https://datascience.stackexchange.com/q/29719) — answer by Silpion, CC-BY-SA 4.0*
**The user is confused about the difference between scikit-learn's `OrdinalEncoder` and `LabelEncoder`, as they appear to function similarly.**
While both encoders numerically represent categorical data, their intended use cases differ. `OrdinalEncoder` is designed for encoding *features* (independent variables) in a dataset with multiple columns, while `LabelEncoder` historically focused on encoding the *target variable* (dependent variable). Modern practice favors using `OrdinalEncoder` even for target variables when appropriate due to its ability to handle multi-dimensional data.
*Source: [Difference between OrdinalEncoder and LabelEncoder](https://datascience.stackexchange.com/q/39317) — answer by ipramusinto, CC-BY-SA 4.0*
**The user needs to implement anomaly detection on time-series log data within a Python environment, having previously explored commercial solutions and encountering issues with open-source R ports or Windows compatibility.**
Effective anomaly detection often relies on identifying deviations from established patterns. Simple but powerful techniques like calculating moving averages and standard deviations can highlight unusual values without requiring complex machine learning models. While more sophisticated probabilistic methods exist, starting with these basic statistical approaches provides a practical foundation for real-world implementation.
*Source: [Open source Anomaly Detection in Python](https://datascience.stackexchange.com/q/6547) — answer by Kasra Manshaei, CC-BY-SA 4.0*
**The user is confused why a loss function parameter `from_logits=True` requires unbounded output values from the model, given that probabilities (used in calculating loss) should be between 0 and 1.**
Setting `from_logits=True` tells the loss function your model outputs raw, unscaled scores ('logits') instead of probabilities. The loss function then internally applies a softmax function to convert these logits into a probability distribution. This approach can sometimes improve numerical stability during training by avoiding potential issues with very small or large probability values.
*Source: [What does from_logits=True do in SparseCategoricalcrossEntropy loss function?](https://datascience.stackexchange.com/q/73093) — answer by today, CC-BY-SA 4.0*
**The user has a large tabular dataset (50 million rows, 200 columns) with both numerical and categorical features and is trying to decide between deep learning and gradient boosting for binary classification.**
Begin with the simplest possible model – in this case, a linear classifier like Logistic Regression or Linear SVM – to establish a performance baseline. Complex models like deep learning are often unnecessary for tabular data and require significant resources and expertise; they excel when dealing with inherent hierarchical structures (like images/text). Don't assume all data must be used immediately; smaller samples can often yield good results, speeding up initial experimentation and model development.
*Source: [Deep Learning vs gradient boosting: When to use what?](https://datascience.stackexchange.com/q/2504) — answer by Simon, CC-BY-SA 4.0*
**The questioner observes that the V-function (state value) seems redundant given the Q-function (state-action value), and is seeking clarification on the distinct roles each plays in reinforcement learning.**
Both V and Q functions estimate expected future rewards, but differ in their scope. The V-function evaluates the *value of being in a state* assuming optimal behavior from that point forward, while the Q-function assesses the *value of taking a specific action within a state*. The V-function can be derived by averaging the Q-values across all possible actions for a given state, weighted by their probabilities under the current policy.
*Source: [What is the Q function and what is the V function in reinforcement learning?](https://datascience.stackexchange.com/q/9832) — answer by Juan Leni, CC-BY-SA 4.0*
**The questioner is seeking confirmation that CNNs excel at deconstructing data into components while RNNs are better suited for combining those components, specifically in tasks like image captioning or translation.**
Both CNNs and RNNs fundamentally identify patterns, but they do so across different dimensions. CNNs detect consistent spatial patterns within a fixed input (like an image), whereas RNNs recognize sequential patterns over time or ordered steps – the context of previous inputs influences current processing. This difference in how they handle information makes CNNs good at feature extraction and RNNs effective for tasks requiring memory or understanding sequence.
*Source: [RNN vs CNN at a high level](https://datascience.stackexchange.com/q/11619) — answer by J. O'Brien Antognini, CC-BY-SA 4.0*
**The questioner understands ReLU avoids 'dying neurons' but struggles to reconcile this with its seemingly linear output, questioning how it can introduce necessary non-linearity for a neural network.**
While ReLU appears piecewise linear when examined in separate domains, it fundamentally *is* non-linear due to failing the mathematical definition of linearity. Its power comes from approximating complex functions through combinations of simple 'rectangular' outputs, similar to Riemann sums. The primary benefit over sigmoid isn’t avoiding neuron death, but preventing vanishing gradients by maintaining a derivative that doesn’t rapidly approach zero in deep networks.
*Source: [Why is ReLU used as an activation function?](https://datascience.stackexchange.com/q/26475) — answer by Tophat, CC-BY-SA 4.0*
**The questioner asks whether overfitting *always* makes a machine learning model worse, even if the data isn't complex, and seeks to understand why this happens beyond simply limiting generalization.**
Overfitting is fundamentally detrimental because it causes a model to learn irrelevant 'noise' in the training data alongside genuine patterns. This leads to high performance on the training set but significantly reduced accuracy when applied to new, unseen data. The gap between training and test performance directly reflects how much noise the model has incorporated, indicating its inability to reliably predict future outcomes. While limited generalization ability can exist independently, overfitting is a specific *way* that generalization fails.
*Source: [Why Is Overfitting Bad in Machine Learning?](https://datascience.stackexchange.com/q/61) — answer by Alex I, CC-BY-SA 4.0*
**The questioner wants to know best practices for updating a machine learning model with new data, specifically how often to retrain it and whether retraining on combined datasets constitutes overfitting.**
Model updates can be approached in three ways: continuously (online), periodically with all data (offline), or periodically with batches of new data. The optimal retraining frequency is tied to the variance between new and existing data – high variance suggests more frequent, smaller batch updates, while low variance allows for less frequent, larger batch updates. Treating batch size as a tunable hyperparameter helps balance adaptation to change with overall model stability.
*Source: [Should a model be re-trained if new observations are available?](https://datascience.stackexchange.com/q/12761) — answer by tombarti, CC-BY-SA 4.0*
**The questioner asks why machine learning models, particularly neural networks, are often called 'black boxes' despite engineers understanding their construction.**
The term 'black box' doesn’t refer to a lack of *how* the model is built, but rather our inability to fully understand *what* function the model ultimately learns. Simpler models like logistic regression or decision trees allow for tracing the logic behind predictions, but complex neural networks create functions so intricate that even experts can't grasp their complete behavior. This opacity makes them vulnerable to subtle input changes (adversarial examples) and highlights a fundamental limit in fully interpreting their reasoning.
*Source: [Why are Machine Learning models called black boxes?](https://datascience.stackexchange.com/q/22335) — answer by noe, CC-BY-SA 4.0*
**The question asks whether retraining a machine learning model with the entire dataset (including the test set) before deployment is always beneficial, especially considering the risk of performance degradation if the test set contains outliers.**
Machine learning inherently involves uncertainty; past performance on held-out data is never a guarantee of future results. While re-training on all available data carries some risk of decreased performance, it also offers the best *potential* for improvement and isn't statistically worse than using less data. Therefore, leveraging the entire dataset for the final production model is generally advisable to maximize the probability of success, accepting that unforeseen issues can still arise.
*Source: [Is it always better to use the whole dataset to train the final model?](https://datascience.stackexchange.com/q/33008) — answer by D.W., CC-BY-SA 4.0*
**The asker observes neural networks dominating many machine learning competitions, particularly in computer vision, and wonders if there are any areas where Bayesian Networks still achieve state-of-the-art results.**
Bayesian Networks excel when understanding *why* a prediction is made is crucial, as they offer inherent interpretability that 'black box' neural networks lack. They also shine when existing domain expertise can be formally incorporated into the model through prior probabilities. This makes them valuable in fields like medicine or situations demanding transparency and trust in the reasoning process.
*Source: [Is there any domain where Bayesian Networks outperform neural networks?](https://datascience.stackexchange.com/q/9818) — answer by MLgeek , CC-BY-SA 4.0*
**The user, new to TensorFlow and neural networks, struggles with selecting the appropriate cost function for different machine learning problems beyond simple cases like XOR or regression, finding tutorials lack explanation of *why* certain functions are chosen.**
Choosing a cost function is crucial for achieving desired results; start with Mean Squared Error for regression and error percentage for classification as defaults. However, optimal performance requires understanding your data, goals, and available tools to define a truly 'good' cost function. Be aware that even if two functions measure the same thing, their computational properties (like smoothness) can significantly impact training effectiveness – prioritize smooth functions for gradient-based optimization.
*Source: [Neural networks: which cost function to use?](https://datascience.stackexchange.com/q/9850) — answer by Winks, CC-BY-SA 4.0*
**The user is deciding whether to train a machine learning model on a dataset mirroring real-world class imbalance (mostly benign traffic) or a balanced dataset with equal representation of both benign and malicious traffic.**
The best approach depends on the goal. For descriptive modeling, use representative data; for predictive modeling, consider your algorithm's sophistication. Simpler algorithms benefit from balanced datasets to overcome inherent biases, while more advanced frameworks can often handle imbalance internally by adjusting sampling during training.
*Source: [Should I go for a 'balanced' dataset or a 'representative' dataset?](https://datascience.stackexchange.com/q/810) — answer by DSea, CC-BY-SA 4.0*
**The user is unsure whether to apply StandardScaler to the entire dataset before splitting into training and testing sets, or to fit the scaler only on the training set and then transform both sets.**
To avoid 'data leakage' and ensure a realistic evaluation of your model’s performance on truly unseen data, standardize *after* splitting. Fitting the scaler on the entire dataset introduces information from the test set into the training process, potentially biasing your model and leading to overly optimistic results. The goal is to simulate how the model would perform on completely new, independent data.
*Source: [StandardScaler before or after splitting data - which is better?](https://datascience.stackexchange.com/q/38395) — answer by redhqs, CC-BY-SA 4.0*
**The questioner wants to know best practices for updating a machine learning model with new data, specifically how often to retrain it and whether retraining on combined datasets constitutes overfitting.**
Model updates can be approached in three ways: continuously (online), periodically with all data (offline), or periodically with batches of new data. The optimal retraining frequency is tied to the variance between new and existing data – high variance suggests more frequent, smaller batch updates, while low variance allows for less frequent, larger batch updates. Treating batch size as a tunable hyperparameter helps balance adaptation to change with overall model stability.
*Source: [Should a model be re-trained if new observations are available?](https://datascience.stackexchange.com/q/12761) — answer by Hima Varsha, CC-BY-SA 4.0*
**The questioner asks if gradient descent is *always* guaranteed to converge to *some* optimum (local or global), even if it initially diverges from one, and seeks a counterexample if not.**
Gradient descent doesn't reliably find any optimum, especially in high-dimensional spaces common in deep learning. While it can move between optima, oscillation is more likely than convergence due to step size issues. Modern understanding shows that instead of clear local minima, models often encounter saddle points; extensive training and careful parameter tuning (like step size) are needed to navigate these complex landscapes.
*Source: [Does gradient descent always converge to an optimum?](https://datascience.stackexchange.com/q/24534) — answer by Green Falcon, CC-BY-SA 4.0*
**The user is asking why complex ensemble methods like XGBoost and Random Forest are necessary instead of simply using a single decision tree. They want to understand the benefits of combining multiple 'weak' learners.**
Ensemble methods improve model stability and accuracy by leveraging the power of many models. Techniques like bagging (Random Forest) reduce variance by creating diverse models from resampled data, making predictions less sensitive to small changes in the training set. Boosting techniques (like XGBoost & Gradient Boosting) further refine this by sequentially building models that correct the errors of their predecessors, reducing both bias and variance – a strategy proven effective in many real-world applications.
*Source: [Why do we need XGBoost and Random Forest?](https://datascience.stackexchange.com/q/23789) — answer by Ricardo Cruz, CC-BY-SA 4.0*
**The user is seeking methods to quantify the similarity between documents (like web pages), prioritizing semantic distance over simple lexical comparisons.**
Choosing a document distance metric involves balancing accuracy with complexity. While basic measures like cosine and Levenshtein distances are easy to implement, they lack nuanced understanding of meaning. More advanced techniques – topic modeling approaches like LDA and Pachinko Allocation, or vector-based methods like word2vec – capture semantic relationships but require more computational resources and potentially specialized implementation.
*Source: [What are some standard ways of computing the distance between documents?](https://datascience.stackexchange.com/q/678) — answer by indico, CC-BY-SA 4.0*
**The question asks about practical guidelines for choosing between Random Forest and Support Vector Machines (SVM) for classification, beyond simply relying on cross-validation.**
Random Forests excel with mixed data types, multiclass problems, and require less preprocessing. SVMs perform best when feature distances are meaningful and the dataset is relatively small due to computational limitations; they also offer interpretable 'support vectors'. While SVM generally outperforms Random Forest *when applicable*, consider scalability and ease of use alongside pure accuracy.
*Source: [When to use Random Forest over SVM and vice versa?](https://datascience.stackexchange.com/q/6838) — answer by lanenok, CC-BY-SA 4.0*
**The user is struggling to understand the practical difference between Keras `Dense` and `TimeDistributedDense` layers, despite understanding the basic definition of `TimeDistributedDense` applying a dense layer at each timestep.**
These layers differ in how they handle sequential data. A standard `Dense` layer flattens the entire sequence before processing, losing temporal information; it treats all timesteps as one large input. `TimeDistributedDense`, however, applies the same dense operation independently to *each* timestep of a sequence, preserving the order and relationships within the time series and allowing for focused interactions at each point in time.
*Source: [The difference between `Dense` and `TimeDistributedDense` of `Keras`](https://datascience.stackexchange.com/q/10836) — answer by Rizky Luthfianto, CC-BY-SA 4.0*
**The questioner is struggling to understand the concept of 'ground truth' in machine learning, specifically whether it refers to labels, target functions, or something else entirely.**
Ground truth represents the actual, measured value for the variable you’re trying to predict – and can usually be considered equivalent to a label. While often treated as definitive, ground truth isn't always perfect; measurements can contain errors, and in some cases, defining an 'objective truth' is inherently subjective. The quality of your ground truth data directly limits the performance of any machine learning model you build with it.
*Source: [What is Ground Truth](https://datascience.stackexchange.com/q/17839) — answer by Neil Slater, CC-BY-SA 4.0*
**The questioner, a data scientist with an R background and limited computer science fundamentals, is evaluating the practicality of learning C or C++ to improve performance and expand capabilities beyond what R offers.**
Rather than fully transitioning to C/C++, leverage those languages strategically by building extensions for existing tools like R or Python. Focus on optimizing only the most computationally intensive parts of your work in C/C++ while retaining the higher-level analytical strengths of languages designed for data science. This hybrid approach balances performance gains with development efficiency and access to a broader ecosystem of statistical and machine learning libraries.
*Source: [Data Science in C (or C++)](https://datascience.stackexchange.com/q/5357) — answer by Andre Holzner, CC-BY-SA 4.0*
**The questioner is unfamiliar with 'training warmup steps' in deep learning and wants to understand their purpose, particularly how they relate to the learning rate.**
Warmup steps are an initial period of training using a very low learning rate. This isn’t about helping the model ‘get used to’ the data, but rather allowing adaptive optimization algorithms (like Adam) to accurately estimate gradient statistics before applying larger updates. Starting with a small learning rate prevents potentially destabilizing initial parameter changes and allows for a smoother transition to the full learning rate schedule.
*Source: [In the context of Deep Learning, what is training warmup steps](https://datascience.stackexchange.com/q/55991) — answer by Mr Tsjolder from codidact, CC-BY-SA 4.0*
**The user is deciding whether to use validation loss or a performance metric (like accuracy) as the stopping criterion for their neural network's early stopping implementation.**
Prioritize monitoring the *loss* function during training over metrics like accuracy. Loss reflects the model’s confidence in its predictions, providing a more granular signal of learning progress than simply counting correct answers. Metrics based on hard classifications don't capture this nuanced information and can be misleading for early stopping.
*Source: [Early stopping on validation loss or on accuracy?](https://datascience.stackexchange.com/q/37186) — answer by qmeeus, CC-BY-SA 4.0*
**The questioner is confused about the purpose of a separate test set when a validation set already provides performance estimates on unseen data, and asks why we don't simply combine the test set with the training set for more data.**
The validation set is used to *tune* the model – selecting optimal hyperparameters or configurations. The test set serves as a final, unbiased evaluation of the fully-tuned model’s performance on completely unseen data. This separation prevents 'optimism bias' and allows for fair comparison between different modeling approaches because the test data wasn't involved in any part of the model selection process.
*Source: [Why use both validation set and test set?](https://datascience.stackexchange.com/q/18339) — answer by Pablo Suau, CC-BY-SA 4.0*
**The user wants to combine two independently built Keras models by connecting a layer from one model as input to another, effectively creating a single, unified model.**
Keras' functional API allows for flexible model construction where layers can accept inputs from multiple sources. By defining intermediate layers as outputs and then using those outputs as inputs to other parts of the combined model, you can 'splice' models together. The key is treating these connections as standard layer-to-layer data flow within a larger network.
*Source: [Merging two different models in Keras](https://datascience.stackexchange.com/q/26103) — answer by Rkz, CC-BY-SA 4.0*
**The questioner noticed inconsistent use of 'model hyperparameters' and 'model parameters' and asked for clarification, seeking to understand the distinction with examples.**
Model parameters are learned *during* training from the data itself – they define the model’s skill. Hyperparameters, conversely, are set *before* training and control aspects of the learning process; they aren't directly learned by the algorithm. While often grouped together when configuring a model, it's important to recognize that hyperparameters guide how parameters are found, not what those parameters ultimately become.
*Source: [What is the difference between model hyperparameters and model parameters?](https://datascience.stackexchange.com/q/14187) — answer by enterML, CC-BY-SA 4.0*
**The asker is deciding between cosine similarity and Euclidean distance to measure the similarity of contexts surrounding multi-word expressions in NLP, specifically considering two different ways to represent those contexts (concatenation vs. averaging word embeddings). They initially lean towards Euclidean distance.**
The core distinction lies in how each metric handles vector magnitude. Cosine similarity focuses on the angle between vectors, effectively ignoring scale differences, while Euclidean distance measures absolute distance and is sensitive to magnitude. If feature scales are consistent (like a fixed number of context words), Euclidean can be appropriate; however, if frequency matters or data is high-dimensional, cosine similarity becomes more robust because it normalizes for length. Ultimately, both methods suffer in very high dimensions, and learning a custom metric might yield the best results.
*Source: [When to use cosine simlarity over Euclidean similarity](https://datascience.stackexchange.com/q/27726) — answer by Martin Thoma, CC-BY-SA 4.0*
**The user is asking for a systematic way to determine the optimal number of layers and nodes within those layers when designing a neural network, beyond what's dictated by input/output data.**
Determining neural network architecture is largely empirical; there’s no formulaic solution. Start with informed guesses based on similar problems you've encountered or read about, then systematically experiment with different configurations while considering how architectural choices interact with other hyperparameters like activation functions and regularization techniques. Don't be afraid to begin with a simple model and incrementally increase complexity, but always prioritize testing and refinement over adhering strictly to 'rules of thumb'.
*Source: [How to decide neural network architecture?](https://datascience.stackexchange.com/q/20222) — answer by Neil Slater, CC-BY-SA 4.0*
**The user is confused about whether to use the `.best_estimator_` attribute or the entire `GridSearchCV` object for making predictions after performing a grid search in scikit-learn, noticing discrepancies in metrics when using one versus the other.**
Both returning `.best_estimator_` (the optimized model instance) and returning the `GridSearchCV` object itself will ultimately yield the same prediction results because the `GridSearchCV` object internally uses the best estimator when making predictions. The choice depends on whether you need access to additional grid search information like optimal parameters; if only the trained model is needed, `.best_estimator_` provides a cleaner interface. The key takeaway is that `GridSearchCV`, by default, 'refits' and stores the best performing estimator internally.
*Source: [How to use the output of GridSearch?](https://datascience.stackexchange.com/q/21877) — answer by Dee Carter, CC-BY-SA 4.0*
**The questioner is unfamiliar with 'training warmup steps' in deep learning and wants to understand their purpose, particularly how they relate to the learning rate.**
Warmup steps are an initial period of training using a very low learning rate. This isn’t about helping the model ‘get used to’ the data, but rather allowing adaptive optimization algorithms (like Adam) to accurately estimate gradient statistics before applying larger updates. Starting with a small learning rate prevents potentially destabilizing initial parameter changes and allows for a smoother transition to the full learning rate schedule.
*Source: [In the context of Deep Learning, what is training warmup steps](https://datascience.stackexchange.com/q/55991) — answer by Ron Schwessinger, CC-BY-SA 4.0*
**The user is questioning whether using K-means clustering with Euclidean distance on latitude/longitude coordinates is appropriate for geolocation data.**
K-means isn't ideal because it minimizes variance based on straight-line distances, which doesn’t account for the curvature of the Earth. Geolocation data requires algorithms that can utilize more accurate geodetic distance functions like Haversine. Algorithms such as hierarchical clustering or DBSCAN are better suited since they aren't limited to linear distance metrics and can handle non-linear spaces.
*Source: [Clustering geo location coordinates (lat,long pairs)](https://datascience.stackexchange.com/q/761) — answer by mike1886, CC-BY-SA 4.0*
**The user asks why training and testing a machine learning model on the same data is problematic, specifically questioning if it leads to memorization instead of genuine understanding.**
Evaluating a model with the same data used for training doesn't reveal its ability to generalize to new, unseen information. True assessment requires independent test data to verify that the model has learned underlying patterns and principles, rather than simply memorizing specific examples. This mirrors effective teaching practices where evaluation focuses on comprehension, not rote recall.
*Source: [Why is it wrong to train and test a model on the same dataset?](https://datascience.stackexchange.com/q/86632) — answer by hH1sG0n3, CC-BY-SA 4.0*
## Python
**The user is confused about the distinction between `fit` and `fit_transform` methods in scikit-learn, specifically why data transformation is necessary and how it applies to both training and testing datasets.**
Data transformations like scaling or encoding are performed using statistics *calculated from* the training data. The `fit` method learns these parameters (like mean and standard deviation) from the training set, while `transform` then applies those learned parameters to transform any dataset – including test sets – ensuring consistency. `fit_transform` is a convenience function that combines learning the transformation parameters and applying them in one step, but it should only be used on the training data.
*Source: [What's the difference between fit and fit_transform in scikit-learn models?](https://datascience.stackexchange.com/q/12321) — answer by K3---rnc, CC-BY-SA 4.0*
**A user is confused about the difference between `isna()` and `isnull()` in Pandas, specifically how they detect missing values and why both methods exist.**
While seemingly redundant, `isna()` and `isnull()` perform identically in Pandas because of historical reasons related to mimicking R's DataFrame structure. R distinguishes between 'NA' and 'NULL', but Pandas (built on NumPy) uses 'NaN' ('Not a Number') for missing data. The two function names are therefore relics of the original design goal to align with R, despite not being functionally different in Python.
*Source: [Difference between isna() and isnull() in pandas](https://datascience.stackexchange.com/q/37878) — answer by Djib2011, CC-BY-SA 4.0*
**An experienced SQL user questions why Pandas is so popular for data manipulation, given its perceived complexity compared to SQL's efficiency and features like optimization and clear error messages.**
Pandas prioritizes a developer-friendly workflow that allows for iterative exploration and debugging of data transformations with single lines of code. While SQL excels at persistent storage and complex queries, Pandas integrates seamlessly into the broader Python ecosystem for quick analysis, visualization, and 'one-and-done' tasks without requiring database setup. Ultimately, many users choose Pandas due to its ease of use within Python and a lack of comprehensive SQL knowledge, even when SQL might be more suitable for repetitive or large-scale data operations.
*Source: [Why do people prefer Pandas to SQL?](https://datascience.stackexchange.com/q/34357) — answer by cvonsteg, CC-BY-SA 4.0*
**The user is confused about how to handle variable length sequences when training an RNN in Keras, specifically regarding the expected input shape and whether padding limits prediction lengths.**
Keras RNN layers can inherently process variable-length sequences within a batch by using `None` for the timestep dimension. While all sequences *within* a single batch must have the same length (achieved through padding), different batches can contain sequences of varying lengths, and inference can be performed on any sequence length. This flexibility allows you to avoid limiting your model's ability to handle inputs longer than those used during training.
*Source: [Training an RNN with examples of different lengths in Keras](https://datascience.stackexchange.com/q/26366) — answer by kbrose, CC-BY-SA 4.0*
**The user is experiencing indefinite execution times when training a Support Vector Regression (SVR) model in scikit-learn with a relatively large dataset (over half a million rows). They've tried adjusting parameters and different environments without success.**
Kernel-based SVMs have high computational complexity that scales poorly with data size, potentially leading to performance bottlenecks. Increasing the cache size can help, but for very large datasets, consider using techniques like data subsampling or switching to linear models. Approximating the kernel function (e.g., via k-means clustering) offers a way to reduce complexity while retaining some of the benefits of kernel methods.
*Source: [SVM using scikit learn runs endlessly and never completes execution](https://datascience.stackexchange.com/q/989) — answer by Jessica Collins, CC-BY-SA 4.0*
**The user encountered a ValueError during prediction with their RandomForestClassifier due to NaN (Not a Number) values in the test dataset and wants to identify and handle these missing values without removing the affected records.**
When dealing with machine learning models, data quality is crucial; unexpected values like NaNs can cause errors. The key is to proactively *inspect* your data for such issues using tools like `np.isnan()` to locate them. Rather than simply dropping problematic rows, consider imputation techniques – replacing missing values with a reasonable estimate (mean, median, or other) – using libraries like scikit-learn's `SimpleImputer` or pandas’ built-in fill methods.
*Source: [ValueError: Input contains NaN, infinity or a value too large for dtype('float32](https://datascience.stackexchange.com/q/11928) — answer by fernando, CC-BY-SA 4.0*
**The user successfully built and ran a Python machine learning application on their local machine, but the same code fails when deployed to another system, leading them to suspect dependency version conflicts.**
Reproducible environments are crucial for deploying Python applications. The best practice is to explicitly define all dependencies using an environment file (like `environment.yml` with conda) which can then be used to recreate the exact same setup on any machine. Utilizing virtual environments isolates project dependencies, preventing conflicts and ensuring consistency across different systems.
*Source: [How to clone Python working environment on another machine?](https://datascience.stackexchange.com/q/24093) — answer by ginge, CC-BY-SA 4.0*
**The user is confused about `torch.no_grad()` in PyTorch, specifically why it's used even when a variable already has `requires_grad=True`. They are unsure what 'being tracked by autograd' means.**
PyTorch’s automatic differentiation (`autograd`) system tracks operations on tensors with `requires_grad=True` to calculate gradients. Using `torch.no_grad()` temporarily disables this tracking, reducing memory consumption and computation time when you only need forward passes (like evaluating a model or updating values without gradient calculation). It's useful when certain calculations don’t contribute to the learning process and shouldn't be included in backpropagation.
*Source: [What is the use of torch.no_grad in pytorch?](https://datascience.stackexchange.com/q/32651) — answer by Adrien D, CC-BY-SA 4.0*
**The user is seeking advice on establishing a reproducible data science workflow in Python, specifically focusing on version control for both code and data, tools to track experiments, a suggested project structure, and automation options.**
Prioritize universal tools over language-specific ones to maintain flexibility as projects evolve. While specialized RR tools exist (like Taverna, Kepler, or VisTrails), leveraging existing version control systems like Git for both code *and* data is often the most practical starting point. Combining this with literate programming approaches using tools like LaTeX, Sweave, or knitr can enhance documentation and reproducibility by integrating code, results, and narrative.
*Source: [Tools and protocol for reproducible data science using Python](https://datascience.stackexchange.com/q/758) — answer by Aleksandr Blekh, CC-BY-SA 4.0*
**The user is encountering an error when using `train_test_split()` because their feature array (X) and target array (Y) have incompatible shapes – a mismatch in the number of samples.**
Data preparation for machine learning requires careful attention to dimensionality. The core issue was an extra, unnecessary dimension introduced during the creation of the feature array, leading to shape incompatibility with the target variable. Correcting this involves reshaping or transposing the data to ensure that both arrays have a consistent number of samples along the appropriate axis.
*Source: [train_test_split() error: Found input variables with inconsistent numbers of sam](https://datascience.stackexchange.com/q/20199) — answer by tuomastik, CC-BY-SA 4.0*
**The user has split their data into ten PySpark DataFrames using `randomSplit` and needs to combine nine of them into a single DataFrame repeatedly for manual cross-validation, but the built-in `unionAll` function only accepts two arguments.**
When dealing with operations that require combining multiple items in a loop (like merging dataframes), leverage Python's functional programming tools like `reduce` to apply the operation iteratively. However, consider if adding an identifying label column and filtering is more efficient than repeated unions, especially when performing cross-validation or similar tasks where you need different subsets of your data.
*Source: [Merging multiple data frames row-wise in PySpark](https://datascience.stackexchange.com/q/11356) — answer by Jan van der Vegt, CC-BY-SA 4.0*
**The user has a large (18GB) CSV file they want to load into a pandas DataFrame for machine learning, but are encountering memory errors despite having 32GB of RAM.**
Attempting to load an entire massive dataset into memory at once is often impractical. Instead of increasing hardware or switching to a database immediately, leverage tools like `pandas.read_csv` with the `chunksize` parameter to process data iteratively in smaller, manageable pieces. This allows you to perform analysis and training without exceeding your RAM capacity.
*Source: [Opening a 20GB file for analysis with pandas](https://datascience.stackexchange.com/q/27767) — answer by Olel Daniel, CC-BY-SA 4.0*
**The user wants to perform linear regression in scikit-learn but needs a way to constrain the resulting feature weights to be non-negative.**
This problem is solved with Non-negative Least Squares (NNLS) which frames weight estimation as a constrained optimization. While not directly available within scikit-learn currently, the SciPy library provides an implementation of NNLS that can address this need. It's important to be aware that community contributions for direct integration into Scikit-Learn are still in progress.
*Source: [How to force weights to be non-negative in Linear regression](https://datascience.stackexchange.com/q/18258) — answer by Dawny33, CC-BY-SA 4.0*
**The user wants to efficiently count the number of missing values in each row of a Pandas DataFrame and then potentially split the DataFrame based on these counts.**
When working with Pandas, vectorized operations are significantly faster than iterating through rows or columns using functions like `apply`. Leverage built-in methods like `isnull().sum(axis=1)` to perform calculations across an entire axis at once. This approach avoids performance bottlenecks and makes your code more scalable for larger datasets.
*Source: [How to count the number of missing values in each row in Pandas dataframe?](https://datascience.stackexchange.com/q/12645) — answer by Icyblade, CC-BY-SA 4.0*
**The user is attempting to calculate KL Divergence between lists of numbers in Python using `sklearn.metrics.mutual_info_score` but consistently receives the same result regardless of input data.**
The function used actually calculates mutual information, not KL divergence directly, and expects probability distributions as input – meaning the values should sum to one. KL Divergence is sensitive to improper probability distributions (values not summing to 1) and can produce undefined results like division by zero. When calculating these types of divergences, be mindful of the base of the logarithm used, as it affects interpretation but provides a constant scaling factor.
*Source: [Calculating KL Divergence in Python](https://datascience.stackexchange.com/q/9262) — answer by Has QUIT--Anony-Mousse, CC-BY-SA 4.0*
## Deep Learning
**The user is asking for a practical example of how to implement class weights in Keras to address imbalanced datasets, specifically how to use the `class_weights` parameter during model fitting.**
When dealing with imbalanced classes, leverage existing tools like scikit-learn's `class_weight` module to automatically calculate appropriate weights based on class frequencies. This avoids manual weight tuning and ensures minority classes are proportionally represented in the learning process. Remember to avoid naming conflicts by using a distinct variable name for your calculated weights compared to the imported module.
*Source: [How to set class weights for imbalanced classes in Keras?](https://datascience.stackexchange.com/q/13490) — answer by PSc, CC-BY-SA 4.0*
**The user is asking for a practical example of how to implement class weights in Keras to address imbalanced datasets, specifically how to use the `class_weights` parameter during model fitting.**
When dealing with imbalanced classes, leverage existing tools like scikit-learn's `class_weight` module to automatically calculate appropriate weights based on class frequencies. This avoids manual weight tuning and ensures minority classes are proportionally represented in the learning process. Remember to avoid naming conflicts by using a distinct variable name for your calculated weights compared to the imported module.
*Source: [How to set class weights for imbalanced classes in Keras?](https://datascience.stackexchange.com/q/13490) — answer by layser, CC-BY-SA 4.0*
**The questioner is trying to understand why one would choose a Gated Recurrent Unit (GRU) over a Long Short-Term Memory network (LSTM), given LSTM's more complex and seemingly controllable structure.**
While LSTMs offer finer-grained control through their three gates, GRUs often achieve comparable performance with greater computational efficiency due to their simpler design. The core function of both is to address the vanishing gradient problem in recurrent networks, but GRUs accomplish this without a dedicated memory unit. Therefore, when resources are limited or speed is critical, a GRU can be a strong alternative to an LSTM.
*Source: [When to use GRU over LSTM?](https://datascience.stackexchange.com/q/14581) — answer by Abhishek, CC-BY-SA 4.0*
**The questioner is confused about the distinction between 'equivariant to translation' and 'invariant to translation', particularly as described in the context of convolutional neural networks.**
While often used interchangeably, these terms have distinct meanings rooted in mathematical principles. Invariance means a feature *doesn’t change* under a transformation (like translation), while equivariance means it *changes predictably* alongside the transformation – its form is preserved even as its position shifts. Understanding this difference helps clarify how network layers process information and respond to input variations, with pooling aiming for invariance and convolutions striving for equivariance.
*Source: [What is the difference between "equivariant to translation" and "invariant to tr](https://datascience.stackexchange.com/q/16060) — answer by Laurent Duval, CC-BY-SA 4.0*
**The questioner observes that ResNet uses both normal and uniform He initialization in different layers and seeks clarification on *why* one might be preferred over the other, especially considering Batch Normalization's impact on reducing sensitivity to initialization.**
Despite theoretical analyses suggesting variance is key regardless of distribution type (normal or uniform), the choice between them appears largely empirical and based on historical precedent. The original papers don’t strongly justify normal vs. uniform; it seems to have been a matter of convention, potentially influenced by early network architectures like AlexNet. Batch Normalization diminishes the importance of precise initialization schemes, making the distinction less critical – it's more of an established practice than a theoretically-driven necessity.
*Source: [When to use (He or Glorot) normal initialization over uniform init? And what are](https://datascience.stackexchange.com/q/13061) — answer by tlorieul, CC-BY-SA 4.0*
**The user is facing memory limitations while training a large LSTM network and wants to know if reducing the `batch_size` parameter in Keras will negatively impact the quality of their model's predictions.**
Batch size directly influences the learning process by affecting gradient estimation; larger batches provide more stable, but potentially less exploratory, gradients. Smaller batch sizes introduce noise that can help escape local minima, but excessively small batches can lead to unstable training or slow convergence. Utilizing GPU parallelism is significantly more efficient with larger batches, and techniques exist to simulate larger batches when memory constraints are present.
*Source: [Does batch_size in Keras have any effects in results' quality?](https://datascience.stackexchange.com/q/12532) — answer by Jan van der Vegt, CC-BY-SA 4.0*
**The user is trying to decide between ARIMA and LSTM for univariate time series forecasting, having already experimented with both, and seeks guidance on how to best compare them beyond the basic characteristics they've identified.**
Effective time series prediction is inherently difficult due to noise and complex underlying factors; focusing solely on model selection can overshadow a robust data science process like cross-validation. Start with simpler, interpretable models like ARIMA (including seasonal variations) to understand data characteristics before moving to more complex methods such as LSTM. While LSTMs *can* be powerful, they demand significant resources, data, and tuning – making them potentially less efficient for initial exploration or smaller datasets.
*Source: [Time series prediction using ARIMA vs LSTM](https://datascience.stackexchange.com/q/12721) — answer by AN6U5, CC-BY-SA 4.0*
**The user observed differing accuracy and loss values across three deep learning models for multi-class classification and questioned the relationship between these metrics, specifically why a higher accuracy could coincide with a higher loss.**
Accuracy and loss are distinct measures of model performance; one doesn't directly imply the other. Loss represents the *magnitude* of errors – how far off predictions are from correct values – while accuracy reflects the *frequency* of those errors. A model can achieve high accuracy by correctly classifying most data, but still have a higher loss if its mistakes are significantly larger than those made by another model. Ultimately, interpreting loss requires considering both the scale of your data and the specific consequences of errors within your problem domain.
*Source: [What is the relationship between the accuracy and the loss in deep learning?](https://datascience.stackexchange.com/q/42599) — answer by Jérémy Blain, CC-BY-SA 4.0*
**The user wants to understand how to calculate the number of trainable parameters within a single LSTM layer. This is important for estimating training resource needs (data and time).**
LSTM layers contain multiple weight matrices – one set for each gate (input, forget, output) and cell state update – that determine the model's complexity. The total parameter count depends on the dimensions of these matrices, specifically relating to the input and hidden state sizes. Understanding this calculation allows practitioners to better assess model size and plan accordingly for training.
*Source: [Number of parameters in an LSTM model](https://datascience.stackexchange.com/q/10615) — answer by wabbit, CC-BY-SA 4.0*
**The user is struggling with underfitting in a deep neural network, consistently performing worse than a Random Forest model despite using a well-established architecture and pre-training techniques.**
Successfully training deep networks requires extensive hyperparameter tuning due to the complex interplay between parameters. Focus on systematically experimenting with a known working example to build intuition about how changes impact performance, and critically evaluate each layer's ability to learn meaningful representations before moving forward. Debugging should be approached sequentially, starting from the initial layers.
*Source: [How to fight underfitting in a deep neural net](https://datascience.stackexchange.com/q/731) — answer by ffriend, CC-BY-SA 4.0*
**The user wants to know how to leverage multiple GPUs within a Keras (TensorFlow) model for faster training, specifically on an EC2 instance with eight GPUs.**
Keras provides a `multi_gpu_model` utility that enables data parallelism – distributing the workload across available GPUs by processing different subsets of the data simultaneously. Modern versions of Keras can automatically detect and utilize all available GPUs without needing to specify the number explicitly, though you can still define it if desired. Monitoring GPU utilization with tools like `nvidia-smi` is crucial to confirm that all resources are being effectively used during training.
*Source: [Multi GPU in Keras](https://datascience.stackexchange.com/q/23895) — answer by weiji14, CC-BY-SA 4.0*
**The questioner is struggling to understand Noise Contrastive Estimation (NCE), particularly how it differs from and compares to Negative Sampling within the context of word embeddings like Word2Vec. They seek an intuitive explanation beyond the mathematical formulas.**
NCE addresses the computational cost of softmax in large vocabulary models by reframing a multi-class prediction problem as a series of binary classification tasks. Instead of directly predicting the next word, the model learns to discriminate between real word pairs and artificially created 'noisy' pairs. Negative Sampling is a specific implementation of NCE that uses a particular strategy for generating these noisy samples, optimizing speed by focusing on a subset of the vocabulary.
*Source: [Intuitive explanation of Noise Contrastive Estimation (NCE) loss?](https://datascience.stackexchange.com/q/13216) — answer by user154812, CC-BY-SA 4.0*
**The questioner asks whether time series stationarity—a requirement for traditional models like ARIMA—is also important when using LSTMs (a type of RNN) for forecasting, and why.**
The core principle in any machine learning task is ensuring the training data represents the expected test data; stationarity helps achieve this in time series. While simpler models require stationary data to reliably apply learned relationships across different segments of a time series, LSTMs' ability to learn complex, non-linear patterns and long-term dependencies makes them more robust to non-stationarity. However, even with RNNs, significant shifts between training and test data distributions can degrade performance, so sufficient data is crucial for the model to generalize effectively.
*Source: [Time Series prediction using LSTMs: Importance of making time series stationary](https://datascience.stackexchange.com/q/24800) — answer by tom, CC-BY-SA 4.0*
## Numbers
**The questioner struggles to consistently choose between `float` and `double` data types for representing real numbers in their code, lacking a clear rationale beyond subjective feelings or memory concerns.**
Prioritize `double` as the default floating-point type due to its greater precision and wider range; it's also the standard return type of many math functions. Only consider `float` if you have a large dataset *and* can confidently prove that reduced accuracy won’t impact your results. `long double` is for specialized cases where even more precision is needed, but platform support varies.
*Source: [When do you use float and when do you use double](https://softwareengineering.stackexchange.com/q/188721) — answer by Bart van Ingen Schenau, CC-BY-SA 4.0*
**The questioner is puzzled why Java still includes `float` despite `double` being generally recommended, especially given the performance constraints in game development frameworks like Libgdx which require using `float`.**
Sometimes technical choices aren't about absolute superiority but practical optimization for a specific context. In high-performance applications like games, minimizing data size and maximizing processing speed are critical, even if it means sacrificing some precision. Utilizing smaller data types like `float` reduces memory usage and bandwidth demands, leading to gains in overall performance—particularly when leveraging GPU acceleration which is often optimized for single-precision floating point numbers.
*Source: [Why are floats still part of the Java language when doubles are mostly recommend](https://softwareengineering.stackexchange.com/q/316822) — answer by Philipp, CC-BY-SA 4.0*
**The questioner asks if there are numbers representable in base-2 (binary) that *cannot* be precisely represented in base-10 (decimal), given C#'s `decimal` type is designed for exact decimal representation and floats/doubles approximate.**
A number's precise representation in any base depends on its factors relative to the base. If a number’s denominator contains prime factors *not* present in the base, it will result in a repeating (non-terminating) expansion. While all numbers representable exactly in binary can also be represented exactly in decimal with enough digits, the reverse isn't always true; some decimals have infinite repeating binary representations due to their prime factorization.
*Source: [Are there numbers that are not representable in base 10 but can be represented i](https://softwareengineering.stackexchange.com/q/237014) — answer by Max, CC-BY-SA 4.0*
**The questioner noticed some programming languages use 'banker's rounding' – rounding to the nearest *even* integer when a number is exactly halfway between two integers – and wants to understand the rationale behind this seemingly unusual approach.**
This rounding method, known as banker’s rounding, isn’t about mathematical simplification but rather minimizing bias in calculations involving many rounded numbers. By alternating between rounding up and down for values equidistant from whole numbers, it prevents systematic errors that could accumulate significantly over time, particularly important in financial applications where even small discrepancies matter. It's a practical solution to ensure fairness and accuracy when dealing with repeated rounding operations.
*Source: [Why do some languages round to the nearest EVEN integer?](https://softwareengineering.stackexchange.com/q/256265) — answer by Loren Pechtel, CC-BY-SA 4.0*
**The questioner noticed some programming languages use 'banker's rounding' – rounding to the nearest *even* integer when a number is exactly halfway between two integers – and wants to understand the rationale behind this seemingly unusual approach.**
This rounding method, known as banker’s rounding, isn’t about mathematical simplification but rather minimizing bias in calculations involving many rounded numbers. By alternating between rounding up and down for values equidistant from whole numbers, it prevents systematic errors that could accumulate significantly over time, particularly important in financial applications where even small discrepancies matter. It's a practical solution to ensure fairness and accuracy when dealing with repeated rounding operations.
*Source: [Why do some languages round to the nearest EVEN integer?](https://softwareengineering.stackexchange.com/q/256265) — answer by hobbs, CC-BY-SA 4.0*
**The user wants to efficiently store numeric ranges (a start and end value) using the fewest bits possible, recognizing that storing two separate numbers might be redundant given their inherent relationship.**
While clever compression seems unlikely to yield significant savings, a mathematical analysis reveals the theoretical limits of range storage. The number of possible ranges within a given set is predictable, and this dictates the minimum bit requirement; in many cases, representing the range still requires nearly as much space as storing the endpoints individually. The benefit of encoding a range instead of two numbers is minimal—typically only one bit saved—making it an impractical optimization for most scenarios.
*Source: [What is the most efficient way to store a numeric range?](https://softwareengineering.stackexchange.com/q/390192) — answer by Glorfindel, CC-BY-SA 4.0*
**The questioner wants to know if Little Endian has definitively 'won' the historical Big vs. Little Endian debate, given current OS/architecture trends and network protocols.**
While Little Endian dominates desktop computing due to x86’s success, the issue is largely irrelevant for most modern developers because of extensive abstraction layers in software. Object-oriented programming and virtual machine-based languages further shield programmers from low-level details like endianness, making it a non-concern for daily work. The problem has effectively faded into obscurity as development practices moved away from 'coding to the metal'.
*Source: [Has Little Endian won?](https://softwareengineering.stackexchange.com/q/165899) — answer by Ellen Spertus, CC-BY-SA 4.0*
## Markets
**The questioner wonders why a company would be concerned with its stock price after initial sale, reasoning that transactions between buyers and sellers shouldn't affect the company once it has received payment.**
A company’s share price remains important beyond the initial public offering because they frequently issue additional shares to raise capital. A higher share price allows them to raise funds with less dilution of existing ownership. Furthermore, stock value is crucial when companies use equity as part of an acquisition deal, determining relative valuations and exchange rates between shares.
*Source: [Why would a company care about the price of its own shares in the stock market?](https://money.stackexchange.com/q/7800) — answer by Chris W. Rea, CC-BY-SA 4.0*
**The questioner wonders why a company would be concerned with its stock price after initial sale, reasoning that transactions between buyers and sellers shouldn't affect the company once it has received payment.**
A company’s share price remains important beyond the initial public offering because they frequently issue additional shares to raise capital. A higher share price allows them to raise funds with less dilution of existing ownership. Furthermore, stock value is crucial when companies use equity as part of an acquisition deal, determining relative valuations and exchange rates between shares.
*Source: [Why would a company care about the price of its own shares in the stock market?](https://money.stackexchange.com/q/7800) — answer by Fixee, CC-BY-SA 4.0*
**The questioner was confused by reports of negative oil prices and wanted to understand if it meant people were literally being paid to take oil.**
Negative pricing in this case wasn't about the inherent value of oil, but a consequence of how futures contracts work when physical storage capacity is exhausted. The price reflected desperation from contract holders to *avoid* taking delivery of an asset they couldn’t store or use. This highlights that financial instruments are tied to real-world logistics and can be dramatically affected by supply chain limitations.
*Source: [What does it mean for the price of oil to be negative?](https://money.stackexchange.com/q/124269) — answer by D Stanley, CC-BY-SA 4.0*
**A new investor is puzzled by the stock market’s relatively muted response to overwhelmingly negative economic news, expecting a more significant and sustained downturn given the severity of the situation.**
Market movements aren't always directly linked to news cycles because investors interpret information differently based on their own beliefs and expectations. There will always be conflicting opinions about the future impact of events, making it impossible for markets to perfectly 'price in' all available data at any given time. Historical precedents show that initial market reactions can be misleading, with rallies sometimes occurring even during prolonged downturns; therefore, predicting a definitive bottom is extremely difficult.
*Source: [Why isn't the market dropping like a stone with all the bad news?](https://money.stackexchange.com/q/123473) — answer by Hart CO, CC-BY-SA 4.0*
**The user is confused about 'maker' and 'taker' fees on cryptocurrency exchanges, specifically how their trading activity determines which fee structure applies.**
These fees differentiate between order types based on liquidity provision. Placing a limit order that isn’t immediately filled (and waits to be matched) designates you as a ‘maker,’ incentivizing you to add volume to the exchange. Conversely, executing an immediate trade via a market order or a limit order that instantly matches existing orders makes you a ‘taker,’ reflecting your consumption of available liquidity.
*Source: [Explain maker taker fees](https://money.stackexchange.com/q/63044) — answer by user41221, CC-BY-SA 4.0*
**A new FOREX trader is confused by sudden, large price swings (pips) and wants to understand what causes them and who has the power to create such movements.**
FOREX trading differs fundamentally from stock investing because it's a zero-sum game – one trader’s gain is another’s loss. This necessitates short-term, speculative 'day-trading', which carries significant risk. Highly efficient FOREX markets rapidly incorporate public information, making profitable timing extremely difficult for individual traders competing against large institutional investors who react instantly to news.
*Source: [What is the cause of sudden price spikes in the FOREX market?](https://money.stackexchange.com/q/76480) — answer by Grade 'Eh' Bacon, CC-BY-SA 4.0*
## Neural Network
**The questioner is confused about what 'deconvolutional layers' (also called transposed convolutions) do in the context of fully convolutional networks, specifically how they perform upsampling.**
Deconvolutional layers aren’t actually ‘deconvolving’ anything; they are effectively a convolution operation performed with learned filters to *increase* spatial resolution. They achieve this by strategically inserting zeros between input values and then performing a standard convolution. This allows the network to learn how best to upsample data, rather than relying on fixed interpolation methods like bilinear sampling.
*Source: [What are deconvolutional layers?](https://datascience.stackexchange.com/q/6107) — answer by David Dao, CC-BY-SA 4.0*
**The questioner is struggling to understand how backpropagation works through a max-pooling layer in a neural network, specifically how gradients are calculated when the max function isn't directly differentiable.**
During backpropagation through max-pooling, only the neuron that *produced* the maximum value receives the gradient from the next layer. All other neurons contributing to the max-pool receive zero gradient because changes to their inputs wouldn’t affect the output. This effectively routes the error signal only along the path of the maximum activation, simplifying the gradient calculation.
*Source: [Backprop Through Max-Pooling Layers?](https://datascience.stackexchange.com/q/11699) — answer by abora, CC-BY-SA 4.0*
**The user is questioning which loss function – `sparse_categorical_crossentropy` or `categorical_crossentropy` – to use in Keras for multi-class classification and how it impacts accuracy.**
Choose your loss function based on *how* you've encoded your target labels. If each data point belongs to only one class, using integer encoding with `sparse_categorical_crossentropy` is more efficient because it avoids unnecessary computation compared to one-hot encoding required by `categorical_crossentropy`. The underlying mathematical formula is equivalent, so the choice shouldn’t affect model accuracy; it's primarily a matter of computational efficiency and data representation.
*Source: [Sparse_categorical_crossentropy vs categorical_crossentropy (keras, accuracy)](https://datascience.stackexchange.com/q/41921) — answer by featuredpeow, CC-BY-SA 4.0*
**The asker wants to understand how a 1x1 convolution can be equivalent to a fully connected layer, specifically relating to Yan LeCun's claim that there are no true fully connected layers in convolutional networks. They provide a simple example of a fully connected network and ask for the corresponding convolutional implementation.**
The core idea is that a fully connected layer performs a weighted sum of inputs followed by an activation, which can be replicated with 1x1 convolutions. By reshaping input features into spatial dimensions (even if just 1x1), you can treat them as 'image' channels and apply 1x1 convolutional filters to achieve the same matrix multiplication and bias addition as a fully connected layer. The 'full connection table' refers to kernels that span the entire depth of the input feature map, effectively connecting every input to every output.
*Source: [How are 1x1 convolutions the same as a fully connected layer?](https://datascience.stackexchange.com/q/12830) — answer by MarvMind, CC-BY-SA 4.0*
**The user has a classification problem with significantly imbalanced classes and wants to know how to best handle this imbalance when choosing a loss function in PyTorch, specifically focusing on identifying minority classes representing deviations from normal behavior.**
When dealing with imbalanced datasets for classification, starting with CrossEntropyLoss and applying class weighting is a solid initial strategy. Weights should be calculated inversely proportional to class frequency – giving higher weight to less frequent classes. While techniques like oversampling (using `WeightedRandomSampler`) can achieve similar results to weighted loss functions, adjusting the loss function itself is often a straightforward and effective first step.
*Source: [What loss function to use for imbalanced classes (using PyTorch)?](https://datascience.stackexchange.com/q/48369) — answer by Esmailian, CC-BY-SA 4.0*
## Price
**The user is confused about the difference between bid, ask, and current stock prices displayed on their brokerage account, specifically why they would pay a higher 'ask' price than the listed 'current' price.**
Displayed prices represent potential transactions, not necessarily what has already happened. The 'bid' is the highest price buyers are *currently willing to pay*, while the 'ask' is the lowest price sellers are *currently willing to accept*. The current price reflects the last completed trade, and a new purchase will likely occur at the prevailing ask price if executed immediately.
*Source: [Can someone explain a stock's "bid" vs. "ask" price relative to "current" price?](https://money.stackexchange.com/q/1063) — answer by Chris W. Rea, CC-BY-SA 4.0*
**The questioner is confused why the stock market's opening price each day isn't identical to the previous day's closing price, observing this discrepancy in candlestick charts and wondering what activity occurs during the intervening time.**
Stock prices aren't fixed like retail goods; they represent the *last* traded price which is constantly subject to change based on new information. Global markets operate around the clock, meaning events occurring outside of US trading hours (news, economic data, other market movements) immediately impact perceived value. Therefore, a significant amount of activity and re-evaluation happens between closing and opening bells, making an identical open/close price highly improbable.
*Source: [In the stock market, why is the "open" price value never the same as previous da](https://money.stackexchange.com/q/3765) — answer by Andrew, CC-BY-SA 4.0*
**The questioner wonders if a stock's price can increase without any actual purchases being made, even if the company itself is performing well.**
Stock prices aren’t determined by potential or expectation; they are solely based on completed transactions. A reported price only exists when a buyer and seller agree on a value, meaning no trade equals no current price. Essentially, price reflects actual exchange, not just positive sentiment or company performance.
*Source: [Can the stock price go up even if no one is buying?](https://money.stackexchange.com/q/90071) — answer by JoeTaxpayer, CC-BY-SA 4.0*
**The questioner wondered if they could profit from the briefly negative oil prices by either taking physical possession of crude oil or artificially increasing demand for gasoline through excessive driving.**
Seizing on unusual market events requires understanding logistical and regulatory realities. While a theoretical price anomaly might *seem* like an opportunity, practical barriers – such as storage requirements, industry-scale transactions, and established supply chains – often prevent individuals from capitalizing on it. Successful business ventures aren't just about identifying a gap, but also assessing your ability to realistically fill it.
*Source: [Could I make money off of the negative oil price?](https://money.stackexchange.com/q/124262) — answer by Nosjack, CC-BY-SA 4.0*
**A customer purchased a PC and discovered it was significantly cheaper just days later, but is unable to fully utilize standard return policies due to damaged packaging.**
When faced with a price drop after purchase, directly requesting a price adjustment is the simplest solution as retailers often prioritize customer satisfaction over processing returns. If that fails, politely presenting alternative solutions – like an exchange using current pricing – can guide a hesitant employee towards a mutually beneficial outcome. Understanding store policies (like serial number tracking) and framing requests ethically are key to navigating these situations.
*Source: [The price dropped on an item I purchased 8 days ago. What can I do?](https://money.stackexchange.com/q/63737) — answer by Ben Miller, CC-BY-SA 4.0*
## Bigdata
**The questioner asks for a clear definition of 'big data', beyond simply large datasets, and wonders if any computational problem can be considered 'big data' given enough input, or if it’s tied to specific applications like data mining.**
True 'big data' isn't defined by volume alone; it's characterized by inherent data quality issues. Traditional databases demand meticulously cleaned ('pristine') data upfront, while big data approaches accept and even *require* a large enough dataset that inaccuracies and missing information become statistically irrelevant. The key is reaching a scale where the 'noise' in the data doesn’t invalidate analytical results, allowing algorithms to function effectively despite imperfections.
*Source: [How big is big data?](https://datascience.stackexchange.com/q/19) — answer by rolfl, CC-BY-SA 4.0*
**The user questions whether the R programming language is appropriate for handling 'Big Data' datasets (around 5TB in size), given its typical in-memory processing.**
While traditionally limited by memory constraints, R *can* be used with Big Data through integration with distributed computing frameworks like Hadoop. The key is to avoid loading the entire dataset into RAM at once; instead, leverage tools like RHadoop to process data in chunks across a cluster of machines. This approach allows R's analytical capabilities to be applied to datasets exceeding single-machine memory limits.
*Source: [Is the R language suitable for Big Data](https://datascience.stackexchange.com/q/41) — answer by MCP_infiltrator, CC-BY-SA 4.0*
**An aspiring data scientist is concerned about whether learning Hadoop is essential for entering the field, given its frequent mention in discussions of 'Big Data'. They are unsure if it’s a mandatory skill.**
Data science is broad and doesn't require mastery of any single tool like Hadoop. While understanding underlying concepts such as distributed systems is valuable, practical data science can be effectively performed on smaller datasets without needing big data technologies. The core skillset lies in analytical thinking, problem-solving, and a willingness to learn – not necessarily expertise in every platform.
*Source: [Do I need to learn Hadoop to be a Data Scientist?](https://datascience.stackexchange.com/q/253) — answer by Steve Kallestad, CC-BY-SA 4.0*
**The user has an 8GB dataset and wants to apply SVD/PCA for dimensionality reduction but is running into memory limitations with standard tools like MATLAB or Octave.**
Before applying complex techniques, consider if dimensionality reduction is truly necessary given the data's existing dimensions. When dealing with large datasets that don’t fit in memory, prioritize incremental methods – processing data in smaller batches rather than attempting to load everything at once. If focusing on covariance matrix-based PCA, calculate it incrementally or use a representative sample; avoid building the full matrix if the number of variables is extremely high.
*Source: [How to do SVD and PCA with big data?](https://datascience.stackexchange.com/q/1159) — answer by ffriend, CC-BY-SA 4.0*
## Nlp
**The questioner is struggling to understand the purpose and mechanics of positional encoding within the Transformer model described in 'Attention is All You Need'. They grasp the general idea of it relating to word position but need clarification on *how* it works.**
Positional encoding addresses a key limitation of the Transformer architecture: its inherent lack of sequential understanding. Because Transformers process all input tokens simultaneously, positional encodings are added to the word embeddings to inject information about word order. The use of sine and cosine functions allows the model to easily learn relationships based on *relative* position – how far apart words are – rather than just their absolute location in a sequence, improving its ability to understand grammatical structures.
*Source: [What is the positional encoding in the transformer model?](https://datascience.stackexchange.com/q/51065) — answer by Esmailian, CC-BY-SA 4.0*
**The questioner is struggling to understand the purpose and mechanics of positional encoding within the Transformer model described in 'Attention is All You Need'. They grasp the general idea of it relating to word position but need clarification on *how* it works.**
Positional encoding addresses a key limitation of the Transformer architecture: its inherent lack of sequential understanding. Because Transformers process all input tokens simultaneously, positional encodings are added to the word embeddings to inject information about word order. The use of sine and cosine functions allows the model to easily learn relationships based on *relative* position – how far apart words are – rather than just their absolute location in a sequence, improving its ability to understand grammatical structures.
*Source: [What is the positional encoding in the transformer model?](https://datascience.stackexchange.com/q/51065) — answer by Batool, CC-BY-SA 4.0*
**The questioner wants to understand the core difference between Latent Dirichlet Allocation (LDA) and Hierarchical Dirichlet Process (HDP) topic modeling techniques, specifically why HDP doesn't require pre-defining the number of topics.**
HDP builds upon LDA by introducing a mechanism to automatically determine the optimal number of topics within a dataset. While LDA requires you to *tell* the model how many topics exist, HDP *learns* this from the data itself using a Dirichlet process. This flexibility is powerful but comes at the cost of increased implementation complexity; if you already have a good idea of the topic count, LDA remains a simpler and perfectly viable option.
*Source: [Latent Dirichlet Allocation vs Hierarchical Dirichlet Process](https://datascience.stackexchange.com/q/128) — answer by Tim Goodman, CC-BY-SA 4.0*
## Xgboost
**The user asks if XGBoost automatically handles multicollinearity and how it affects predictions, given their dataset has many features created through one-hot encoding.**
XGBoost, being based on decision trees, is inherently resistant to the negative effects of multicollinearity because each tree will effectively select only one of highly correlated features during splitting. While XGBoost isn't *harmed* by multicollinearity, it’s still best practice to reduce redundancy in your data for model efficiency and interpretability. Feature importance metrics within XGBoost (like 'Gain') can help identify the most valuable features to retain.
*Source: [Does XGBoost handle multicollinearity by itself?](https://datascience.stackexchange.com/q/12554) — answer by Sandeep S. Sandhu, CC-BY-SA 4.0*
**The user questioned whether feature scaling (normalization) with techniques like `MinMaxScaler()` is needed when preparing data for XGBoost models, given that decision trees – on which XGBoost is based – don't require it.**
XGBoost, being an ensemble of decision trees, inherits the property of not needing normalized input features. Normalization isn’t inherently harmful, but it doesn’t provide any benefit to model performance with tree-based algorithms like XGBoost. Focus on feature importance and engineering rather than scaling when working with these models.
*Source: [Is it necessary to normalize data for XGBoost?](https://datascience.stackexchange.com/q/60950) — answer by desertnaut, CC-BY-SA 4.0*
**The user wants to prioritize more recent data points when training an XGBoost model, effectively giving them a stronger influence on the outcome.**
XGBoost allows for sample weighting during training. You can directly incorporate time-based weights into your `xgb.DMatrix` object by calculating a weight value based on each sample's timestamp; newer samples receive higher weights. This approach provides a flexible way to bias the model towards learning from more current information without altering the core algorithm.
*Source: [xgboost: give more importance to recent samples](https://datascience.stackexchange.com/q/9488) — answer by wacax, CC-BY-SA 4.0*
## Scikit Learn
**The questioner routinely uses LabelEncoder for categorical data in machine learning models but wants guidance on when it's more appropriate to use One-Hot Encoding or DictVectorizer instead.**
While LabelEncoding can be space efficient and work with some algorithms like tree-based methods, it introduces artificial ordering that can mislead models. One-Hot Encoding avoids this by creating orthogonal binary features, but suffers from dimensionality issues with many categories. A strong approach is to combine One-Hot Encoding with Principal Component Analysis (PCA) to reduce dimensionality while retaining important information.
*Source: [When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?](https://datascience.stackexchange.com/q/9443) — answer by AN6U5, CC-BY-SA 4.0*
**The user is confused by the shape of the probabilities returned by `predict_proba` when using a `MultiOutputClassifier`. They expected a single matrix but received a list of arrays, and are unsure how to interpret the results for each output.**
When working with multioutput classification, `predict_proba` returns separate probability estimates for *each* target variable. Each array in the returned list corresponds to one output, containing probabilities for each class within that specific output. To access the probability of a particular class across all samples for a single output, you need to index into the result using column selection (e.g., `[:, 1]` for the second class).
*Source: [Understanding predict_proba from MultiOutputClassifier](https://datascience.stackexchange.com/q/22762) — answer by chrisckwong821, CC-BY-SA 4.0*
## Data Mining
**The user has a dataset with both numeric and categorical features and wants to know if converting the single categorical feature into multiple binary (0/1) variables is a valid way to use standard K-Means clustering.**
Applying K-Means directly to mixed data types is problematic because distance calculations are not meaningful for discrete, non-ordered categorical values. While one approach is to binarize the categories, more robust algorithms like k-modes or k-prototypes exist that are specifically designed to handle categorical and mixed datasets by using appropriate distance metrics (like Hamming distance). Exploring specialized clustering methods will likely yield better results than forcing a standard numeric algorithm onto non-numeric data.
*Source: [K-Means clustering for mixed numeric and categorical data](https://datascience.stackexchange.com/q/22) — answer by Tim Goodman, CC-BY-SA 4.0*
**The asker questions whether Support Vector Machines (SVMs) are still widely used, given a comment suggesting they've fallen out of favor. They seek to understand if this is due to newer algorithms or changes in computing power.**
While SVMs remain powerful classifiers with benefits like efficiency and kernel flexibility, they suffer from sensitivity to parameter tuning which requires significant effort for optimization. Newer methods, such as Random Forests, have gained popularity because of their ease of use – requiring fewer parameters to tune while still achieving strong performance. The rise of these simpler-to-implement algorithms has led to a shift in preference despite SVM's inherent strengths.
*Source: [Are Support Vector Machines still considered "state of the art" in their niche?](https://datascience.stackexchange.com/q/711) — answer by Debasis, CC-BY-SA 4.0*
## Artificial Intelligence
**The asker is struggling with users posting unformatted code on their platform (Stack Overflow), requiring manual intervention to fix it. They're seeking a reliable method to detect potential code snippets within user-submitted text.**
Instead of aiming for perfect detection, focus on identifying *indicators* strongly associated with code. Combining multiple simple heuristics – like the presence of specific characters, syntax elements (semicolons, braces), or naming conventions (camelCase) – can create a surprisingly effective signal. This approach allows for a warning to users without needing complex machine learning initially, and provides data points that could *eventually* feed into a more sophisticated model.
*Source: [Simple method for reliably detecting code in text?](https://softwareengineering.stackexchange.com/q/87611) — answer by Yevgeniy Brikman, CC-BY-SA 4.0*
**A consultant is struggling with a team member who relies on AI (specifically Copilot) to generate code without understanding it themselves, leading to low-quality pull requests and a time-consuming review cycle where the programmer simply accepts suggested fixes without comprehension.**
The core issue isn't necessarily the use of AI tools, but rather a lack of fundamental skill and accountability masked by their perceived efficiency. When management lacks deep technical understanding, they may overestimate the capabilities of these tools and misinterpret superficial progress as genuine productivity. The best approach is to disengage from enabling this behavior – allow the individual to work independently on isolated tasks without support, forcing them (and management) to confront the limitations directly.
*Source: [How to deal with a programmer who acts as a proxy for AI?](https://softwareengineering.stackexchange.com/q/460875) — answer by JimmyJames, CC-BY-SA 4.0*
## R
**The user is building a regression model and needs appropriate methods to assess correlations between categorical variables (both two-way and one categorical with one continuous), as well as alternatives to VIF for multi-level categorical predictors.**
When examining relationships between categorical variables, the Chi-Squared test of independence determines if they are associated. A significant p-value suggests a correlation, which can be quantified using Cramér's V. For relating a categorical variable to a continuous one, ANOVA is appropriate; it compares variance *within* groups defined by the categorical variable to variance *between* those groups to detect dependence. These tests assess association rather than linear correlation like Pearson’s coefficient.
*Source: [How to get correlation between two categorical variable and a categorical variab](https://datascience.stackexchange.com/q/893) — answer by Alexey Grigorev, CC-BY-SA 4.0*
**The user, accustomed to robust IDEs from other programming languages, is curious about alternatives to RStudio for R development, specifically focusing on features beyond basic coding like debugging and deployment.**
While RStudio remains a popular choice, several alternative IDEs cater to different needs and workflows. These range from browser-based options like Radiant and JupyterLab offering collaboration and reproducibility, to extensions for existing IDEs like Visual Studio (RTVS) or Atom (Rbox). The best option depends on the user's preferred environment and whether they prioritize features like visual data analysis (Rattle), flexible layouts (RIDE), or integration with other tools.
*Source: [IDE alternatives for R programming (RStudio, IntelliJ IDEA, Eclipse, Visual Stud](https://datascience.stackexchange.com/q/5345) — answer by karupakalas, CC-BY-SA 4.0*
## Pandas
**The user has a list of lists and wants to efficiently transform it into a Pandas DataFrame with each inner list representing a row and each element within those lists becoming a column.**
When constructing DataFrames from lists of lists, `pd.DataFrame.from_records()` is often the most direct approach. This method automatically infers column structure based on the order of elements in each sublist. While other methods exist, this one streamlines the process and avoids manual column assignment.
*Source: [Convert a list of lists into a Pandas Dataframe](https://datascience.stackexchange.com/q/26333) — answer by Emre, CC-BY-SA 4.0*
**The user wants to identify matching records (based on name) between two different pandas DataFrames.**
When comparing data across multiple DataFrames, using `pd.merge` is generally more performant than filtering with `.where()`, especially for larger datasets. Merging creates a new DataFrame containing only the rows where the specified columns match in both original DataFrames, effectively identifying common records. This approach leverages optimized pandas operations for efficient comparison.
*Source: [How do I compare columns in different data frames?](https://datascience.stackexchange.com/q/33053) — answer by Tarek, CC-BY-SA 4.0*
## Reinforcement Learning
**The asker is seeking clarification on the "experience replay" technique used in reinforcement learning, specifically as described in DeepMind's Atari paper. They want to understand how it works and what benefits it provides.**
Experience replay decouples data collection from the learning process by storing past experiences (state, action, reward, next state) for later use. This allows for more efficient training, especially when acquiring new data is expensive, as previously gathered information can be reused multiple times. By randomizing the order of samples during learning, it reduces correlations and improves convergence stability – effectively making the learning process resemble supervised learning with independent and identically distributed (i.i.d.) data.
*Source: [What is "experience replay" and what are its benefits?](https://datascience.stackexchange.com/q/20535) — answer by Neil Slater, CC-BY-SA 4.0*
**The questioner asks for clarification on the concept of 'bootstrapping' within reinforcement learning, specifically why Temporal Difference (TD) methods are considered bootstrapping while Monte Carlo methods aren't.**
Bootstrapping in RL means updating an estimate using *other* estimates – essentially learning from your own predictions. While this introduces potential bias and instability because of reliance on potentially inaccurate values, it allows for faster learning as updates don’t require complete episodes to finish. Monte Carlo methods avoid bootstrapping by relying solely on observed rewards, offering unbiased but often slower convergence due to higher variance.
*Source: [What exactly is bootstrapping in reinforcement learning?](https://datascience.stackexchange.com/q/26938) — answer by Neil Slater, CC-BY-SA 4.0*
## Model Evaluations
**The user is observing a significant difference between micro and macro averaged performance metrics (precision, recall, F1-score) in a multiclass classification problem with imbalanced classes, and doesn't understand why the micro average values are all equal.**
When evaluating models on imbalanced datasets, it’s crucial to understand how different averaging methods interpret results. Macro-averaging treats each class equally, potentially masking poor performance in minority classes, while micro-averaging considers overall accuracy across all instances and is more sensitive to the dominant class(es). Choosing the right metric depends on whether you prioritize equal representation of all classes or overall system performance given the existing distribution.
*Source: [Micro Average vs Macro average Performance in a Multiclass classification settin](https://datascience.stackexchange.com/q/15989) — answer by pythiest, CC-BY-SA 4.0*
## Dataset
**The questioner observes a lot of redundant effort in data science projects – repeatedly collecting and cleaning common datasets like Twitter feeds or Wikipedia articles – and asks if a central repository for pre-processed, reusable datasets exists.**
While several public dataset repositories *do* exist (AWS Public Datasets, UCI Machine Learning Repository, KDnuggets, etc.), widespread data sharing is hampered by privacy concerns and restrictions on sensitive information. A key improvement would be better organization of these datasets based on their potential applications, allowing users to easily find resources relevant to specific analytical goals. The challenge isn't a lack of data, but discoverability and usability.
*Source: [Publicly Available Datasets](https://datascience.stackexchange.com/q/155) — answer by Rubens, CC-BY-SA 4.0*
## Classification
**The user asks when to choose cosine similarity over the dot product for measuring how similar two feature vectors are, given their mathematical relationship.**
Both methods assess similarity based on angle between vectors, but the dot product is also influenced by vector length. Cosine similarity focuses *solely* on angular difference, making it useful when magnitude isn't meaningful or data is already normalized. If the size of the vectors carries important information, then using the dot product will provide a more nuanced comparison.
*Source: [Cosine similarity versus dot product as distance metrics](https://datascience.stackexchange.com/q/744) — answer by Memming, CC-BY-SA 4.0*
## Historical Data
**The questioner is puzzled why banks limit access to transaction history despite low storage costs, contrasting it with services like email which retain data for much longer periods.**
Established organizations, particularly in heavily regulated industries like banking, often prioritize stability and cost control over innovation. Legacy systems and deeply ingrained operational norms create significant inertia, making even relatively simple improvements difficult to implement without a clear financial incentive or compelling business need. Change requires overcoming cultural resistance to new ways of doing things, even when the technical barriers are minimal.
*Source: [Why don't banks give access to all your transaction activity?](https://money.stackexchange.com/q/37480) — answer by Chris W. Rea, CC-BY-SA 4.0*
## Feature Selection
**The user asks for an explanation of dimensionality reduction and clarification on the distinction between feature selection and feature extraction techniques.**
Dimensionality reduction aims to simplify data by reducing the number of variables used. Feature selection involves choosing the *most relevant* original features, while feature extraction creates entirely *new* features based on the originals. Extraction often loses some information as it transforms the data into a new representation, whereas selection retains the original features.
*Source: [What is dimensionality reduction? What is the difference between feature selecti](https://datascience.stackexchange.com/q/130) — answer by damienfrancois, CC-BY-SA 4.0*
## Cross Validation
**The questioner understands K-fold cross-validation and is trying to understand the core difference between it and bootstrapping for evaluating machine learning models, noting they both involve creating multiple training/testing subsets.**
Both techniques are resampling methods, but their primary goals differ. Cross-validation focuses on estimating how well a model will *generalize* to unseen data through systematic, non-replacement sampling. Bootstrapping aims to understand the distribution of statistics (like model variation) by resampling *with replacement*, and while it can estimate generalization error, its performance is often similar to cross-validation with slightly more bias but less variance.
*Source: [What is the difference between bootstrapping and cross-validation?](https://datascience.stackexchange.com/q/32264) — answer by cbeleites, CC-BY-SA 4.0*
## Market Indexes
**The asker questions the long-term safety of consistently investing in the stock market, citing examples like the Nikkei 225 which experienced prolonged stagnation after initial investment during a specific period.**
Long-term stock market investing isn't guaranteed to avoid losses, but consistent investment—even through downturns—increases the probability of overall gains. The strategy relies on 'dollar-cost averaging,' where buying at both high and low points eventually balances out, capitalizing on future growth. Diversification across multiple indexes can further mitigate risk by smoothing out individual market fluctuations.
*Source: [How safe it is in the long run to regularly invest in the stock market?](https://money.stackexchange.com/q/120904) — answer by Daniel, CC-BY-SA 4.0*
## Correlation
**The user is asking for a breakdown of Pearson, Spearman, and Kendall correlation coefficients – what each measures, and how they differ in terms of underlying assumptions.**
These three methods all quantify the association between variables, but differ in their requirements. Pearson requires continuous data with a linear relationship and is sensitive to outliers; Spearman assesses monotonic relationships using ranked data, making it robust to non-linearities and outliers; Kendall's tau, also rank-based, focuses on concordant/discordant pairs and is often preferred for smaller datasets or when dealing with tied ranks. Choosing the right method depends heavily on the nature of your data and the assumptions you can reasonably meet.
*Source: [Pearson vs Spearman vs Kendall](https://datascience.stackexchange.com/q/64260) — answer by Pluviophile, CC-BY-SA 4.0*
## Normalization
**The user is unsure whether to normalize data before or after splitting it into training and testing sets.**
Data normalization should occur *after* the train-test split, using only the training data to calculate normalization parameters. This prevents 'data leakage' from the test set influencing the model training process, ensuring a more realistic evaluation of performance on truly unseen data. The same transformation applied to the training data must then be consistently used for the testing data.
*Source: [Data normalization before or after train-test split?](https://datascience.stackexchange.com/q/54908) — answer by Erwan, CC-BY-SA 4.0*
## Computer Vision
**The questioner is confused by the notation 'mAP@[.5:.95]' in an object detection paper, specifically how it relates to Intersection over Union (IoU) thresholds and calculating mean Average Precision (mAP). They understand mAP at a single IoU threshold but not this range.**
This notation represents averaging the mAP score across a *range* of IoU thresholds, from 0.5 to 0.95 with increments of 0.05. This provides a more comprehensive evaluation of detection performance than relying on a single IoU threshold because it assesses accuracy at varying levels of overlap between predicted and ground truth bounding boxes. It's a standard metric used in challenges like the MS COCO dataset, indicating robustness across different localization accuracies.
*Source: [What does the notation mAP@[.5:.95] mean?](https://datascience.stackexchange.com/q/16797) — answer by Icyblade, CC-BY-SA 4.0*
## Activation Function
**The questioner is seeking a simplified explanation of the GELU activation function used in the BERT paper, specifically how its mathematical definition relates to the commonly used approximation formula.**
Mathematical functions often require approximations for computational efficiency. These approximations are achieved by finding a similar, more easily calculated function and then 'fitting' it to the original function’s behavior using parameter optimization. Leveraging known relationships – like matching derivatives at specific points – can significantly streamline this fitting process and lead to more elegant (and analytically useful) results. Furthermore, understanding symmetries within functions allows for constraints that simplify the approximation search.
*Source: [What is GELU activation?](https://datascience.stackexchange.com/q/49522) — answer by Esmailian, CC-BY-SA 4.0*
## Keras
**The user is questioning how Keras' `validation_split` parameter works, specifically if it always selects validation data from the end of the dataset without shuffling and whether a random selection is possible.**
Using a fixed validation split within Keras can be problematic if your data isn’t randomly ordered, potentially leading to biased evaluation. It's best practice to create separate training and validation sets *before* model training to avoid overfitting and ensure consistent evaluation. For more control over the split – especially for class imbalances – leverage tools like `train_test_split` from scikit-learn to achieve a truly random and representative division of your data.
*Source: [How does the validation_split parameter of Keras' fit function work?](https://datascience.stackexchange.com/q/38955) — answer by JahKnows, CC-BY-SA 4.0*
## Feature Extraction
**The user has a cyclic numerical feature ('hour' of the day) and wants to transform it in a way that preserves the proximity of values at the boundaries (e.g., 23 and 0 should be considered close). They are using this data for a random forest classifier.**
Instead of simply reducing the range, represent cyclic features with trigonometric functions like sine and cosine. This creates two new features that capture the cyclical nature smoothly, avoiding abrupt jumps at boundaries. Consider adding a linear time component as well to model both cyclical *and* progressive trends in your data for richer insights.
*Source: [What is a good way to transform Cyclic Ordinal attributes?](https://datascience.stackexchange.com/q/5990) — answer by AN6U5, CC-BY-SA 4.0*

21
references/profile.md Normal file
View File

@@ -0,0 +1,21 @@
# Occupation profile — data scientist
- **ESCO URI:** http://data.europa.eu/esco/occupation/258e46f9-0075-4a2e-adae-1ff0477e0f30
- **ESCO code:** 2511.4
- **ISCO-08 group:** 2511 — n/a
- **O*NET-SOC:** 15-2041.00 — Statisticians (match: closeMatch)
## Description (ESCO)
Data scientists find and interpret rich data sources, manage large amounts of data, merge data sources, ensure consistency of data-sets, and create visualisations to aid in understanding data. They build mathematical models using data, present and communicate data insights and findings to specialists and scientists in their team and if required, to a non-expert audience, and recommend ways to apply the data.
## Definition
nan
## Alternative labels
- data expert
- research data scientist
- data research scientist
- electromobility data scientist

130
references/skills.md Normal file
View File

@@ -0,0 +1,130 @@
# Competences — data scientist
Source: ESCO v1.2.1 occupation-skill relations (http://data.europa.eu/esco/occupation/258e46f9-0075-4a2e-adae-1ff0477e0f30).
## Essential
- **apply for research funding** (skill/competence)
- **apply research ethics and scientific integrity principles in research activities** (skill/competence)
- **build recommender systems** (skill/competence)
- **collect ICT data** (skill/competence)
- **communicate with a non-scientific audience** (skill/competence)
- **conduct research across disciplines** (skill/competence)
- **data engineering** (knowledge)
- **data ethics** (knowledge)
- **data mining** (knowledge)
- **data models** (knowledge)
- **data science** (knowledge)
- **data visualisation software** (knowledge)
- **deliver visual presentation of data** (skill/competence)
- **demonstrate disciplinary expertise** (skill/competence)
- **design database scheme** (skill/competence)
- **develop data processing applications** (skill/competence)
- **develop professional network with researchers and scientists** (skill/competence)
- **disseminate results to the scientific community** (skill/competence)
- **draft scientific or academic papers and technical documentation** (skill/competence)
- **empirical analysis** (knowledge)
- **establish data processes** (skill/competence)
- **evaluate research activities** (skill/competence)
- **execute analytical mathematical calculations** (skill/competence)
- **handle data samples** (skill/competence)
- **implement data quality processes** (skill/competence)
- **increase the impact of science on policy and society** (skill/competence)
- **information categorisation** (knowledge)
- **information extraction** (knowledge)
- **integrate gender dimension in research** (skill/competence)
- **interact professionally in research and professional environments** (skill/competence)
- **interpret current data** (skill/competence)
- **manage data collection systems** (skill/competence)
- **manage findable accessible interoperable and reusable data** (skill/competence)
- **manage intellectual property rights** (skill/competence)
- **manage open publications** (skill/competence)
- **manage personal professional development** (skill/competence)
- **manage research data** (skill/competence)
- **mathematical modelling** (knowledge)
- **mentor individuals** (skill/competence)
- **normalise data** (skill/competence)
- **online analytical processing** (knowledge)
- **operate open source software** (skill/competence)
- **perform data cleansing** (skill/competence)
- **perform project management** (skill/competence)
- **perform scientific research** (skill/competence)
- **promote open innovation in research** (skill/competence)
- **promote the participation of citizens in scientific and research activities** (skill/competence)
- **promote the transfer of knowledge** (skill/competence)
- **publish academic research** (skill/competence)
- **quantitative analysis** (knowledge)
- **query languages** (knowledge)
- **report analysis results** (skill/competence)
- **resource description framework query language** (knowledge)
- **scientific literature** (knowledge)
- **speak different languages** (skill/competence)
- **statistical modeling techniques** (knowledge)
- **statistics** (knowledge)
- **synthesise information** (skill/competence)
- **think abstractly** (skill/competence)
- **use data processing techniques** (skill/competence)
- **use databases** (skill/competence)
- **visual presentation techniques** (knowledge)
- **write scientific publications** (skill/competence)
## Optional
- apply blended learning (skill/competence)
- business analytics (knowledge)
- business intelligence (knowledge)
- computational biology (knowledge)
- computer simulation (knowledge)
- create data models (skill/competence)
- data quality assessment (knowledge)
- define data quality criteria (skill/competence)
- design database in the cloud (skill/competence)
- digital curation (knowledge)
- Hadoop (knowledge)
- healthcare analytics (knowledge)
- image recognition (knowledge)
- integrate ICT data (skill/competence)
- LDAP (knowledge)
- LINQ (knowledge)
- make data-driven decisions (skill/competence)
- manage data (skill/competence)
- manage ICT data architecture (skill/competence)
- manage ICT data classification (skill/competence)
- marketing analytics (knowledge)
- MDX (knowledge)
- multidisciplinary research (knowledge)
- N1QL (knowledge)
- perform data mining (skill/competence)
- research design (knowledge)
- scientific computing (knowledge)
- social network analysis (knowledge)
- SPARQL (knowledge)
- state estimation (knowledge)
- teach in academic or vocational contexts (skill/competence)
- unstructured data (knowledge)
- use spreadsheets software (skill/competence)
- XQuery (knowledge)
<!-- market-evidence -->
## Market evidence (job-ad analysis, 46 ads, as of 2026-07-11)
Share of analyzed job ads mentioning the item (threshold ≥ 20 %). Source: JSearch/Adzuna APIs.
### Hard skills
- machine learning — **67 %**
- data analysis — **57 %**
- statistical modeling — **37 %**
- statistical analysis — **33 %**
- predictive modeling — **28 %**
- data mining — **26 %**
- data visualization — **20 %**
- natural language processing — **20 %**
### Responsibilities
- data analysis — **35 %**
- model development — **24 %**
<!-- market-evidence -->

45
references/tasks.md Normal file
View File

@@ -0,0 +1,45 @@
# Tasks & work activities — data scientist
Source: O*NET 30.3, occupation 15-2041.00 (Statisticians).
## Task statements
- **[Core]** Report results of statistical analyses in peer-reviewed papers and technical manuals.
- **[Core]** Develop software applications or programming for statistical modeling and graphic analysis.
- **[Core]** Present statistical and nonstatistical results, using charts, bullets, and graphs, in meetings or conferences to audiences such as clients, peers, and students.
- **[Core]** Determine whether statistical methods are appropriate, based on user needs or research questions of interest.
- **[Core]** Report results of statistical analyses, including information in the form of graphs, charts, and tables.
- **[Core]** Process large amounts of data for statistical modeling and graphic analysis, using computers.
- **[Core]** Identify relationships and trends in data, as well as any factors that could affect the results of research.
- **[Core]** Analyze and interpret statistical data to identify significant differences in relationships among sources of information.
- **[Core]** Prepare data for processing by organizing information, checking for inaccuracies, and adjusting and weighting the raw data.
- **[Core]** Evaluate the statistical methods and procedures used to obtain data to ensure validity, applicability, efficiency, and accuracy.
- **[Core]** Evaluate sources of information to determine any limitations, in terms of reliability or usability.
- **[Core]** Plan data collection methods for specific projects, and determine the types and sizes of sample groups to be used.
- **[Core]** Design research projects that apply valid scientific techniques, and use information obtained from baselines or historical data to structure uncompromised and efficient analyses.
- **[Core]** Supervise and provide instructions for workers collecting and tabulating data.
- **[Core]** Apply sampling techniques, or use complete enumeration bases to determine and define groups to be surveyed.
- **[Core]** Adapt statistical methods to solve specific problems in many fields, such as economics, biology, and engineering.
- **[Core]** Develop and test experimental designs, sampling techniques, and analytical methods.
- **[Core]** Examine theories, such as those of probability and inference, to discover mathematical bases for new or improved methods of obtaining and evaluating numerical data.
- **[Supplemental]** Prepare and structure data warehouses for storing data.
## Detailed work activities
- Analyze data to identify trends or relationships among variables.
- Apply mathematical principles or statistical approaches to solve problems in scientific or applied fields.
- Design research studies to obtain scientific information.
- Design software applications.
- Determine appropriate methods for data analysis.
- Evaluate data quality.
- Evaluate project designs to determine adequacy or feasibility.
- Evaluate technical data to determine effect on designs or plans.
- Implement security measures for computer or information systems.
- Install computer software.
- Prepare analytical reports.
- Prepare data for analysis.
- Prepare graphics or other visual representations of information.
- Present research results to others.
- Supervise information technology personnel.
- Update knowledge about emerging industry or technology trends.
- Write computer programming code.

91
references/tools.md Normal file
View File

@@ -0,0 +1,91 @@
# Tools & technology — data scientist
Source: O*NET 30.3 'Software Skills' for 15-2041.00.
| Software | Category | Hot technology |
|---|---|---|
| IBM SPSS Statistics | Analytical or scientific software | yes |
| SAS | Analytical or scientific software | yes |
| The MathWorks MATLAB | Analytical or scientific software | yes |
| Apache Spark | Business intelligence and data analysis software | yes |
| Tableau | Business intelligence and data analysis software | yes |
| Apache Hadoop | Data base management system software | yes |
| Teradata Database | Data base management system software | yes |
| Amazon Redshift | Data base user interface and query software | yes |
| Amazon Web Services AWS software | Data base user interface and query software | yes |
| IBM DB2 | Data base user interface and query software | yes |
| Microsoft Access | Data base user interface and query software | yes |
| Microsoft SQL Server | Data base user interface and query software | yes |
| Structured query language SQL | Data base user interface and query software | yes |
| Microsoft Visual Basic | Development environment software | yes |
| Microsoft Visual Basic for Applications VBA | Development environment software | yes |
| Extensible markup language XML | Enterprise application integration software | yes |
| SAP software | Enterprise resource planning ERP software | yes |
| C++ | Object or component oriented development software | yes |
| Python | Object or component oriented development software | yes |
| R | Object or component oriented development software | yes |
| Microsoft Office software | Office suite software | yes |
| Linux | Operating system software | yes |
| UNIX | Operating system software | yes |
| Microsoft PowerPoint | Presentation software | yes |
| Microsoft Excel | Spreadsheet software | yes |
| Microsoft Word | Word processing software | yes |
| Aptech Systems GAUSS | Analytical or scientific software | |
| Automatic Forecasting Systems Autobox | Analytical or scientific software | |
| Camfit Data Limited Microfit | Analytical or scientific software | |
| Cytel StatXact | Analytical or scientific software | |
| DataDescription DataDesk | Analytical or scientific software | |
| Econometric Software LIMDEP | Analytical or scientific software | |
| GraphPad Software GraphPad Prism | Analytical or scientific software | |
| IBM SPSS Amos | Analytical or scientific software | |
| IBM SPSS AnswerTree | Analytical or scientific software | |
| Insightful S-PLUS | Analytical or scientific software | |
| Minitab | Analytical or scientific software | |
| Muthen & Muthen Mplus | Analytical or scientific software | |
| NCSS Power Analysis and Sample Size PASS | Analytical or scientific software | |
| Quantitative Micro Software EViews | Analytical or scientific software | |
| RAT-STATS | Analytical or scientific software | |
| SAS JMP | Analytical or scientific software | |
| Scientific Software International SSI Hierarchical Linear and Non-Linear Modeling HLM | Analytical or scientific software | |
| Scientific Software International SSI LISREL | Analytical or scientific software | |
| StataCorp Stata | Analytical or scientific software | |
| Stat-Ease Design-Ease | Analytical or scientific software | |
| Stat-Ease Design-Expert | Analytical or scientific software | |
| STATISTICA | Analytical or scientific software | |
| Statistical software | Analytical or scientific software | |
| Statistical Solutions BMDP | Analytical or scientific software | |
| StatPoint STATGRAPHICS Centurion | Analytical or scientific software | |
| SuperANOVA | Analytical or scientific software | |
| Systat Software SigmaPlot | Analytical or scientific software | |
| Systat Software SigmaStat | Analytical or scientific software | |
| UNISTAT Statistical Package | Analytical or scientific software | |
| Visual Numerics TS-WAVE | Analytical or scientific software | |
| Wolfram Research Mathematica | Analytical or scientific software | |
| XGobi | Analytical or scientific software | |
| XLISP-STAT | Analytical or scientific software | |
| Qlik Tech QlikView | Business intelligence and data analysis software | |
| Apache Pig | Data base management system software | |
| Angoss KnowledgeSEEKER | Data mining software | |
| NCR Teradata Warehouse Miner | Data mining software | |
| SAS Enterprise Miner | Data mining software | |
| Common business oriented language COBOL | Development environment software | |
| Formula translation/translator FORTRAN | Development environment software | |
| SAS/CONNECT | Enterprise application integration software | |
| SAP PowerBuilder | Object or component oriented development software | |
| Sun Microsystems Java | Object or component oriented development software | |
| Microsoft Visual FoxPro | Object oriented data base management software | |
<!-- market-evidence -->
## Market evidence (job-ad analysis, 46 ads, as of 2026-07-11)
Share of analyzed job ads mentioning the item (threshold ≥ 20 %). Source: JSearch/Adzuna APIs.
### Tools
- Python — **78 %**
- SQL — **46 %**
- Spark — **24 %**
- AWS — **22 %**
<!-- market-evidence -->