1.8 KiB
1.8 KiB
name, description
| name | description |
|---|---|
| data-engineer | Occupational skill for the role 'data engineer' (also: data engineer expert, research data engineer). Use when the user asks for typical data engineer work such as: typical data engineer responsibilities |
Data Engineer
Data engineers develop the architecture needed to process, manage, and store large amounts of data which will be used by data scientists for analysis. They design the infrastructure and maintain data pipelines and warehouses to leverage data for strategic advantage.
Core workflow
How to use this skill
- Read references/profile.md for the occupation profile and scope.
- Consult references/tasks.md for the full task and activity inventory.
- Check references/skills.md for essential vs. optional competences.
- Check references/tools.md for the software commonly used in this role.
- See references/ai-skills.md — matched external AI agent skills (per-source attribution).
Key competences (essential)
- cloud technologies
- computer science
- create data sets
- data analytics
- data models
- data storage
- data warehouse
- database management systems
- design database in the cloud
- develop data processing applications
- digital data processing
- establish data processes
- implement data warehousing techniques
- manage data
- manage ICT data architecture
Hot technologies
- Alteryx software
- Amazon Elastic Compute Cloud EC2
- Amazon Redshift
- Amazon Web Services AWS software
- Apache Airflow
- Apache Cassandra
- Apache Hadoop
- Apache Hive
- Apache Kafka
- Apache Spark
Sources: ESCO v1.2.1 (http://data.europa.eu/esco/occupation/2079755f-d809-49e6-8037-4de6180e54c0), ONET 30.3 (15-2051.00, manual nearest match). See manifest.json for licensing/attribution.*