Senior Data Engineer

Simulmedia

Що робити

Design and build batch data pipelines that ingest, validate and transform multi-billion-row datasets from external data providers and internal systems
Model complex real-world data: dimensional models, reference data, and temporal data whose attributes change over time (e.g. slowly changing dimensions), and evolve those models safely as upstream sources change their schemas and semantics
Develop and operate workloads on our lakehouse platform (Databricks / Spark / Delta) and our data warehouse (Redshift), including migrating existing pipelines from the warehouse to the lakehouse
Orchestrate pipelines with Airflow: scheduling, dependencies, retries, backfills and alerting
Prove correctness, not just completion: design parity checks and reconciliation queries when replacing an existing pipeline, run large historical backfills, and investigate data discrepancies down to the row level

Bachelor’s degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
7+ years of work experience as a data engineer
Python as your primary language for pipelines and services in recent years
3+ years of hands-on Spark (PySpark) in production batch pipelines: you wrote the jobs, deployed them through CI/CD and kept them running at scale
Production experience with Databricks: Jobs/Workflows, Delta Lake, and deploying through CI/CD. Notebook-only experience does not count. A strong candidate without Databricks must show deep production PySpark on AWS EMR instead