Senior Data Engineer

Що робити

Design and build batch data pipelines that ingest, validate and transform multi-billion-row datasets from external data providers and internal systems;
Model complex real-world data: dimensional models, reference data, and temporal data whose attributes change over time (e.g. slowly changing dimensions), and evolve those models safely as upstream sources change their schemas and semantics;
Develop and operate workloads on our lakehouse platform (Databricks / Spark / Delta) and our data warehouse (Redshift), including migrating existing pipelines from the warehouse to the lakehouse;
Orchestrate pipelines with Airflow: scheduling, dependencies, retries, backfills and alerting;
Prove correctness, not just completion: design parity checks and reconciliation queries when replacing an existing pipeline, run large historical backfills, and investigate data discrepancies down to the row level;

Proficiency in Python and using it as the primary development language in recent years;
Expert-level SQL: comfortable writing, reading and tuning complex analytical queries against very large tables, and debugging why two result sets disagree;
Hands-on experience with a distributed data processing platform (Spark/Databricks strongly preferred; EMR, Snowflake or BigQuery also relevant) and with a columnar data warehouse (Redshift, Snowflake, BigQuery, ClickHouse, etc);
Ability to design complex data models: normalized, dimensional and temporal (slowly changing dimensions, effective-dated records, point-in-time correctness);
Experience with workflow orchestration tools (Airflow or similar): building DAGs, managing dependencies and running backfills;