Senior Data Engineer

Що робити

Design and build batch data pipelines that ingest, validate and transform multi-billion-row datasets from external data providers and internal systems
Model complex real-world data: dimensional models, reference data, and temporal data whose attributes change over time (e.g. slowly changing dimensions), and evolve those models safely as upstream sources change their schemas and semantics
Develop and operate workloads on our lakehouse platform (Databricks / Spark / Delta) and our data warehouse (Redshift), including migrating existing pipelines from the warehouse to the lakehouse
Orchestrate pipelines with Airflow: scheduling, dependencies, retries, backfills and alerting
Prove correctness, not just completion: design parity checks and reconciliation queries when replacing an existing pipeline, run large historical backfills, and investigate data discrepancies down to the row level

Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
7+ years of work experience as a data engineer
Proficiency in Python and using it as the primary development language in recent years
Expert-level SQL: comfortable writing, reading and tuning complex analytical queries against very large tables, and debugging why two result sets disagree
Hands-on experience with a distributed data processing platform (Spark/Databricks strongly preferred; EMR, Snowflake or BigQuery also relevant) and with a columnar data warehouse (Redshift, Snowflake, BigQuery, ClickHouse, etc)

Знайдено через trackr.help/jobs · Канал: @trackrhelp · Бот для персональних сповіщень: @trackrhelpBot