Consultant Data Engineer IRC302341 (UK)

Що робити

Lakehouse Engineering: Build and maintain scalable data pipelines to migrate Consumer Risk datasets from Teradata to Amazon S3, utilizing Apache Iceberg for the lakehouse layer and Databricks for governed warehousing.
Code Modernization (BTEQ to dbt): Perform complex code conversions, transforming up to 500 legacy Teradata BTEQ scripts into modular, version-controlled, and highly performant dbt (data build tool) models.
Pipeline Orchestration: Design, develop, and monitor complex Directed Acyclic Graphs (DAGs) using Astronomer (Apache Airflow) to orchestrate end-to-end data workflows.
Data Lineage & Transparency: Integrate pipelines with OpenLineage to ensure complete, end-to-end data traceability, providing the transparency required for regulatory compliance.
Reconciliation & Parity: Work alongside the QA/Automation team to execute automated reconciliation frameworks, debugging and resolving data discrepancies to ensure strict attribute-level parity and timing equivalence during governed parallel runs

Cloud Platforms: Hands-on engineering experience within the AWS ecosystem (S3, IAM, Glue) and deep expertise in Databricks (SparkSQL, PySpark, governed cataloging).
Modern Data Stack: Strong proficiency in dbt (Core or Cloud) for data transformation and understanding of modern table formats, specifically Apache Iceberg.
Orchestration: Experience building and maintaining data pipelines using Apache Airflow (Astronomer experience is a strong plus).
Languages: Expert-level SQL (ability to read Teradata SQL/BTEQ and write highly optimized Spark/Databricks SQL) and strong Python skills for pipeline development and Airflow DAGs.
Legacy Systems: Familiarity with Teradata architecture and utilities (FastExport, BTEQ) to effectively understand the source systems being decommissioned.

Our goal is to build an inclusive positive culture where everyone can feel comfortable being themselves, empowering people to create their own high standards and therefore more value. We work together to promote fairness while recognising, valuing and embracing differences – providing a transparent support structure and generous training budget to help our people develop skills to progress thei...
You'll benefit from a comprehensive health and wellness plan, private healthcare (clinical and mental wellbeing), discounted gym memberships, and in office yoga sessions and massages. We offer a f

Знайдено через trackr.help/jobs · Канал: @trackrhelp · Бот для персональних сповіщень: @trackrhelpBot