Senior Data Engineer (Azure Databricks)

Zoral

Що робити

Review an existing data lake, its pipelines, jobs, notebooks and code as deployed; assess code quality, operability, testing and deployment practice.
Build the inventory of sources, feeds, entities, keys, volumes and refresh patterns.
Profile data at scale and establish a data-quality baseline (completeness, validity, consistency, uniqueness, timeliness); propose a candidate set of data-quality rules.
Design and build ingestion (batch, change-data capture, files, APIs) and transformation pipelines across layered architectures, including Data Vault 2.0 integration layers and Kimball dimensional models.
Implement data cleansing and standardisation, de-duplication and customer matching (entity resolution), and build a proof of concept on a data sample with measured precision and recall.

6+ years of data engineering, including at least 3 years on Databricks in production.
Expert SQL and strong Python with PySpark.
Hands-on with Delta Lake, Unity Catalog, Lakeflow Declarative Pipelines (formerly Delta Live Tables), Lakeflow Jobs (formerly Workflows), Auto Loader and Databricks Asset Bundles.
Hands-on experience with, or strong exposure to, Microsoft Fabric (Lakehouse, Data Warehouse, Real-Time Intelligence, semantic models).
Working knowledge of the Azure data estate: ADLS Gen2, Azure Data Factory, Azure Key Vault, Azure Functions and Azure Monitor.

Знайдено через trackr.help/jobs · Канал: @trackrhelp · Бот для персональних сповіщень: @trackrhelpBot