Senior Data Engineer (Databricks / AWS)

Що робити

Build the lakehouse: Design and develop scalable batch and streaming pipelines on Databricks using Spark and Delta Lake, following a medallion architecture (Bronze / Silver / Gold).
Own data on AWS: Work with S3, IAM, and related AWS services to ingest, store, and secure data from internal systems and third-party financial data providers.
Model for the business: Build clean, scalable, and well-documented data models for analytics, reporting, and downstream consumers.
Ensure data quality and trust: Implement data quality checks, monitoring, and lineage. Manage access and governance using Unity Catalog.
Optimize performance and cost: Tune Spark jobs, cluster configurations, and storage layouts to improve performance and control cloud costs.

6+ years of experience in data engineering, including significant hands-on experience with Databricks in production.
Strong knowledge of Apache Spark / PySpark and advanced SQL.
Solid Python skills for data engineering.
Hands-on experience with AWS as the underlying cloud platform for data systems, including S3, IAM, and related data services.
Experience with Delta Lake and lakehouse / medallion architecture.

Experience migrating from Snowflake to Databricks or working on other large-scale data platform migrations.
Experience with Unity Catalog governance and access management.
Experience building streaming pipelines using Structured Streaming, Kafka, or Kinesis.
Experience with Terraform or Databricks Asset Bundles.
Experience with dbt.

Знайдено через trackr.help/jobs · Канал: @trackrhelp · Бот для персональних сповіщень: @trackrhelpBot