Senior Databricks Engineer The role requires significant daily overlap with the US team during US Eastern Time (EST) working hours. Candidates must be able to work within the required US EST schedule.
TechBar is a software services company working with clients across the US and Europe. We are investor-backed, growing steadily, and known for putting senior, thoughtful engineers in front of our clients.
We are looking for an experienced Senior Databricks Engineer to join an enterprise-level data modernization initiative for a US-based client.
The project focuses on modernizing a large-scale enterprise data platform as part of a broader cloud transformation journey. You will work with the Databricks Lakehouse Platform to build scalable data solutions supporting advanced analytics, AI/ML initiatives, and enterprise-wide data products.
What you’ll do
Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Spark SQL
Build robust ETL/ELT processes for large-scale structured and unstructured data
Implement Medallion Architecture (Bronze, Silver, Gold) and Delta Lake solutions
Develop batch and near real-time data processing solutions using Spark Streaming, Kafka, and Delta Live Tables (DLT)
Integrate data from enterprise sources, including SAP, Teradata, Mainframe, DB2, SQL Server, Oracle, Tibco, Kafka, and REST APIs
Work across both Microsoft Azure and AWS environments as part of the client's cloud data modernization program
Develop and maintain Databricks notebooks, workflows, jobs, and clusters
Implement Unity Catalog, access controls, data governance, and security standards
Support deployments and environment promotion across Dev, QA, UAT, and Production
Optimize workloads for performance, scalability, reliability, and cost efficiency
Implement data validation, reconciliation, monitoring, and automated data quality frameworks
Contribute to data lineage, metadata management, compliance, and enterprise data governance
Build reusable frameworks and automation for data ingestion, orchestration, monitoring, and quality management
Work closely with Data Scientists, BI developers, architects, and business stakeholders to translate requirements into scalable technical solutions
Participate in architecture and solution design discussions
Mentor junior engineers and contribute to engineering best practices
Participate in Agile ceremonies and collaborate with distributed engineering teams
Must-have requirements
7+ years of professional experience in Data Engineering
Strong hands-on experience with Databricks
Databricks certification is mandatory: Databricks Certified Data Engineer Associate or Professional
Strong practical experience with both Microsoft Azure and AWS. Experience in only one cloud platform is not sufficient.
Solid hands-on experience with:
Databricks Workflows / Jobs
Delta Lake
Delta Live Tables (DLT)
Unity Catalog
Databricks SQL
Strong programming skills in Python, PySpark, Spark SQL, and SQL
Strong understanding of ETL/ELT, data modeling, data warehousing, and distributed data processing
Experience with Apache Kafka, Spark Streaming, and event-driven architectures
Hands-on experience with Azure, particularly:
Azure Databricks
Azure Data Factory
ADLS Gen2
Azure Key Vault
Hands-on experience with AWS, particularly:
Amazon S3
IAM
Experience working with enterprise databases such as:
Teradata
SQL Server
Oracle
DB2
Experience with Git and CI/CD, preferably Azure DevOps or GitHub Actions
Strong communication skills and the ability to work effectively with technical and non-technical stakeholders
Ability to work independently, take ownership, and contribute to architectural decisions
Nice to have
Azure Data Engineer Associate certification
Experience working on large-scale enterprise data modernization programs
Experience migrating workloads from Teradata or other legacy platforms to Databricks
Experience with SAP Datasphere, SAP BW, or SAP Busi
