Senior Data Engineer, Azure Databricks

,    •    Direct Hire   •    November 12, 2026

Back to Career Search   

To Apply for this Job Click Here

Job Title: Senior Data Engineer, Azure Databricks
Location: Fully Remote  (on-site in Dallas, Memphis, Cleveland, or Pittsburgh is optional)
Employment Status: Applicants must be currently authorized to work in the U.S. without the need for employment-based sponsorship, now or in the future, including: H-1B, L-1, TN, O-1, E-3, H-1B1, F-1, J-1, OPT, CPT, or any other employment-based visa programs.


Position Overview

The Senior Data Engineer will design, develop, and optimize enterprise data warehouse solutions and production data pipelines within an Azure Databricks environment.  This hands-on position will support the continued development and modernization of the data ecosystem including the transition toward real-time, event-driven data processing, while ensuring data quality, reliability, and performance in a highly regulated environment.

Key Responsibilities
 

• Design, develop, and maintain enterprise data warehouse solutions and production pipelines across Bronze, Silver, and Gold layers using Azure Databricks, Delta Lake, and medallion architecture.
• Develop and optimize data ingestion, transformation, and modeling solutions using SQL, Python, PySpark, and Azure Data Factory.
• Build dimensional and harmonized data models, incorporating data conformance, survivorship logic, and standardized transformation processes.
• Support the transition from traditional batch processing toward continuous, low-latency, event-driven data ingestion using Azure Event Hubs, Structured Streaming, and emerging Databricks capabilities.
• Own assigned production data domains, ensuring pipeline reliability, data integrity, performance, and compliance with established service-level requirements.
• Monitor and optimize pipeline performance, processing latency, scalability, and Databricks compute costs.
• Troubleshoot complex production issues, perform root-cause analysis, and implement sustainable improvements to data reliability and system performance.
• Implement automated data validation, reconciliation, quality controls, and testing practices throughout the data lifecycle.
• Support data governance initiatives involving Unity Catalog, metadata management, lineage, access controls, and Gold-layer data assets.
• Modernize and consolidate legacy transformation processes while maintaining data accuracy and operational continuity.
• Establish and maintain engineering best practices for code quality, testing, documentation, version control, and CI/CD deployment within a highly regulated environment.
• Collaborate with Data Architecture, Analytics Engineering, QA, Data Governance, and business stakeholders to deliver reliable, scalable data solutions.
• Develop and enhance internal and external partner data integrations to support evolving business and operational requirements.
• Provide technical guidance and mentorship to Data Engineers through code reviews, knowledge sharing, and engineering best practices.

• 6+ years of progressive data engineering experience including hands-on development and production support of enterprise data platforms.
• Deep expertise in Azure Databricks, Delta Lake, Medallion Architecture, Unity Catalog, and production pipeline development and optimization.
• Hands-on experience developing and operating event-driven streaming data pipelines using Azure Event Hubs, Kafka, AWS Kinesis, or Confluent Cloud.
• Strong enterprise data warehousing experience encompassing dimensional modeling, complex transformations, data integration, and production operations.
• Advanced proficiency in SQL, Python, and PySpark, with the ability to independently develop, troubleshoot, review, and optimize complex data pipelines.
• Hands-on experience with Azure Data Lake Storage, Azure Data Factory, and CI/CD practices using Azure DevOps.
• Demonstrated ownership of production data pipelines, including monitoring, incident response, performance optimization, and service-level accountability.
• Experience implementing data contracts using ODCS or equivalent standards preferred.
• Experience implementing data validation, reconciliation, automated testing, and data quality controls.
• Strong understanding of data governance, metadata management, lineage, and secure data handling practices.
• Familiarity with data observability platforms such as Monte Carlo preferred.
• Experience operating within highly regulated environments including healthcare, pharmacy, pharmaceutical, medical, or another highly regulated industry involving sensitive clinical or operational data is preferred.
• Ability to analyze complex legacy data environments, improve existing architectures, and produce comprehensive technical documentation.
• Familiarity with Structured Streaming, Zerobus Ingestion, and Spark Declarative Pipelines preferred.
• Experience with enterprise data catalog platforms such as Atlan, Collibra, or equivalent technologies preferred.
• Experience developing governed data assets with documented metadata, lineage, and access controls.
• Strong communication, problem-solving, and cross-functional collaboration skills.

To Apply for this Job Click Here

Sign up for job alerts