Data Engineering Manager, Azure Databricks
Job Details
Data Engineering Manager, Azure Databricks
, • Direct Hire • November 12, 2026
To Apply for this Job Click Here
Job Title: Data Engineering Manager, Azure DatabricksLocation: Fully Remote (on-site in Dallas, Memphis, Cleveland, or Pittsburgh is optional)
Employment Status: Applicant must be currently authorized to work in the U.S. without the need for employment-based sponsorship, now or in the future, including: H-1B, L-1, TN, O-1, E-3, H-1B1, F-1, J-1, OPT, CPT, or any other employment-based visa programs.
Position Overview
Lead and develop a team of data engineering professionals while overseeing the design, development, and ongoing optimization of the Azure Databricks platform and enterprise data warehouse, supporting business analytics, reporting, and the evolution toward real-time, event-driven data integration.
Key Responsibilities
• Own source-system ingestion, Delta Lake pipelines, Medallion Architecture, and enterprise data warehouse design, development, and operations within Azure Databricks.
• Provide hands-on technical leadership across Azure Databricks, Azure Data Lake Storage, Unity Catalog, PySpark, Python, and SQL including architecture reviews, code reviews, performance optimization, and technical mentoring.
• Support transition from batch processing to continuous, near-real-time data ingestion and processing using Azure Event Hubs and Structured Streaming to meet demanding service-level requirements within a highly regulated environment.
• Monitor and optimize data warehouse reliability, processing performance, scalability, and cost efficiency while maintaining the integrity and availability of critical business data.
• Direct production incident response, troubleshoot complex pipeline and data processing issues, and establish proactive monitoring, documentation, and knowledge-sharing practices.
• Establish and enforce engineering standards for code quality, testing, deployment, security, and documentation, ensuring reliable, governed data operations within a highly regulated healthcare environment.
• Collaborate with Data Architecture, Analytics Engineering, QA, and Data Governance on Gold-layer design, data validation, lineage, metadata, and observability.
• Support internal business requirements and external partner data services through the development, enhancement, and integration of data pipelines.
• Lead, coach, mentor, and develop a team of Data Engineers, establishing clear accountability, managing performance, and fostering a collaborative, high-performing engineering culture prepared for significant growth.
Qualifications
• 7+ years of progressive data engineering experience, including 2+ years of team leadership or people management.
• Deep, hands-on Databricks expertise, including medallion architecture, Delta Lake, Unity Catalog, and production pipeline development and operations.
• Strong enterprise data warehouse experience encompassing architecture, data modeling, transformation logic, and production operations.
• Proficiency in PySpark, SQL, and Python with the technical depth to review code, evaluate architectural decisions, and mentor engineers.
• Proven experience with Databricks, Azure Data Lake Storage, and Azure Event Hubs.
• Strong experience developing and operating data pipelines, preferably with Azure Event Hubs or comparable technologies such as Kafka, Kinesis, or Confluent Cloud.
• Demonstrated ability to lead and develop engineers, manage competing priorities, and maintain accountability in a high-volume production environment.
• Strong communication, organizational, and cross-functional collaboration skills with the ability to work effectively across technical, business, partner, and client teams.