To Apply for this Job Click Here
Job Title: Senior Data Engineer, Azure Databricks
Location: Fully Remote (on-site in Dallas, Memphis, Cleveland, or Pittsburgh is optional)
Employment Status: Applicants must be currently authorized to work in the U.S. without the need for employment-based sponsorship, now or in the future, including: H-1B, L-1, TN, O-1, E-3, H-1B1, F-1, J-1, OPT, CPT, or any other employment-based visa programs.
Position Overview
The Senior Data Engineer will design, develop, and optimize enterprise data warehouse solutions and production data pipelines within an Azure Databricks environment. This hands-on position will support the continued development and modernization of the data ecosystem including the transition toward real-time, event-driven data processing, while ensuring data quality, reliability, and performance in a highly regulated environment.
Key Responsibilities
- Design, develop, and maintain enterprise data warehouse solutions and production pipelines across Bronze, Silver, and Gold layers using Azure Databricks, Delta Lake, and medallion architecture.
- Develop and optimize data ingestion, transformation, and modeling solutions using SQL, Python, PySpark, and Azure Data Factory.
- Build dimensional and harmonized data models, incorporating data conformance, survivorship logic, and standardized transformation processes.
- Support the transition from traditional batch processing toward continuous, low-latency, event-driven data ingestion using Azure Event Hubs, Structured Streaming, and emerging Databricks capabilities.
- Own assigned production data domains, ensuring pipeline reliability, data integrity, performance, and compliance with established service-level requirements.
- Monitor and optimize pipeline performance, processing latency, scalability, and Databricks compute costs.
- Troubleshoot complex production issues, perform root-cause analysis, and implement sustainable improvements to data reliability and system performance.
- Implement automated data validation, reconciliation, quality controls, and testing practices throughout the data lifecycle.
- Support data governance initiatives involving Unity Catalog, metadata management, lineage, access controls, and Gold-layer data assets.
- Modernize and consolidate legacy transformation processes while maintaining data accuracy and operational continuity.
- Establish and maintain engineering best practices for code quality, testing, documentation, version control, and CI/CD deployment within a highly regulated environment.
- Collaborate with Data Architecture, Analytics Engineering, QA, Data Governance, and business stakeholders to deliver reliable, scalable data solutions.
- Develop and enhance internal and external partner data integrations to support evolving business and operational requirements.
- Provide technical guidance and mentorship to Data Engineers through code reviews, knowledge sharing, and engineering best practices.
Qualifications
- 6+ years of progressive data engineering experience including hands-on development and production support of enterprise data platforms.
- Deep expertise in Azure Databricks, Delta Lake, Medallion Architecture, Unity Catalog, and production pipeline development and optimization.
- Hands-on experience developing and operating event-driven streaming data pipelines using Azure Event Hubs, Kafka, AWS Kinesis, or Confluent Cloud.
- Strong enterprise data warehousing experience encompassing dimensional modeling, complex transformations, data integration, and production operations.
- Advanced proficiency in SQL, Python, and PySpark, with the ability to independently develop, troubleshoot, review, and optimize complex data pipelines.
- Hands-on experience with Azure Data Lake Storage, Azure Data Factory, and CI/CD practices using Azure DevOps.
- Demonstrated ownership of production data pipelines, including monitoring, incident response, performance optimization, and service-level accountability.
- Experience implementing data contracts using ODCS or equivalent standards preferred.
- Experience implementing data validation, reconciliation, automated testing, and data quality controls.
- Strong understanding of data governance, metadata management, lineage, and secure data handling practices.
- Familiarity with data observability platforms such as Monte Carlo preferred.
- Experience operating within highly regulated environments including healthcare, pharmacy, pharmaceutical, medical, or another highly regulated industry involving sensitive clinical or operational data is preferred.
- Ability to analyze complex legacy data environments, improve existing architectures, and produce comprehensive technical documentation.
- Familiarity with Structured Streaming, Zerobus Ingestion, and Spark Declarative Pipelines preferred.
- Experience with enterprise data catalog platforms such as Atlan, Collibra, or equivalent technologies preferred.
- Experience developing governed data assets with documented metadata, lineage, and access controls.
- Strong communication, problem-solving, and cross-functional collaboration skills.
To Apply for this Job Click Here
Amy.Tillman