Data Engineer

Job Summary:
We are seeking a skilled Data Engineer to design, build, and maintain high-quality, scalable data pipelines with a strong focus on Silver and Gold layer transformations, data quality automation, and CI/CD-driven deployments. The ideal candidate will have hands-on experience with Python, PySpark, Git-based CI/CD pipelines, ARM templates, and data quality libraries such as Great Expectations.
Key Responsibilities:
• Design and implement Silver and Gold layer data models following medallion architecture best practices.
• Perform data cleansing, standardization, enrichment, and aggregation to support analytics and reporting.
• Build optimized PySpark-based transformations for large-scale data processing.
• Ensure data consistency, performance, and scalability across datasets.
• Build and maintain CI/CD pipelines using Git-based workflows (Azure DevOps / GitHub).
• Use ARM templates (or IaC equivalents) for automated infrastructure provisioning.
• Enable automated deployment of data pipelines, notebooks, and configurations.
• Follow DevOps best practices for version control, branching, and release management.
• Develop reusable Python-based automation scripts for pipeline orchestration, validation, and monitoring.
• Create modular, maintainable, and testable Python code.
• Support automation of metadata, logging, alerting, and operational tasks.
• Implement data quality libraries such as Great Expectations.
• Define and automate data quality rules (completeness, accuracy, freshness, consistency).
• Integrate data quality checks into data pipelines and CI/CD workflows.
• Monitor, log, and troubleshoot data quality issues proactively.
• Work closely with data architects, analysts, QA, and business stakeholders.
• Translate business and analytical requirements into robust data engineering solutions.
• Participate in Agile ceremonies and support sprint-based delivery.
• Provide production support, performance tuning, and troubleshooting
Required Skills and Qualifications:
5-10+ years of experience in data engineering or analytics engineering.
Strong hands-on experience with Silver & Gold layer development, Python (automation and data processing) and PySpark. Also with Great Expectations (data quality framework) and Experience with CI/CD pipelines using Git-based tools. Hands-on experience with ARM templates or infrastructure-as-code concepts. Strong understanding of data modeling and medallion architecture. Experience working with large datasets in distributed environments.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.