Data Engineer

Primary skills: Python, SQL, data lakes, azure Experience: 4+ years Immediate Joiner Key Responsibilities Pipeline Development & Automation · Design, build, and maintain CI/CD pipelines to automate deployment of DQ rules and data services across environments. · Optimize data pipelines and jobs for efficiency, scalability, and enterprise-grade reliability within Azure and Databricks. · Implement best practices in version control, testing, and automated deployments. Data Quality & Rule Development · Develop, optimize, and maintain DQ rules in PySpark/Python for self-serve capabilities. · Design and implement profiling frameworks for rule generation and automated remediation. · Ensure DQ frameworks align with enterprise standards, governance, and audit requirements. UI & Self-Serve Integration · Collaborate with front-end teams (Node.js/Angular) to enable rule configuration, validation, and monitoring via a low-code UI. · Develop APIs and services to expose DQ results and outputs for dashboards and self-service tools. Collaboration & Governance · Partner with Data Stewards, GPOs, and business owners to translate requirements into technical solutions. · Contribute to BRDs, SRDs, and design reviews for DQ rules and pipelines. · Provide inputs on governance, compliance, and change management processes. · Support periodic reviews and continuous improvements of pipelines, rules, and dashboards. Required Skills & Qualifications Strong programming skills in PySpark, Python, and SQL. Hands-on experience with Databricks (clusters, notebooks, Delta Lake). Experience designing and implementing CI/CD pipelines (Azure DevOps / GitHub Actions / Jenkins). Familiarity with Node.js & Angular for UI integration. Solid understanding of data governance, DQ frameworks, and MDM concepts. Strong analytical, problem-solving, and stakeholder engagement skills. Cloud expertise, preferably Azure Data Lake, Data Factory, and Synapse. Education & Experience Bachelor's or Master's degree in Computer Science, Information Systems, or related field. 4–8 years of relevant experience in data engineering, CI/CD, and DQ rule development. Proven expertise in Databricks, Azure data services, and low-code self-serve frameworks.