Sr. Data Engineer

You will be responsible for: - Designing, implementing, and optimizing end-to-end data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data. - Developing data pipelines to extract and transform data in near real-time using cloud-native technologies. - Implementing data validation and quality checks to ensure accuracy and consistency. - Monitoring system performance, troubleshooting issues, and implementing optimizations to enhance reliability and efficiency. - Collaborating with business users, analysts, and other stakeholders to understand data requirements and deliver tailored solutions. - Documenting technical designs, workflows, and best practices to facilitate knowledge sharing and maintain system documentation. - Providing technical guidance and support to team members and stakeholders as needed. Desirable competencies include: - 8+ years of work experience. - Proficiency in writing complex SQL queries on MPP systems (Snowflake/Redshift). - Experience in Databricks and Delta tables. - Data engineering experience with Spark/Scala/Python. - Experience in Microsoft Azure stack (Azure Storage Accounts, Data Factory, and Databricks). - Experience in Azure DevOps and CI/CD pipelines. - Working knowledge of Python. - Comfortable participating in 2-week sprint development cycles. You will be responsible for: - Designing, implementing, and optimizing end-to-end data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data. - Developing data pipelines to extract and transform data in near real-time using cloud-native technologies. - Implementing data validation and quality checks to ensure accuracy and consistency. - Monitoring system performance, troubleshooting issues, and implementing optimizations to enhance reliability and efficiency. - Collaborating with business users, analysts, and other stakeholders to understand data requirements and deliver tailored solutions. - Documenting technical designs, workflows, and best practices to facilitate knowledge sharing and maintain system documentation. - Providing technical guidance and support to team members and stakeholders as needed. Desirable competencies include: - 8+ years of work experience. - Proficiency in writing complex SQL queries on MPP systems (Snowflake/Redshift). - Experience in Databricks and Delta tables. - Data engineering experience with Spark/Scala/Python. - Experience in Microsoft Azure stack (Azure Storage Accounts, Data Factory, and Databricks). - Experience in Azure DevOps and CI/CD pipelines. - Working knowledge of Python. - Comfortable participating in 2-week sprint development cycles.