Data Engineer-Pyspark

JOB OVERVIEW We are seeking a Data Engineer who can work closely with cross-functional engineering, data science, and product teams to design, build, and enhance scalable data pipelines across batch and streaming systems. This role is responsible for maintaining high-quality, high-performance data products that support analytics, machine learning, personalization, and real-time business operations. The Data Engineer will contribute to quarterly business and technical objectives by modernizing core data assets, improving operational reliability, and ensuring best-in-class data quality standards. KEY RESPONSIBILITIES Design, build, and maintain scalable, efficient, and reliable data pipelines for ingestion, transformation, and integration across diverse data sources and destinations. Develop and optimize batch workflows (PySpark, SQL, orchestration) and support real-time/streaming pipelines (Kafka or similar) when applicable. Improve pipeline performance, cost efficiency, and scalability across large and complex datasets. Implement and maintain automated data quality checks, regression testing, and validation frameworks to ensure accuracy, reliability, and compliance with organizational standards Draft and review architectural diagrams, support processes to ensure clarity and alignment across engineering teams Work closely with engineering, product, and data science partners to deliver high-quality, production-ready data solutions REQUIRED SKILLS Data Processing: PySpark, SQL, Spark engine/architecture knowledge, performance tuning Programming: Python (Good to have) Cloud & Platforms: Databricks, Azure Streaming: Kafka or similar streaming platforms (nice to have) Version Control / CI-CD: Git, GitHub, GitHub Actions, CI/CD best practices Collaboration: JIRA, Confluence, MS Teams PREFERRED TRAITS Understanding modern software design patterns, data modeling practices, and distributed system fundamentals. Ability to write clean, testable, scalable code following engineering best practices. Follows established architecture patterns and engineering standards across the data platform. Proactively suggests improvements to minimize tech debt and increase reliability. Capable of identifying, triaging, and resolving data defects in both production and non-prod environments. Partners with QA/Automation teams to implement functional and data testing strategies. Experience working in Agile environments with cross-functional engineering teams of 5 or more. Collaborate effectively with outside teams to support product adoption and operational stability. Demonstrates strong technical communication skills: can explain trade-offs, ask the right questions, and provide/receive feedback constructively. PREFERRED TRAITS Understanding modern software design patterns, data modeling practices, and distributed system fundamentals. Ability to write clean, testable, scalable code following engineering best practices. Follows established architecture patterns and engineering standards across the data platform. Proactively suggests improvements to minimize tech debt and increase reliability. Capable of identifying, triaging, and resolving data defects in both production and non-prod environments. Partners with QA/Automation teams to implement functional and data testing strategies. Experience working in Agile environments with cross-functional engineering teams of 5 or more. Collaborate effectively with outside teams to support product adoption and operational stability. Demonstrates strong technical communication skills: can explain trade-offs, ask the right questions, and provide/receive feedback constructively. Skills: spark,pyspark,sql