Data Engineer (Spark)

Please make sure you are well experienced with Azure Synapse/Databricks and Apache Spark. Without these skills, please do not apply for this position. We are also looking for someone who can join immediately(Within a week). Job Description We are seeking a skilled Data Engineer with proficient knowledge in Spark and SQL to join our dynamic team. The ideal candidate will be responsible for designing, implementing, and optimizing data pipelines on our Data platform. You will work closely with data architects, and other stakeholders to ensure data accessibility, reliability, and performance. Key Responsibilities Data Pipeline Development : Design, develop, and maintain scalable data pipelines using Azure Synapse, Databricks & Apache Spark (PySpark). Data Integration : Integrate data from various sources, ensuring data quality and consistency. Performance Optimization : Optimize data processing workflows for performance and cost-efficiency. Collaboration : Work with data architects, analysts, and product owners to understand data requirements and deliver solutions. Monitoring and Troubleshooting : Monitor data pipelines and troubleshoot issues to ensure data integrity. Documentation : Document data workflows, processes, and best practices. Skills Technical Skills : Proficiency in Azure Synapse/Databricks and Apache Spark. Strong PySpark and SQL skills for data manipulation and querying. Familiarity with Delta Live Tables and Databricks workflows. Experience with ETL tools and processes. Knowledge of cloud platforms (AWS, Azure, GCP). Soft Skills : Excellent problem-solving abilities. Strong communication and collaboration skills. Ability to work in a fast-paced environment and manage multiple priorities.