Data Engineer

As a Data Engineer at our company, you will be responsible for designing, developing, and optimizing scalable data processing solutions using Spark, Scala/Python, Airflow, and Azure Cloud. Your main responsibilities will include: - Designing, developing, and maintaining scalable data pipelines and ETL/ELT workflows using Spark, Scala, and Python. - Building high-performance batch and streaming data processing solutions for large-scale datasets. - Configuring, optimizing, and troubleshooting Spark applications, cluster settings, and resource management for maximum performance. - Developing and managing workflow orchestration using Apache Airflow. - Implementing data engineering solutions on Azure Cloud services. - Collaborating with data architects, analysts, and business stakeholders to understand data requirements and deliver robust solutions. - Monitoring, maintaining, and enhancing data pipeline reliability, scalability, and performance. - Performing code reviews and ensuring adherence to coding standards and best practices. - Troubleshooting production issues and providing timely resolutions. - Contributing to the design and implementation of data platform architecture and modernization initiatives. You should possess the following qualifications and skills: - 58 years of experience in Data Engineering and Big Data technologies. - Strong hands-on experience with Apache Spark, including Spark architecture, performance tuning, configuration, and cluster optimization. - Good experience in Scala and/or Python development. - Experience with Apache Airflow for workflow orchestration and scheduling. - Strong knowledge of Azure Cloud services and data engineering ecosystem. - Experience with distributed data processing and large-scale data platforms. - Good understanding of ETL/ELT concepts and data pipeline development. - Strong problem-solving and debugging skills. Preferred qualifications include experience with Azure Data Lake, Azure Databricks, Azure Synapse, or related Azure data services, knowledge of CI/CD practices and DevOps methodologies, experience working in Agile/Scrum environments, familiarity with data warehousing concepts and data modeling, and exposure to real-time data processing frameworks. Educational Qualification: Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field. Key Competencies required for this role include strong analytical and troubleshooting skills, ability to work independently and in a collaborative team environment, excellent communication and stakeholder management skills, and a focus on performance optimization, scalability, and solution quality. As a Data Engineer at our company, you will be responsible for designing, developing, and optimizing scalable data processing solutions using Spark, Scala/Python, Airflow, and Azure Cloud. Your main responsibilities will include: - Designing, developing, and maintaining scalable data pipelines and ETL/ELT workflows using Spark, Scala, and Python. - Building high-performance batch and streaming data processing solutions for large-scale datasets. - Configuring, optimizing, and troubleshooting Spark applications, cluster settings, and resource management for maximum performance. - Developing and managing workflow orchestration using Apache Airflow. - Implementing data engineering solutions on Azure Cloud services. - Collaborating with data architects, analysts, and business stakeholders to understand data requirements and deliver robust solutions. - Monitoring, maintaining, and enhancing data pipeline reliability, scalability, and performance. - Performing code reviews and ensuring adherence to coding standards and best practices. - Troubleshooting production issues and providing timely resolutions. - Contributing to the design and implementation of data platform architecture and modernization initiatives. You should possess the following qualifications and skills: - 58 years of experience in Data Engineering and Big Data technologies. - Strong hands-on experience with Apache Spark, including Spark architecture, performance tuning, configuration, and cluster optimization. - Good experience in Scala and/or Python development. - Experience with Apache Airflow for workflow orchestration and scheduling. - Strong knowledge of Azure Cloud services and data engineering ecosystem. - Experience with distributed data processing and large-scale data platforms. - Good understanding of ETL/ELT concepts and data pipeline development. - Strong problem-solving and debugging skills. Preferred qualifications include experience with Azure Data Lake, Azure Databricks, Azure Synapse, or related Azure data services, knowledge of CI/CD practices and DevOps methodologies, experience working in Agile/Scrum environments, familiarity with data warehousing concepts and data modeling, and exposure to real-time data processing frameworks. Educational Qualification: Bachelor's or Master's degree in