Data Engineer (Java + Spark)

Job Description: As a Data Engineer, your role will involve designing and implementing client requirements, building data pipelines, and performing transformations using Java and Spark in Databricks. You will be responsible for creating automated workflows with triggers and scheduled jobs in Airflow and designing Airflow DAGs to orchestrate data processing jobs. Additionally, you will develop code to ensure timely notifications through email alerts in case of job failures or critical system issues. Your responsibilities will also include interacting directly with customers to gather requirements, providing technical input for analysis and design work, and supporting customer queries. Key Responsibilities: - Design and implement client requirements - Build data pipelines - Perform transformations using Java and Spark in Databricks - Create automated workflows with triggers and scheduled jobs in Airflow - Design Airflow DAGs to orchestrate data processing jobs - Develop code for timely email notifications in case of job failures - Interact directly with customers to gather requirements - Provide technical input for analysis and design work - Support customer queries Qualifications Required: - Proficiency in Java with a good understanding of its ecosystems - Experience in Data Engineering with Java and Spark - Basic knowledge of Linux/Linux scripting - Knowledge of Spark Architecture including Spark Core, Spark SQL, RDD, Data Set, and Data Frames - Performance tuning using optimization techniques, caching data in memory, broadcast, etc. - Familiarity with Azure/Cloud DevOps concepts and CI/CD pipeline - Understanding of Spark application architecture - Knowledge of deployment pipelines - Hands-on experience with Git Bash - Experience working with Scrum methodology - Strong communication skills for interacting with end clients via email and phone - Ability to understand business needs, identify data sources, and develop scalable data pipelines Note: The skills required for this role include DevOps, CI/CD, cloud platform, and Git Bash. Job Description: As a Data Engineer, your role will involve designing and implementing client requirements, building data pipelines, and performing transformations using Java and Spark in Databricks. You will be responsible for creating automated workflows with triggers and scheduled jobs in Airflow and designing Airflow DAGs to orchestrate data processing jobs. Additionally, you will develop code to ensure timely notifications through email alerts in case of job failures or critical system issues. Your responsibilities will also include interacting directly with customers to gather requirements, providing technical input for analysis and design work, and supporting customer queries. Key Responsibilities: - Design and implement client requirements - Build data pipelines - Perform transformations using Java and Spark in Databricks - Create automated workflows with triggers and scheduled jobs in Airflow - Design Airflow DAGs to orchestrate data processing jobs - Develop code for timely email notifications in case of job failures - Interact directly with customers to gather requirements - Provide technical input for analysis and design work - Support customer queries Qualifications Required: - Proficiency in Java with a good understanding of its ecosystems - Experience in Data Engineering with Java and Spark - Basic knowledge of Linux/Linux scripting - Knowledge of Spark Architecture including Spark Core, Spark SQL, RDD, Data Set, and Data Frames - Performance tuning using optimization techniques, caching data in memory, broadcast, etc. - Familiarity with Azure/Cloud DevOps concepts and CI/CD pipeline - Understanding of Spark application architecture - Knowledge of deployment pipelines - Hands-on experience with Git Bash - Experience working with Scrum methodology - Strong communication skills for interacting with end clients via email and phone - Ability to understand business needs, identify data sources, and develop scalable data pipelines Note: The skills required for this role include DevOps, CI/CD, cloud platform, and Git Bash.