Software Engineer

Responsibilities

• Design and implement highly efficient Spark applications in Java, focusing on performance optimization, including tuning Spark jobs, optimizing transformation, and leveraging advanced Spark features for batch and real-time processing.

• Responsible for developing data pipelines using Apache Spark (Big Data Processing )

• Responsible for analyzing performance concerns and issues. Identifying the bottlenecks that require changes to improve performance.

• Utilize in-depth knowledge of functional and technical expertise in Java/J2EE Micro-services (Spring Boot) and Big Data in delivering solutions.

• Responsible for configuring the applications for DevOps using Jenkins and GIT and build pipeline using Jenkins.

• Deploy the applications to the cloud environment using various cloud services and maintaining the application in Open Shift and AWS

• Responsible for developing Staging Layer using Delta Lake for the Ingestion Framework.

• Implement and manage data publishing and consumption using Kafka.

• Implement solutions to transform and serialize datasets into JSON messages using Spark SQL

• Understanding of RESTful web services and API development .

Skills

• Technologies & Skills: Java, Apache Spark, Delta Lake, Spark SQL, MySQL, Kafka, drools, Spring Boot, REST APIs, Maven, Jenkins, Docker, Kubernetes