Source description
About the role
As a Big Data Engineer, your main role will be to design, develop, and maintain end-to-end big data pipelines for enterprise applications. This involves working on Apache Hadoop and Apache Spark-based data processing frameworks to manage and optimize data pipelines, scripts, and production systems for performance and stability. Your responsibilities will include performing unit testing of data pipelines, supporting User Acceptance Testing (UAT) activities, and debugging and resolving issues in existing data pipelines, scripts, and production workflows. You will also be expected to modify and enhance existing data processing systems based on business requirements, review design, code, and deliverables to ensure high-quality output, and build and maintain scalable and secure big data solutions using cloud platforms. Key Responsibilities: - Design, develop, and maintain end-to-end big data pipelines for enterprise applications - Work on Apache Hadoop and Apache Spark-based data processing frameworks - Manage and optimize data pipelines, scripts, and production systems for performance and stability - Perform unit testing of data pipelines and support UAT activities - Debug and resolve issues in existing data pipelines, scripts, and production workflows - Modify and enhance existing data processing systems based on business requirements - Review design, code, and deliverables to ensure high-quality output - Build and maintain scalable and secure big data solutions using cloud platforms - Ensure implementation of data governance, security, and compliance standards - Mentor junior team members and contribute to knowledge sharing initiatives - Participate in Agile development practices and cross-team collaboration - Handle Proof of Concepts (PoCs) and deliver solutions within timelines Qualifications Required: - Experience with Apache Spark and Apache Hadoop - Proficiency in SQL and NoSQL databases In addition to the above responsibilities and qualifications, if any additional details of the company are provided in the job description, please provide them for a more comprehensive understanding of the role. As a Big Data Engineer, your main role will be to design, develop, and maintain end-to-end big data pipelines for enterprise applications. This involves working on Apache Hadoop and Apache Spark-based data processing frameworks to manage and optimize data pipelines, scripts, and production systems for performance and stability. Your responsibilities will include performing unit testing of data pipelines, supporting User Acceptance Testing (UAT) activities, and debugging and resolving issues in existing data pipelines, scripts, and production workflows. You will also be expected to modify and enhance existing data processing systems based on business requirements, review design, code, and deliverables to ensure high-quality output, and build and maintain scalable and secure big data solutions using cloud platforms. Key Responsibilities: - Design, develop, and maintain end-to-end big data pipelines for enterprise applications - Work on Apache Hadoop and Apache Spark-based data processing frameworks - Manage and optimize data pipelines, scripts, and production systems for performance and stability - Perform unit testing of data pipelines and support UAT activities - Debug and resolve issues in existing data pipelines, scripts, and production workflows - Modify and enhance existing data processing systems based on business requirements - Review design, code, and deliverables to ensure high-quality output - Build and maintain scalable and secure big data solutions using cloud platforms - Ensure implementation of data governance, security, and compliance standards - Mentor junior team members and contribute to knowledge sharing initiatives - Participate in Agile development practices and cross-team collaboration - Handle Proof of Concepts (PoCs) and deliver solutions within timelines Qualifications Required: - Experience with Apache Spark and Apache Hadoop - Proficiency in SQL and NoSQL databases In addition to the above responsibilities and qualifications, if any additional details of the company are provided in the job description, please provide them for a more comprehensive understanding of the role.
More at Logic Planet