Specialist Software Engineer - Python, Elastic

J ob Summary: We are seeking a highly skilled and motivated Specialist Software Engineer with deep expertise in Big Data technologies , data pipeline orchestration , and observability tooling . The ideal candidate will be responsible for designing, developing, and maintaining scalable data processing systems and integrating observability solutions to ensure system reliability and performance. Key Responsibilities: Big Data Engineering: Design and implement robust data pipelines using Apache Kafka , Apache NiFi , Apache Spark , and Sqoop . Manage and optimize distributed data storage systems including Hadoop , HDFS , Druid , and ElasticSearch . Integrate and maintain data visualization and monitoring tools like Kibana , Grafana , and Logstash . Ensure efficient data ingestion, transformation, and delivery across various platforms. Programming & Scripting: Develop automation scripts and data processing utilities using Python 3 and Shell scripting . Build reusable components and libraries for data manipulation and system integration. Observability & Monitoring: Implement and configure observability agents such as Fluentd , Telegraf , and Logstash . Collaborate with platform teams to integrate OpenTelemetry for distributed tracing and metrics collection (good to have). Maintain dashboards and alerts for system health and performance monitoring. DevOps & CI/CD: Contribute to CI/CD pipeline development using GitHub Actions . Collaborate with DevOps teams to ensure seamless deployment and integration of data services. Collaboration & Documentation: Work closely with cross-functional teams including data scientists, platform engineers, and product managers. Document system architecture, data flows, and operational procedures. Participate in code reviews, knowledge sharing sessions, and technical mentoring. Required Skills & Qualifications: Bachelor’s or Master’s degree in Computer Science, Information Technology, or related field. 5+ years of experience in Big Data engineering and scripting. Strong hands-on experience with: Kafka, NiFi, Hadoop, HDFS, Spark, Sqoop ElasticSearch, Druid, Kibana, Grafana Python3, Shell scripting Logstash, Fluentd, Telegraf

Familiarity with GitHub Actions and basic DevOps practices. Exposure to OpenTelemetry is a plus. Excellent problem-solving, analytical, and communication skills. Preferred Qualifications: Experience in building real-time data streaming applications. Knowledge of data governance, security, and compliance in Big Data environments. Certifications in Big Data technologies or cloud platforms (AWS/GCP/Azure) are a plus.