Data Streaming and Testing Engineer_Immediate

Responsibilities- Develop and test real-time data streaming pipelines using Apache Kafka, Apache Spark Streaming, or Apache Flink. Validate streaming data for accuracy, completeness, consistency, and timeliness. Design and execute test cases for batch and streaming data pipelines. Write SQL and Python scripts for data validation and automated testing. Monitor data pipelines, identify failures, and troubleshoot production issues. Perform ETL and end-to-end testing for data ingestion workflows. Verify schema changes and ensure data quality across systems. Collaborate with data engineers, developers, QA teams, and business stakeholders. Automate testing and integrate it into CI/CD pipelines. Required Skills- Apache Kafka Apache Spark (Spark Streaming or Structured Streaming) SQL Python ETL Testing Data Validation Linux/Unix Git REST APIs JSON, Avro, or Parquet data formats Basic cloud knowledge (AWS, Azure, or GCP) Preferred Skills Apache Flink Kafka Connect Airflow Docker and Kubernetes Jenkins or Azure DevOps Monitoring tools (Grafana, Prometheus) .