Data Engineer

Role Title: Sr Data Engineer I Responsibilities: Pipeline Development & Operations: • Design, maintain, and optimize Spark and Flink pipelines running in Databricks and Airflow. • Ensure reliability and availability of data workflows in production. • Debug and resolve issues in notebooks, jobs, and orchestration workflows. Performance Optimization: • Tune Spark jobs, clusters, and data pipelines for performance and cost efficiency. • Analyze logs, queries, and workloads to identify bottlenecks and implement improvements. Monitoring & Observability: • Build and maintain dashboards, alerts, and metrics with Datadog, CloudWatch, or Prometheus. • Proactively monitor pipeline health and escalate issues as needed. Incident Response & RCA: • Participate in incident response for pipeline issues, helping diagnose and resolve outages. • Contribute to RCA and preventative actions to improve long-term stability. Collaboration & Knowledge Sharing: • Partner with data engineering, BI, and platform teams to support delivery and operations. • Document best practices, contribute to runbooks, and mentor junior engineers. Basic Qualifications: • 6+ years of experience in data engineering or big data platform operations. • Hands-on experience with Databricks, Spark, Flink, or Airflow (MWAA) in production. • Strong SQL skills and experience with ETL/ELT pipeline development. • Familiarity with AWS services (S3, Glue, Athena, EMR, EKS) for pipeline management. • Working knowledge of observability tools (Datadog, CloudWatch, Prometheus). • Proficiency in Python, Scala, or Java for pipeline development and automation. • Familiarity with infrastructure-as-code (Terraform or similar). • Strong collaboration and problem-solving skills, with experience in distributed teams. Preferred Qualifications: • Experience with incident response and RCA in production data environments. • Familiarity with CI/CD pipelines (GitLab, Jenkins, or GitHub Actions). • Exposure to data governance, access control, and cost optimization practices. • Certifications such as Databricks Data Engineer Associate or AWS Solutions Architect – Associate. Required Education: • Bachelor's degree in Computer Science, Engineering, or related technical field (or equivalent experience).