Automation Test Lead Data Platform Engineering

Job Title: Automation Test Lead Data & Platform Engineering Overview We are seeking a highly hands on Automation Test Lead who can architect and scale enterprise grade test automation frameworks across data platforms, APIs, and cloud native systems. This role is not about managing testing. It is about owning quality engineering as a discipline . You will design frameworks that teams adopt, enforce standards that prevent bad releases, and build systems that proactively catch failures before they hit production. You should be equally comfortable writing Python frameworks, designing data validation strategies, and challenging engineers on weak implementations . Also embrace AI ways of working wherever possible to accelerate automation. Core Expectations You do not wait for bugs, you design systems that make bugs difficult to exist You build frameworks that multiple teams adopt without friction You challenge architecture, not just test outcomes You reduce false positives, flaky tests, and wasted compute You think in terms of data reliability, contracts, and system behaviour under stress Key Responsibilities Test Automation Architecture & Strategy Own the design and evolution of a scalable, modular, metadata driven test automation framework supporting: Data pipelines (batch and streaming) APIs and backend services End to end data product validation Enable plug and play components, parallel execution, environment isolation, and deterministic runs. Data Testing Framework Engineering Build reusable frameworks supporting: SQL based assertions and reconciliation Schema validation and evolution checks Data contracts and producer consumer validation Lineage and freshness validation Config driven test definitions (YAML or JSON) Drive adoption by making frameworks easy to integrate into existing pipelines. Break Systems Proactively (Destructive Testing Mindset) Design tests for: Schema drift and backward incompatibility Late arriving data and partial failures Duplicate, missing, or out of order events High volume stress and concurrency issues Failure scenarios such as retries, DLQ handling, and backpressure ETL and Streaming Validation at Scale Implement: Row, aggregate, and hash based reconciliation Incremental and backfill validation Exactly once or at least once semantics validation Window based and time based correctness checks Data Quality & Observability Integration Integrate and extend tools like: Great Expectations, Soda Build custom validations for: Accuracy, completeness, uniqueness, timeliness Expose quality metrics, SLAs, and test outcomes through dashboards. CI/CD & DataOps Enforcement Embed testing into pipelines: Pre merge gates and release blockers Selective and parallel test execution Integration with GitHub Actions, Jenkins, or similar Ensure no pipeline reaches production without passing quality gates. Test Data Management Define strategies for: Synthetic data generation Masking and anonymization Deterministic datasets for repeatable testing Edge case and boundary condition simulation Performance & Reliability Testing Design and execute: Pipeline and query performance benchmarks Concurrency and stress testing Data skew and partitioning analysis Continuously optimize for cost and execution time. Security & Compliance Validation Automate checks for: PII or PHI exposure Encryption and access control Data retention and audit requirements Support compliance frameworks like GxP, SOX, or ISO standards. Cross Functional Quality Leadership Work with data engineers, platform teams, and architects to embed quality early Challenge poor design decisions that impact reliability Mentor engineers to adopt strong testing practices Incident Analysis & Prevention Analyze production data issues and failure patterns Reduce flaky tests and false alarms Drive root cause fixes, not surface level patches Mandatory Skills Programming & Framework Development Strong Python with experience building test frameworks, libraries, and CLI tools Advanced SQL for validation and reconciliation Data Engineering Ecosystem Hands on with DBT, Airflow, and Snowflake or similar platforms Strong understanding of ETL or ELT and data modeling concepts Data Quality & Observability Experience with Great Expectations, Soda, or similar tools Understanding of lineage and catalog systems Streaming Systems Experience testing Kafka, Kinesis, or similar systems Understanding of delivery semantics and event processing challenges CI/CD & DevOps Strong Git workflows CI/CD pipelines using Jenkins, GitHub Actions, or equivalent Cloud & Infrastructure AWS experience across data and compute services Familiarity with IaC tools like Terraform or CloudFormation Desired Skills Experience with API contract testing and tools like PACT Basic UI automation for end to end validation Exposure to data mesh or data product architectures Observability stacks such as Prometheus or Grafana Experience in regulated domains (healthcare, life sciences, finance) Experience 9+ years in automation or quality engineering 5+ years building frameworks for data platforms at scale Proven track record of improving system reliability and reducing production incidents What Success Looks Like (First 6–12 Months) A widely adopted, low friction test automation framework across teams Significant reduction in production data incidents and schema related failures Strong enforcement of quality gates in CI/CD pipelines Measurable improvement in data SLAs and observability Reduced test execution time with higher coverage and lower flakiness