Senior Databricks Data QA Engineer

Client: TCS | Engagement: Full-time | Work mode: ONSITE | Experience: 8-10 | Publisher job id: JOB-000606

Role Overview

We are seeking a highly skilled Data QA Engineer with extensive experience in ETL, Databricks, and AI/ML model validation to join our team. The ideal candidate will be responsible for ensuring the quality, integrity, and performance of large-scale data pipelines and machine learning integrations. Key Responsibilities: Validate end-to-end ETL/ELT pipelines and perform Source-to-Target reconciliation. Execute data quality checks on structured and semi-structured datasets, including schema and Delta Lake validations. Write complex SQL queries for backend validation and data integrity testing. Validate Databricks notebooks, Workflows, Jobs, Delta Live Tables, and Unity Catalog implementations. Test Bronze, Silver, and Gold layer data transformations and performance of Spark/PySpark processes. Validate AI/ML model integrations, including model scoring, data drift, feature engineering, and MLOps validation. Test GenAI/RAG workflows, prompt-response accuracy, and grounding mechanisms. Design and develop automation frameworks for ETL, API, and ML model validation using Python. Integrate automated tests into CI/CD pipelines and develop quality gates for deployment. Required Skills: 6+ years of experience in ETL/Data Testing. 3+ years of experience testing Databricks or Spark-based platforms. Hands-on experience in AI/ML Model Testing or MLOps validation. Strong coding proficiency in Python and SQL. Experience with batch and streaming data pipeline testing. Expertise in contract testing and API validation. Qualifications: 8-10 years of total relevant experience. Strong understanding of data flow between source systems, data lakes, and consuming applications.