Source description
About the role
Job Overview We are seeking a meticulous Senior QA Engineer / Data Validation Engineer to own the quality assurance, data integrity, and model validation processes for our enterprise Demand Forecasting platform. Unlike traditional software testing, this role focuses heavily on data accuracy, pipeline reliability, and statistical outcome validation. You will work side-by-side with our Data Science and Data Engineering teams within Azure Databricks and Azure AI Foundry to ensure that data feeding our algorithms is flawless, pipelines are resilient, and the forecasted outputs match expected statistical and business boundaries. Key Responsibilities 1. Data Integrity & Pipeline Validation Design, implement, and execute comprehensive test plans to validate complex ETL/ELT pipelines within Azure Databricks and Azure Data Factory. Write advanced SQL queries and Python validation scripts to verify data transformations, aggregations, schema conformance, and data completeness across Delta Lakes. Implement automated data quality checks, including missing time-series intervals, duplicate records, null values, and out-of-bounds historical data. 2. Machine Learning & Forecasting Outcome Testing Establish testing frameworks to validate model inputs, features, and forecast outputs generated by XGBoost, LSTM, and similar forecasting models. Validate model performance metrics, including MAPE, RMSE, and WAPE, against historical baselines to support regression testing of model updates. Verify the integrity of model artifacts, versioning, and endpoint deployments managed within Azure AI Foundry and MLflow. Design sanity-testing logic to catch forecast anomalies such as negative demand predictions, massive unexplained spikes, or flatline forecasts. 3. Test Automation & MLOps Integration Build and maintain automated testing suites using Python-based frameworks such as PyTest, Excellent Expectations, or Great Expectations integrated with Databricks. Integrate data quality test Job Overview We are seeking a meticulous Senior QA Engineer / Data Validation Engineer to own the quality assurance, data integrity, and model validation processes for our enterprise Demand Forecasting platform. Unlike traditional software testing, this role focuses heavily on data accuracy, pipeline reliability, and statistical outcome validation. You will work side-by-side with our Data Science and Data Engineering teams within Azure Databricks and Azure AI Foundry to ensure that data feeding our algorithms is flawless, pipelines are resilient, and the forecasted outputs match expected statistical and business boundaries. Key Responsibilities 1. Data Integrity & Pipeline Validation Design, implement, and execute comprehensive test plans to validate complex ETL/ELT pipelines within Azure Databricks and Azure Data Factory. Write advanced SQL queries and Python validation scripts to verify data transformations, aggregations, schema conformance, and data completeness across Delta Lakes. Implement automated data quality checks, including missing time-series intervals, duplicate records, null values, and out-of-bounds historical data. 2. Machine Learning & Forecasting Outcome Testing Establish testing frameworks to validate model inputs, features, and forecast outputs generated by XGBoost, LSTM, and similar forecasting models. Validate model performance metrics, including MAPE, RMSE, and WAPE, against historical baselines to support regression testing of model updates. Verify the integrity of model artifacts, versioning, and endpoint deployments managed within Azure AI Foundry and MLflow. Design sanity-testing logic to catch forecast anomalies such as negative demand predictions, massive unexplained spikes, or flatline forecasts. 3. Test Automation & MLOps Integration Build and maintain automated testing suites using Python-based frameworks such as PyTest, Excellent Expectations, or Great Expectations integrated with Databricks. Integrate data quality test
More at G3. GG Tech Global