Source description
About the role
Senior Data QA Engineer Big Data & Azure Databricks Experience: 510 Years Location: Remote CTC: Up to 22 LPA Position: Senior Engineer Environment: 60% On-Premise Big Data | 40% Azure Cloud Role Overview We are looking for an experienced Senior Data QA Engineer / Big Data Test Engineer with solid hands-on expertise in Hadoop, Hive, PySpark, Advanced SQL, Python, and Azure Databricks. The role involves validating large-scale data pipelines, performing source-to-target reconciliation, testing ETL/data transformations, and automating data validation across on-premise Hadoop ecosystems and Azure cloud environments. Key Responsibilities - Perform end-to-end ETL/Data Warehouse and Big Data testing across source, transformation, and target layers. - Validate large datasets within Hadoop/HDFS, including CSV and Parquet files. - Write complex HiveQL and Advanced SQL queries for data validation, reconciliation, and transformation testing. - Use PySpark/Spark for large-scale data processing and validation. - Perform data testing and validation within Azure Databricks. - Navigate Databricks workspace, manage files using commands such as %fs, and work with Unity Catalog. - Validate data movement and consistency between on-prem Hadoop/Hive and Azure/cloud environments. - Develop Python-based automation scripts for data validation and reconciliation. - Perform source-to-target, record-level, aggregate-level, and apple-to-apple data comparison. - Work with Snowflake or similar cloud data warehouse platforms. - Use UNIX/Linux shell scripting for file validation, log analysis, and ETL testing activities. - Monitor and validate scheduled ETL/data pipeline jobs using Control-M. - Work with database environments including SQL Server/SSMS. - Track requirements, user stories, defects, and test activities using Jira and Confluence. Must-Have Skills - Hadoop / HDFS Strong hands-on experience - Apache Hive / HiveQL Strong hands-on experience - PySpark / Apache Spark Strong h .