Big Data Engineer

Role & responsibilities Must Have - 6+ years data engineering experience - Strong programming skills in Python - Hands on experience with Big Data tech (e.g., Hadoop) and Spark - Comfort working in Agile, highly collaborative teams - Strong written and verbal communication skills Good to have - Experience with cloud platforms (AWS EMR, S3 Lamda, or Azure), Terraform (provisioning and Infra), Scala, Java, Databricks - Performance tuning and code optimization experience Must Have 8+yrs - PySpark: Experience with Apache Spark using the PySpark for distributed data processing, including DataFrame operations, window functions, and Spark SQL. - Python Programming: Strong knowledge of Python, including advanced features and best practices. - AWS Services: Familiarity with AWS S3 (for data storage and access), EMR, Lambda, SQS, SNS, IAM etc and Boto3 (AWS SDK for Python). - Data Engineering: Understanding of ETL processes, schema design, and data validation. - SQL: Proficiency in writing and optimizing SQL queries, especially within Spark SQL - Strong analytical and Problem solving skills Preferred / Ideal to have - Experience with other AWS services - SQS, SNS, IAM etc - Infrastructure as Code (IaC) with tools like Terraform - Experience with Git and branching strategies and Code quality tools (SonarQube) - RESTful API development and documentation (e.g., Swagger/OpenAPI/Fast API) - Data pipeline orchestration (Airflow, Step Functions) - Scripting for automation (Bash, Python, PowerShell) Good to have - Any knowledge on Databricks is added advantage - Data Quality and Validation Frameworks WFO all 5days Bangalore/Hyderabad F2F interview 6+yrs Immediate only Mok@teksystems.com