Padmi

Data Engineer with Databricks & GCP

MumbaiPosted 2 months ago
Software engineeringSeniorFull Time; Regular
Apply at IIT JOBS INC

Opens the source posting on shine.com

Source description

About the role

View original

You will design and lead data engineering solutions using Databricks and Google Cloud Platform. You will own the architecture, implementation, and optimization of large-scale data pipelines. Responsibilities Architect and build data models, schemas, and governance frameworks. Develop and optimize data pipelines using Databricks (Autoloader, DLT, Delta Lake, CDF) and PySpark. Implement CI/CD pipelines and infrastructure-as-code using Terraform. Provide architectural guidance and lead technical teams. Evaluate and integrate Gen AI concepts into practical data applications. Required Skills 10+ years of experience as a Data Engineer or Architect. Extensive hands-on experience with Databricks, including Autoloader, DLT, Delta Lake, and CDF. Strong expertise in PySpark and Python for data engineering and automation. Advanced SQL skills with a focus on query optimization. Deep proficiency with GCP services: BigQuery, Cloud Functions, Cloud Run, Pub/Sub, Dataflow, and GCS. Proven experience with CI/CD pipelines and DevOps practices. Experience with infrastructure-as-code, specifically Terraform. Strong leadership and stakeholder management skills. You will design and lead data engineering solutions using Databricks and Google Cloud Platform. You will own the architecture, implementation, and optimization of large-scale data pipelines. Responsibilities Architect and build data models, schemas, and governance frameworks. Develop and optimize data pipelines using Databricks (Autoloader, DLT, Delta Lake, CDF) and PySpark. Implement CI/CD pipelines and infrastructure-as-code using Terraform. Provide architectural guidance and lead technical teams. Evaluate and integrate Gen AI concepts into practical data applications. Required Skills 10+ years of experience as a Data Engineer or Architect. Extensive hands-on experience with Databricks, including Autoloader, DLT, Delta Lake, and CDF. Strong expertise in PySpark and Python for data engineering and automation. Advanced SQL skills with a focus on query optimization. Deep proficiency with GCP services: BigQuery, Cloud Functions, Cloud Run, Pub/Sub, Dataflow, and GCS. Proven experience with CI/CD pipelines and DevOps practices. Experience with infrastructure-as-code, specifically Terraform. Strong leadership and stakeholder management skills.

One address, no account. We’ll tell you when matching roles go live.

More at IIT JOBS INC

Related open roles

View all roles