Senior Data Engineer - Databricks

As a Senior Data Engineer specializing in Databricks, your role will involve designing, building, and optimizing scalable data pipelines. You will be responsible for developing and maintaining ETL/ELT workflows for both batch and streaming data. Working with structured and semi-structured data from various sources, you will optimize Spark jobs for performance, cost, and reliability. It will be your responsibility to implement data quality, validation, and monitoring frameworks. Collaboration with data scientists, analysts, and stakeholders is crucial to deliver analytics-ready datasets. Additionally, managing data models in data lakes and lakehouse architectures, ensuring data security, governance, and compliance best practices are also part of your responsibilities. Key Responsibilities: - Design, build, and optimize scalable data pipelines using Databricks (Spark, Delta Lake) - Develop and maintain ETL/ELT workflows for batch and streaming data - Work with structured and semi-structured data from multiple sources - Optimize Spark jobs for performance, cost, and reliability - Implement data quality, validation, and monitoring frameworks - Collaborate with data scientists, analysts, and stakeholders to deliver analytics-ready datasets - Manage data models in data lakes and lakehouse architectures - Ensure data security, governance, and compliance best practices Qualifications Required: - 5+ years of experience as a Data Engineer - Strong hands-on experience with Databricks & Apache Spark - Proficiency in Python and/or Scala, SQL - Experience with Delta Lake, Lakehouse architecture - Strong knowledge of ETL/ELT, data modeling, and performance tuning - Experience with cloud platforms (AWS / Azure / GCP) - Familiarity with streaming frameworks (Kafka, Structured Streaming) - Understanding of data governance, security, and access controls - Excellent problem-solving and communication skills As a Senior Data Engineer specializing in Databricks, your role will involve designing, building, and optimizing scalable data pipelines. You will be responsible for developing and maintaining ETL/ELT workflows for both batch and streaming data. Working with structured and semi-structured data from various sources, you will optimize Spark jobs for performance, cost, and reliability. It will be your responsibility to implement data quality, validation, and monitoring frameworks. Collaboration with data scientists, analysts, and stakeholders is crucial to deliver analytics-ready datasets. Additionally, managing data models in data lakes and lakehouse architectures, ensuring data security, governance, and compliance best practices are also part of your responsibilities. Key Responsibilities: - Design, build, and optimize scalable data pipelines using Databricks (Spark, Delta Lake) - Develop and maintain ETL/ELT workflows for batch and streaming data - Work with structured and semi-structured data from multiple sources - Optimize Spark jobs for performance, cost, and reliability - Implement data quality, validation, and monitoring frameworks - Collaborate with data scientists, analysts, and stakeholders to deliver analytics-ready datasets - Manage data models in data lakes and lakehouse architectures - Ensure data security, governance, and compliance best practices Qualifications Required: - 5+ years of experience as a Data Engineer - Strong hands-on experience with Databricks & Apache Spark - Proficiency in Python and/or Scala, SQL - Experience with Delta Lake, Lakehouse architecture - Strong knowledge of ETL/ELT, data modeling, and performance tuning - Experience with cloud platforms (AWS / Azure / GCP) - Familiarity with streaming frameworks (Kafka, Structured Streaming) - Understanding of data governance, security, and access controls - Excellent problem-solving and communication skills