Senior Data Engineer

Role Overview: Xenon7 is seeking a skilled and experienced Data Engineer with deep expertise in the Databricks ecosystem to join the data engineering team. As a Data Engineer, you will play a crucial role in building, optimizing, and maintaining scalable data pipelines on Databricks, leveraging Delta Lake, PySpark, and cloud-native services (AWS, Azure, or GCP). Collaboration with data scientists, analysts, and business stakeholders will be essential to ensure clean, high-quality, and governed data is available for analytics and machine learning use cases. Key Responsibilities: - Build, optimize, and maintain scalable data pipelines on Databricks - Utilize Delta Lake features such as ACID transactions, schema enforcement, time travel, and vacuuming - Collaborate with data scientists, analysts, and business stakeholders to ensure high-quality data for analytics and machine learning - Proficiency in PySpark and SQL for large-scale data processing - Experience with cloud platforms such as AWS (Glue, S3), Azure (Data Lake, ADF), or GCP (BigQuery, GCS) - Hands-on experience with Databricks Auto Loader, Structured Streaming, and job scheduling - Familiarity with Unity Catalog for multi-workspace governance and fine-grained data access - Experience integrating with orchestration tools (Airflow, ADF) and using infrastructure-as-code for deployment - Comfortable with version control and automation using Git, Databricks Repos, dbx, or Terraform - Experience in performance tuning, Z-Ordering, caching strategies, and partitioning best practices Qualifications Required: - 6+ years of experience as a Data Engineer, with at least 4 years hands-on experience with Databricks in production environments - Proficiency in PySpark and SQL for large-scale data processing - Deep understanding of Delta Lake features such as ACID transactions, schema enforcement, time travel, and vacuuming - Experience working with cloud platforms: AWS (Glue, S3), Azure (Data Lake, ADF), or GCP (BigQuery, GCS) - Hands-on experience with Databricks Auto Loader, Structured Streaming, and job scheduling - Familiarity with Unity Catalog for multi-workspace governance and fine-grained data access - Experience integrating with orchestration tools (Airflow, ADF) and using infrastructure-as-code for deployment - Comfortable with version control and automation using Git, Databricks Repos, dbx, or Terraform - Experience with performance tuning, Z-Ordering, caching strategies, and partitioning best practices Note: Additional details of the company were not provided in the job description. Role Overview: Xenon7 is seeking a skilled and experienced Data Engineer with deep expertise in the Databricks ecosystem to join the data engineering team. As a Data Engineer, you will play a crucial role in building, optimizing, and maintaining scalable data pipelines on Databricks, leveraging Delta Lake, PySpark, and cloud-native services (AWS, Azure, or GCP). Collaboration with data scientists, analysts, and business stakeholders will be essential to ensure clean, high-quality, and governed data is available for analytics and machine learning use cases. Key Responsibilities: - Build, optimize, and maintain scalable data pipelines on Databricks - Utilize Delta Lake features such as ACID transactions, schema enforcement, time travel, and vacuuming - Collaborate with data scientists, analysts, and business stakeholders to ensure high-quality data for analytics and machine learning - Proficiency in PySpark and SQL for large-scale data processing - Experience with cloud platforms such as AWS (Glue, S3), Azure (Data Lake, ADF), or GCP (BigQuery, GCS) - Hands-on experience with Databricks Auto Loader, Structured Streaming, and job scheduling - Familiarity with Unity Catalog for multi-workspace governance and fine-grained data access - Experience integrating with orchestration tools (Airflow, ADF) and using infrastructure-as-code for deployment - Comfortable with version control and automation using Git, Databricks Repos, dbx, or Terraform - Experience in performance tuning, Z-Ordering, caching strategies, and partitioning best practices Qualifications Required: - 6+ years of experience as a Data Engineer, with at least 4 years hands-on experience with Databricks in production environments - Proficiency in PySpark and SQL for large-scale data processing - Deep understanding of Delta Lake features such as ACID transactions, schema enforcement, time travel, and vacuuming - Experience working with cloud platforms: AWS (Glue, S3), Azure (Data Lake, ADF), or GCP (BigQuery, GCS) - Hands-on experience with Databricks Auto Loader, Structured Streaming, and job scheduling - Familiarity with Unity Catalog for multi-workspace governance and fine-grained data access - Experience integrating with orchestration tools (Airflow, ADF) and using infrastructure-as-code for deployment - Comfortable with version control and automation using Git, Databricks Repos, dbx, or Terraform - Experience wi