Senior / Staff Full-Stack Data Engineer

As a Senior / Staff Full-Stack Data Engineer specializing in Databricks at Codvo, you will play a crucial role in designing, building, and maintaining scalable data and machine learning pipelines. Your primary focus will be on operationalizing analytics and ML workloads with a strong emphasis on reliability, performance, and cost efficiency. Here is what your role entails: - Design, build, and maintain ETL/ELT pipelines on Databricks using Spark, Delta Lake, and Databricks Workflows. - Build and operate batch and real-time data pipelines for ingestion, transformation, and orchestration. - Operationalize machine learning inference pipelines developed by data scientists for both batch and real-time scenarios. - Ensure consistency between model training and inference environments. - Implement data quality checks, validation rules, monitoring, alerting, and automated recovery. - Collaborate with data scientists to productionize models and optimize inference performance and cost. - Implement CI/CD, DevOps, and MLOps best practices for data pipelines and ML workflows. - Optimize compute, storage, and job configurations to enhance performance and cost efficiency. - Implement and manage enterprise data governance using Unity Catalog for schemas, lineage, ownership, and documentation. - Work closely with Databricks infrastructure and platform configurations. Your skills and experience should include: - Strong hands-on experience with Databricks, Apache Spark, and Delta Lake. - Proven experience in building and operating production-grade data pipelines. - Experience in operationalizing machine learning models and inference pipelines. - Strong understanding of data reliability, observability, and monitoring practices. - Experience with CI/CD, DevOps, and MLOps workflows. - Experience working with cloud platforms such as AWS or Azure. - Familiarity with Unity Catalog and enterprise data governance concepts. - Experience with spec-driven development and coding agents. Nice to have: - Experience with Databricks infrastructure tuning and cost optimization. - Exposure to streaming frameworks and real-time data processing. - Experience with Infrastructure-as-Code tools like Terraform or similar. Success in this role will be defined by: - Reliable, scalable, and cost-efficient Databricks data and ML pipelines. - Smooth productionization of ML models with strong collaboration across teams. - High data quality, observability, and platform stability. - Well-governed data assets with clear ownership and lineage. As a Senior / Staff Full-Stack Data Engineer specializing in Databricks at Codvo, you will play a crucial role in designing, building, and maintaining scalable data and machine learning pipelines. Your primary focus will be on operationalizing analytics and ML workloads with a strong emphasis on reliability, performance, and cost efficiency. Here is what your role entails: - Design, build, and maintain ETL/ELT pipelines on Databricks using Spark, Delta Lake, and Databricks Workflows. - Build and operate batch and real-time data pipelines for ingestion, transformation, and orchestration. - Operationalize machine learning inference pipelines developed by data scientists for both batch and real-time scenarios. - Ensure consistency between model training and inference environments. - Implement data quality checks, validation rules, monitoring, alerting, and automated recovery. - Collaborate with data scientists to productionize models and optimize inference performance and cost. - Implement CI/CD, DevOps, and MLOps best practices for data pipelines and ML workflows. - Optimize compute, storage, and job configurations to enhance performance and cost efficiency. - Implement and manage enterprise data governance using Unity Catalog for schemas, lineage, ownership, and documentation. - Work closely with Databricks infrastructure and platform configurations. Your skills and experience should include: - Strong hands-on experience with Databricks, Apache Spark, and Delta Lake. - Proven experience in building and operating production-grade data pipelines. - Experience in operationalizing machine learning models and inference pipelines. - Strong understanding of data reliability, observability, and monitoring practices. - Experience with CI/CD, DevOps, and MLOps workflows. - Experience working with cloud platforms such as AWS or Azure. - Familiarity with Unity Catalog and enterprise data governance concepts. - Experience with spec-driven development and coding agents. Nice to have: - Experience with Databricks infrastructure tuning and cost optimization. - Exposure to streaming frameworks and real-time data processing. - Experience with Infrastructure-as-Code tools like Terraform or similar. Success in this role will be defined by: - Reliable, scalable, and cost-efficient Databricks data and ML pipelines. - Smooth productionization of ML models with strong collaboration across teams. - High data quality, observability, and pl