Full Stack Data Engineer (DataBricks & ML)

Role Overview: As a Full Stack DE Expert at Codvo, you will be responsible for designing, building, and maintaining Databricks data pipelines for ingestion, transformation, and orchestration. You will collaborate closely with data scientists to operationalize machine learning models, ensure data reliability and quality, and optimize performance and cost across compute clusters, jobs, and storage layers. Additionally, you will implement best-practice DevOps/MLOps processes and manage the enterprise data catalog. Key Responsibilities: - Design, build, and maintain Databricks data pipelines (ETL/ELT) using Spark/Delta Lake/Databricks Workflows. - Operationalize machine learning models by building inference pipelines and ensuring consistency between training and inference environments. - Ensure data reliability, quality, and observability through robust validation, monitoring, alerting, and automated recovery mechanisms. - Collaborate with data scientists to productionize models, manage model deployment lifecycles, and optimize inference performance and cost. - Implement best-practice DevOps/MLOps processes such as CI/CD for pipelines, model versioning, environment promotion, and infrastructure-as-code. - Optimize performance and cost across compute clusters, jobs, and storage layers. - Implement and manage the enterprise data catalog, including schema design, table ownership, lineage, governance, and documentation. - Experience with some Databricks infrastructure. - Experience with building BI dashboards and visualization. - Experience with coding agents and best practices (spec-driven development, etc.). Qualifications Required: Must Have: - Databricks platform experience. - Python development for data processing and ETL pipelines. - Unity Catalog knowledge. - AWS data services (S3, IAM, VPC, potentially Glue/Lambda). - Data lake/lakehouse architecture patterns. - Dashboard building experience. Nice to Have: - RESTful API design and development (Flask, FastAPI, or similar). - Authentication/authorization patterns (OAuth, API keys, IAM roles). - Query optimization and performance tuning. - PySpark optimization experience. - ML/AI pipeline experience. - Databricks AI/BI. Role Overview: As a Full Stack DE Expert at Codvo, you will be responsible for designing, building, and maintaining Databricks data pipelines for ingestion, transformation, and orchestration. You will collaborate closely with data scientists to operationalize machine learning models, ensure data reliability and quality, and optimize performance and cost across compute clusters, jobs, and storage layers. Additionally, you will implement best-practice DevOps/MLOps processes and manage the enterprise data catalog. Key Responsibilities: - Design, build, and maintain Databricks data pipelines (ETL/ELT) using Spark/Delta Lake/Databricks Workflows. - Operationalize machine learning models by building inference pipelines and ensuring consistency between training and inference environments. - Ensure data reliability, quality, and observability through robust validation, monitoring, alerting, and automated recovery mechanisms. - Collaborate with data scientists to productionize models, manage model deployment lifecycles, and optimize inference performance and cost. - Implement best-practice DevOps/MLOps processes such as CI/CD for pipelines, model versioning, environment promotion, and infrastructure-as-code. - Optimize performance and cost across compute clusters, jobs, and storage layers. - Implement and manage the enterprise data catalog, including schema design, table ownership, lineage, governance, and documentation. - Experience with some Databricks infrastructure. - Experience with building BI dashboards and visualization. - Experience with coding agents and best practices (spec-driven development, etc.). Qualifications Required: Must Have: - Databricks platform experience. - Python development for data processing and ETL pipelines. - Unity Catalog knowledge. - AWS data services (S3, IAM, VPC, potentially Glue/Lambda). - Data lake/lakehouse architecture patterns. - Dashboard building experience. Nice to Have: - RESTful API design and development (Flask, FastAPI, or similar). - Authentication/authorization patterns (OAuth, API keys, IAM roles). - Query optimization and performance tuning. - PySpark optimization experience. - ML/AI pipeline experience. - Databricks AI/BI.