Senior Data Engineer - Databricks

Own Databricks production support for the Sugar Predict data platform, including monitoring, alerting, and incident response across all production data flows

Maintain and report on SLA performance metrics for data pipeline delivery, ensuring visibility into platform health and accountability across internal and external stakeholders

Identify and implement pipeline optimizations that reduce Databricks compute costs, improve throughput, and reduce processing windows while tracking impact s through measurable KPIs

Migrate legacy ETL /ELT pipelines to Databricks, building automation tooling to reduce manual intervention and ensure uninterrupted data delivery during transitions

Support new customer s onboarding by provisioning, validating , and hardening tenant data pipelines that deliver reliable, isolated data from day one

Design and build high-performance Databricks pipelines that ingest, transform, and serve ERP and CRM data at scale across both Azure and AWS environments

Own the Delta Lake architecture including schema design, partitioning strategies, data quality enforcement, and incremental processing patterns

Enforce data security best practices across Databricks environments, including role-based access control, secrets management, and compliance requirements for enterprise CRM and ERP data

Implement data quality monitoring and observability across pipeline health and ML model inputs, ensuring data integrity that directly supports Sugar Predict prediction accuracy

Apply and enforce multi-tenant data isolation patterns ensuring reliable, secure data delivery across Sugar Predict enterprise customers

Partner with the Enterprise Architecture team to ensure Sugar Predict data pipelines integrate seamlessly with the broader SugarAI product ecosystem

Support a globally distributed operation through on-call rotation and after-hours incident response, meeting SLAs across multiple time zones

Maintain technical documentation, runbooks, and architectural decision records, contributing to team knowledge sharing and operational readiness across on-call and incident response scenarios

Apply CI/CD best practices to data pipeline development, including version control, automated testing, and deployment tooling to ensure reliable and repeatable pipeline delivery