Senior Data Engineer - GenAI & Unstructured Data Pipelines

Role Overview: You will be responsible for owning the end-to-end ML lifecycle, which includes data ingestion, feature engineering, training, evaluation, deployment, monitoring, retraining, and rollback. Additionally, you will design, build, and operate production-grade ML pipelines using Azure native services with solid CI/CD and automation practices. Your role will involve using Azure Machine Learning and MLflow for experiment tracking, model registry, and governed promotion across Dev/Test/Prod environments. You will also be designing and deploying Generative AI solutions using Azure OpenAI, embeddings, vector search, and RAG pipelines. Furthermore, you will build Agentic AI workflows with multi-step reasoning, tool usage, guardrails, observability, reliability, and cost control. Key Responsibilities: - Design, build, and operate production-grade ML pipelines using Azure native services - Use Azure Machine Learning and MLflow for experiment tracking, model registry, and governed promotion - Design and deploy Generative AI solutions using Azure OpenAI, embeddings, vector search, and RAG pipelines - Build Agentic AI workflows with multi-step reasoning, tool usage, guardrails, observability, reliability, and cost control - Build scalable data and feature pipelines using Azure Databricks for batch and streaming processing - Develop RAG pipelines including chunking, embedding, and retrieval workflows - Design and manage vector search systems such as Azure AI Search and Pinecone - Build batch and real-time data ingestion pipelines using Spark and Kafka Qualifications Required: - Solid experience in end-to-end ML lifecycle processes - Proficiency in using Azure Machine Learning and MLflow - Strong background in designing and deploying Generative AI solutions - Experience in building scalable data and feature pipelines using Azure Databricks - Knowledge of building batch and real-time data ingestion pipelines using Spark and Kafka Role Overview: You will be responsible for owning the end-to-end ML lifecycle, which includes data ingestion, feature engineering, training, evaluation, deployment, monitoring, retraining, and rollback. Additionally, you will design, build, and operate production-grade ML pipelines using Azure native services with solid CI/CD and automation practices. Your role will involve using Azure Machine Learning and MLflow for experiment tracking, model registry, and governed promotion across Dev/Test/Prod environments. You will also be designing and deploying Generative AI solutions using Azure OpenAI, embeddings, vector search, and RAG pipelines. Furthermore, you will build Agentic AI workflows with multi-step reasoning, tool usage, guardrails, observability, reliability, and cost control. Key Responsibilities: - Design, build, and operate production-grade ML pipelines using Azure native services - Use Azure Machine Learning and MLflow for experiment tracking, model registry, and governed promotion - Design and deploy Generative AI solutions using Azure OpenAI, embeddings, vector search, and RAG pipelines - Build Agentic AI workflows with multi-step reasoning, tool usage, guardrails, observability, reliability, and cost control - Build scalable data and feature pipelines using Azure Databricks for batch and streaming processing - Develop RAG pipelines including chunking, embedding, and retrieval workflows - Design and manage vector search systems such as Azure AI Search and Pinecone - Build batch and real-time data ingestion pipelines using Spark and Kafka Qualifications Required: - Solid experience in end-to-end ML lifecycle processes - Proficiency in using Azure Machine Learning and MLflow - Strong background in designing and deploying Generative AI solutions - Experience in building scalable data and feature pipelines using Azure Databricks - Knowledge of building batch and real-time data ingestion pipelines using Spark and Kafka