Data Engineer - AI/ML

As a Data Engineer in this role, your responsibilities will include: - Designing and maintaining robust ETL/ELT pipelines for ML datasets. - Implementing real-time data streaming for inference using technologies like Kafka and Flink. - Implementing data lakes and warehouses optimized for AI workloads. - Implementing partitioning, indexing, and caching strategies for high-performance data retrieval. - Optimizing data storage and retrieval for AI workloads, including data lakes and warehouses. - Establishing data validation and lineage tracking to ensure integrity and compliance. - Applying best practices for handling sensitive data in AI contexts, such as GDPR and CCPA. In the Machine Learning Integration aspect, you will: - Collaborate on feature engineering and dataset preparation. - Implement automated data preprocessing for ML models, including normalization, encoding, and augmentation. - Collaborate with data scientists to create feature stores and reusable feature pipelines. - Deploy ML models using MLOps practices like CI/CD and model versioning. - Maintain version control for datasets, models, and pipelines. - Monitor model performance and automate retraining workflows. Qualifications required for this role include: - 5+ years of experience in data engineering or ML engineering roles. - Experience in building end-to-end ML pipelines. - Familiarity with vector databases such as Pinecone and Weaviate, and embedding techniques. - Exposure to generative AI and LLM-based applications. - Proficiency in programming languages like Python and SQL, as well as experience with Spark. - Familiarity with ML Frameworks like TensorFlow, PyTorch, and Scikit-learn. - Knowledge of data tools such as Airflow, dbt, and Kafka. - Experience with MLOps tools like MLflow, Kubeflow, and TensorFlow Serving. - Familiarity with cloud platforms like AWS, Azure, and GCP. Please note that this position may require evening and weekend work for time-sensitive project implementations. As a Data Engineer in this role, your responsibilities will include: - Designing and maintaining robust ETL/ELT pipelines for ML datasets. - Implementing real-time data streaming for inference using technologies like Kafka and Flink. - Implementing data lakes and warehouses optimized for AI workloads. - Implementing partitioning, indexing, and caching strategies for high-performance data retrieval. - Optimizing data storage and retrieval for AI workloads, including data lakes and warehouses. - Establishing data validation and lineage tracking to ensure integrity and compliance. - Applying best practices for handling sensitive data in AI contexts, such as GDPR and CCPA. In the Machine Learning Integration aspect, you will: - Collaborate on feature engineering and dataset preparation. - Implement automated data preprocessing for ML models, including normalization, encoding, and augmentation. - Collaborate with data scientists to create feature stores and reusable feature pipelines. - Deploy ML models using MLOps practices like CI/CD and model versioning. - Maintain version control for datasets, models, and pipelines. - Monitor model performance and automate retraining workflows. Qualifications required for this role include: - 5+ years of experience in data engineering or ML engineering roles. - Experience in building end-to-end ML pipelines. - Familiarity with vector databases such as Pinecone and Weaviate, and embedding techniques. - Exposure to generative AI and LLM-based applications. - Proficiency in programming languages like Python and SQL, as well as experience with Spark. - Familiarity with ML Frameworks like TensorFlow, PyTorch, and Scikit-learn. - Knowledge of data tools such as Airflow, dbt, and Kafka. - Experience with MLOps tools like MLflow, Kubeflow, and TensorFlow Serving. - Familiarity with cloud platforms like AWS, Azure, and GCP. Please note that this position may require evening and weekend work for time-sensitive project implementations.