Data Engineer

Roles & Responsibilities: Design, develop, implement, test, and maintain scalable and efficient data pipelines for large scale structured and unstructured datasets, including document, image, and event data used in GenAI and ML use cases. Collaborate closely with data scientists, AI/ML engineers, MLOps and Product Owners to understand data requirements and ensure data availability and quality. Build and optimize data architectures for both batch and real-time processing. Develop and maintain data warehouses and data lakes to store and manage large volumes of structured and unstructured data. Implement data validation and monitoring processes to ensure data integrity. Implement and manage vector databases (eg. pgVector, Pinecone, FAISS, etc) and embedding pipelines to support retrieval-augmented architectures. Support data sourcing and ingestion strategies, including API, data lakes, and message queues. Enforce data quality, lineage, observability, and governance standards for AI workloads Work with cross-functional IT and business teams in an Agile environment to deliver successful data solutions. Help foster a data-driven culture via information sharing, design for scalability, and operational efficiency. Stay updated with the latest trends and best practices in data engineering and big data technologies