Data Engineer

Data Engineer – AI-Powered Marketing Personalization Platform We're seeking an experienced Data Engineer to help build and scale our next-generation AI-powered marketing personalization platform (V2.0) . You'll design and implement a robust multi-database infrastructure that enables real-time personalization, vector search, graph analytics, and large-scale data processing. This is a greenfield opportunity to architect data pipelines from the ground up using vector and graph databases and LLM-based systems . You'll play a key role in migrating our existing platform while creating a scalable foundation powering AI agents across thousands of marketing campaigns. Core Responsibilities Data Architecture & Infrastructure (40%) Design and implement multi-database systems (MongoDB, Redis, Milvus, Neo4j, BigQuery) Build scalable real-time pipelines and ETL/ELT workflows Implement data governance, quality, and high-throughput optimization Vector & Graph Systems (25%) Optimize Milvus collections for semantic search Design Neo4j schemas for customer journeys and relationships Develop hybrid search (vector + graph + text) with performance tuning ML Data Infrastructure (20%) Build data pipelines for LLM fine-tuning and GNN training Manage data versioning, lineage, and A/B testing systems Implement real-time feature computation for contextual models Analytics & Monitoring (15%) Create BigQuery schemas and real-time dashboards Implement observability (Prometheus, Grafana) and anomaly detection Optimize data costs across cloud environments Tech Stack Databases: MongoDB, Redis, Milvus, Neo4j, BigQuery Processing: Airflow/Prefect, Pandas/Polars, dbt, Spark ML Pipeline: vLLM, MLflow, Sentence Transformers, PyTorch, TensorFlow Cloud & Infra: GCP, Docker, Kubernetes, Terraform, GitHub Actions Languages: Python (3.10+), SQL, Bash Requirements Must-Have 5+ years of data engineering experience in production systems Advanced Python and SQL expertise Experience with 3+ database types (SQL, NoSQL, Vector, Graph) Proven ability to build high-scale data pipelines (>1M records/day) Strong data modeling, validation, and optimization skills Experience with cloud data warehouses (BigQuery, Redshift, or Snowflake) Preferred Experience with Milvus, Pinecone, or Weaviate Graph databases (Neo4j, Neptune) and embedding-based search ML/Feature store experience Background in marketing technology or CDPs Key Projects Phase 1 – Foundation: Migrate 10M+ vectors, implement MongoDB schemas, Neo4j models, and BigQuery warehouse Phase 2 – Optimization: Build data quality monitoring, caching (Redis), and Phase 3 – ML Infrastructure: Create LLM fine-tuning pipelines, GNN feature stores, and A/B testing systems Why Join Us Collaborate with ML engineers and data scientists on cutting-edge AI systems High ownership and real product impact Modern tools and a flexible environment Clear growth path to Senior, Lead, or Principal roles Shape the future of AI-driven marketing personalization