Lead - Market Intelligence & Data Engineering

As an experienced professional leading Market Intelligence and Data Engineering initiatives, your role will involve designing scalable data collection systems, building reliable data pipelines, and delivering insights to support strategic business decisions. - Design and maintain scalable data collection systems to gather structured information from online marketplaces and public data sources. - Collect and monitor data like product listings, pricing, promotions, product rankings, reviews, and competitor activity. - Develop systems capable of operating across multiple e-commerce platforms and geographic markets. - Build real-time or near-real-time data pipelines for monitoring price changes, promotions, and marketplace trends. - Ensure reliable data collection while managing platform constraints like rate limits and regional restrictions. - Transform and normalize data from multiple sources into a unified and structured data model. - Develop systems to map products, brands, and categories across platforms. - Implement monitoring processes to maintain data quality, reliability, and freshness. - Enhance datasets with taxonomy, categorization, and relevant metadata. Provide data insights supporting teams in Marketing, Pricing, Inventory & Buying, and Market Expansion by analyzing competitive signals, search performance insights, category visibility, market prices, discounts, competitive positioning, demand signals, category trends, seasonal patterns, and identifying opportunities across regions and categories. - Collaborate with internal data and analytics teams to combine external market intelligence with operational data. - Develop data products and dashboards that support business decision-making. - Maintain reliable, production-grade pipelines for both batch analytics and real-time insights. Continuously improve and innovate by expanding data coverage across additional platforms and markets, exploring new data sources such as social insights or market signals, and applying AI/ML techniques to enhance data extraction, matching, and predictive insights. Required Skills: - 6 - 10 years of experience in data engineering, web crawling, or market data infrastructure. - Strong expertise in Python and SQL. - Experience with web crawling frameworks like Scrapy, Playwright, Puppeteer, or similar tools. - Experience building data pipelines using tools like Airflow, Dagster, or similar orchestration frameworks. - Experience working with real-time data processing tools such as Kafka, Flink, or similar platforms. - Experience deploying scalable systems on cloud infrastructure (AWS preferred). - Strong understanding of data quality, monitoring, and observability practices. Preferred Experience: - Experience working with e-commerce or marketplace data environments. - Knowledge of large-scale data collection and structured data systems. - Exposure to AI/ML techniques for data extraction and analysis. As an experienced professional leading Market Intelligence and Data Engineering initiatives, your role will involve designing scalable data collection systems, building reliable data pipelines, and delivering insights to support strategic business decisions. - Design and maintain scalable data collection systems to gather structured information from online marketplaces and public data sources. - Collect and monitor data like product listings, pricing, promotions, product rankings, reviews, and competitor activity. - Develop systems capable of operating across multiple e-commerce platforms and geographic markets. - Build real-time or near-real-time data pipelines for monitoring price changes, promotions, and marketplace trends. - Ensure reliable data collection while managing platform constraints like rate limits and regional restrictions. - Transform and normalize data from multiple sources into a unified and structured data model. - Develop systems to map products, brands, and categories across platforms. - Implement monitoring processes to maintain data quality, reliability, and freshness. - Enhance datasets with taxonomy, categorization, and relevant metadata. Provide data insights supporting teams in Marketing, Pricing, Inventory & Buying, and Market Expansion by analyzing competitive signals, search performance insights, category visibility, market prices, discounts, competitive positioning, demand signals, category trends, seasonal patterns, and identifying opportunities across regions and categories. - Collaborate with internal data and analytics teams to combine external market intelligence with operational data. - Develop data products and dashboards that support business decision-making. - Maintain reliable, production-grade pipelines for both batch analytics and real-time insights. Continuously improve and innovate by expanding data coverage across additional platforms and markets, exploring new data sources such as social insights or market signals, and applying AI/ML techniques to enhance data extracti