Solution architect Web Scraping & Data Engineering

Solution Architect Web Scraping & Data Engineering (RFP Role) Role Overview Senior Solution Architect responsible for designing and delivering large-scale web scraping solutions (70%) with supporting data engineering architecture (30%) . Acts as the technical authority from pre-sales to delivery . Key Responsibilities 1. Pre-Sales & Proposal (Mandatory) Co-author technical sections of RFP/RFI proposals Lead discovery calls & assess target-site feasibility (anti-bot, login, geo restrictions) Own effort estimation (build, infra, proxies, CAPTCHA, maintenance) Create architecture diagrams, SLAs, KPIs, assumptions Participate in client discussions and sign off on technical feasibility Maintain estimation frameworks and knowledge base 2. Scraping Architecture (Primary) Design scalable scraping systems (crawl extract parse store) Handle dynamic sites, CAPTCHA, rate limits, IP blocking Define anti-bot strategies and proxy setups Ensure data quality, validation, and schema evolution 3. Data Engineering (Secondary) Design pipelines for ingestion, transformation, and analytics Define data models (dimensional/Data Vault) Build ETL/ELT workflows with orchestration and SLA tracking Implement data quality, lineage, and observability 4. Delivery Leadership Create HLD, LLD, ADRs Guide teams, review code, and mentor engineers Lead PoCs for complex use cases 5. Compliance & Operations Ensure legal compliance (GDPR, etc.) and ethical scraping Define monitoring, alerting, and cost optimization strategies Establish SLAs (freshness, accuracy, completeness) Required Skills Web Scraping (Primary) Python (Scrapy, BeautifulSoup), Playwright/Selenium, Puppeteer Anti-bot strategies (Cloudflare, DataDome, CAPTCHA handling) Proxy management (residential/mobile/datacenter) Parsing (XPath, APIs, GraphQL) Distributed crawling & orchestration (Airflow, Kafka, etc.) Data Engineering (Secondary) ETL/ELT, Airflow/ADF/Glue, Spark Warehouses (Snowflake, BigQuery) & lakehouses (Delta, Iceberg) Data modelling, dbt, data quality tools Cloud (AWS/GCP/Azure), Docker, Kubernetes, CI/CD, observability Experience 10+ years overall; 5+ years web scraping; 3+ years data engineering Experience designing high-scale scraping systems Prior Solution Architect / Tech Lead experience Strong pre-sales & client-facing experience Deliverables Proposal solutions, estimation models, feasibility reports HLD, LLD, ADRs, PoCs Runbooks, monitoring playbooks, governance documents Knowledge transfer & documentation Nice to Have ML/LLM-based extraction API reverse engineering Domain experience (e-commerce, finance, travel, etc.) Evaluation Criteria Scraping expertise (primary focus) Data engineering capability Proposal/estimation experience Project scale & complexity Communication & commercial competitiveness