Data Engineer

Job Description: Role Overview: As a skilled Data Engineer, you will be responsible for designing and maintaining robust data pipelines, working with large-scale structured and semi-structured pharma datasets, and collaborating closely with Data Scientists and Python Developers to deliver clean, validated, production-grade data infrastructure. Key Responsibilities: - Pipeline Design & Development - Design and build scalable ETL/ELT pipelines for ingesting, transforming, and loading pharmaceutical data from multiple sources - Process indication-level market forecast data from Excel workbooks into structured JSON and DB-ready formats for frontend visualization - Build and maintain data extraction pipelines from clinical trial registries, conference abstracts, and drug databases - Implement multi-step validation workflows to ensure pipeline output accuracy and completeness before downstream handoff - Data Modelling & Architecture - Design normalized relational schemas in SQL Server to support pharmaceutical analytics use cases - Collaborate with Data Scientists on data preparation and feature engineering pipelines - Maintain data dictionaries, schema documentation, and lineage records for all active pipelines - Contribute to architecture decisions on data storage, processing strategies, and API integrations - Quality, Validation & Reliability - Own data quality across assigned pipelines, implement automated checks, alerting, and audit trails - Investigate and resolve silent data errors including missing fields, incorrect labels, zero-value nodes, and schema mismatches - Write thorough unit and integration tests for all pipeline components - Collaboration & Standards - Work within an agile delivery framework, participate in sprint planning, estimation, reviews, and retrospectives - Follow and contribute to team-wide GitHub standards: branching strategy, PR reviews, naming conventions, and documentation - Actively use AI coding tools (GitHub Copilot, Claude, etc.) as part of everyday development - Produce transparent technical documentation and handoff notes for every delivery Qualification Required: - Technical Skills - Tools & Platforms - Python (3+ years, production pipelines) - SQL Server (primary) - SQL complex queries, stored procedures, optimization - Git / GitHub (branching, PRs, code review) - ETL/ELT design and implementation - Pandas, SQLAlchemy, Pydantic - Data modeling and schema design - Excel / openpyxl for source data handling - REST API integration and data extraction - Docker or containerized environments (desirable) - Data validation and testing practices - Selenium / web scraping tools (desirable) - JSON / XML data processing - Airflow or equivalent orchestration (desirable) - Nice to Have - Experience working with pharmaceutical or life sciences data clinical trials, market forecasting, drug pipelines, or therapy area analytics - Familiarity with NLP pipelines or LLM-assisted data processing workflows - Knowledge of multi-agent AI architectures or experience building pipelines that feed AI/ML models - Experience building Streamlit or lightweight data exploration tools Note: No additional details about the company were provided in the job description. Job Description: Role Overview: As a skilled Data Engineer, you will be responsible for designing and maintaining robust data pipelines, working with large-scale structured and semi-structured pharma datasets, and collaborating closely with Data Scientists and Python Developers to deliver clean, validated, production-grade data infrastructure. Key Responsibilities: - Pipeline Design & Development - Design and build scalable ETL/ELT pipelines for ingesting, transforming, and loading pharmaceutical data from multiple sources - Process indication-level market forecast data from Excel workbooks into structured JSON and DB-ready formats for frontend visualization - Build and maintain data extraction pipelines from clinical trial registries, conference abstracts, and drug databases - Implement multi-step validation workflows to ensure pipeline output accuracy and completeness before downstream handoff - Data Modelling & Architecture - Design normalized relational schemas in SQL Server to support pharmaceutical analytics use cases - Collaborate with Data Scientists on data preparation and feature engineering pipelines - Maintain data dictionaries, schema documentation, and lineage records for all active pipelines - Contribute to architecture decisions on data storage, processing strategies, and API integrations - Quality, Validation & Reliability - Own data quality across assigned pipelines, implement automated checks, alerting, and audit trails - Investigate and resolve silent data errors including missing fields, incorrect labels, zero-value nodes, and sch