Data Engineer

Senior Data Engineer (AWS & Snowflake Specialist) Experience Level: 5–6 Years Role Overview This role is for a data powerhouse who can bridge the gap between high-performance big data processing on AWS and elite-level data warehousing in Snowflake. You will be responsible for the entire lifecycle of data, ensuring that every pipeline includes robust validation and observability. Key Responsibilities Big Data Processing: Leverage AWS EMR (Spark/Hive) for heavy-lift data processing and Athena for serverless ad-hoc querying of S3 data lakes. Data Validation: Implement Great Expectations (GX) or equivalent Python-based validation libraries to prevent data silent failure within EMR and Snowflake ingestion layers. Snowflake Engineering: Architect end-to-end Snowflake environments, utilizing SnowSQL for scripting and SnowPipe for continuous, automated data ingestion. System Integration: Build and maintain Snowflake Integrations (Storage Integrations with S3, API Integrations, and Notification Services). Advanced ETL: Develop robust pipelines that transition data from raw AWS storage into optimized Snowflake schemas. Technical Requirements Core Languages (Strong Proficiency Required): SQL: Expert mastery of Snowflake-specific SQL extensions, CTEs, and performance tuning (Micro-partitioning). Python: Advanced scripting for automation, Boto3 for AWS resource management, and data manipulation. Data Quality: Proven experience implementing automated data testing and validation frameworks within production pipelines. AWS Stack: Extensive experience with EMR, Redshift, Athena, S3, and IAM security policies. Snowflake Mastery: * Proven experience with Snowflake Data Engineering features (Streams, Tasks, and External Tables). Hands-on usage of SnowPipe for real-time or near-real-time data loading.