azure Data engineer

Job Title: Azure Data Engineer

Location: Chicago, IL - must do one day week onsite

Length: 6 months

This will require an onsite interview or require the candidate to pick up their laptop in person

Requirements/Top Skills Needed

• ADF

• Databricks

• Python

• SQL

• Retail experience would be helpful, but not required

• Job Title: Azure Data Engineer

• Job Summary: We are seeking a highly skilled Azure Data Engineer to design, build, and optimize our next-generation cloud data platform. You will build scalable data pipelines, manage enterprise data warehouses, and transform massive datasets using Azure Cloud, ADF, Databricks, Spark, and PySpark.

• Key Responsibilities

• · Pipeline Orchestration: Design and implement scalable ETL/ELT pipelines using Azure Data Factory (ADF).

• · Big Data Processing: Build high-performance, distributed data transformation workloads via Azure Databricks and Apache Spark/PySpark.

• · Data Warehousing: Architect, model, and maintain cloud-native Data Warehouses and Data Lakes (ADLS Gen2).

• · Architecture Implementation: Apply modern architectural patterns like the Medallion Architecture (Bronze, Silver, Gold layers) and Delta Lake formats.

• · Performance Tuning: Optimize Spark jobs, cluster configurations, and SQL queries to minimize latency and cloud compute costs.

• · Governance & Security: Implement security frameworks, data lineage, and user access policies using Unity Catalog.

• · CI/CD Automation: Deploy, monitor, and automate workflows using Azure DevOps and automated alert configurations.

• Required Technical Skills:

• · Cloud Platform: Microsoft Azure Ecosystem (ADLS Gen2, Azure SQL, Synapse Analytics, Key Vault, Storages, Azure Functions, Logic Apps).

• · Data Orchestration: Advanced proficiency in Azure Data Factory (ADF).

• · Compute & Analytics: Deep hands-on experience with Azure Databricks.

• · Big Data Frameworks: Strong mastery of Apache Spark and PySpark (including Spark SQL).

• · Languages: Expert level in Python and SQL.

• · Data Warehousing: Solid understanding of Data Modeling (Star/Snowflake schema) and warehousing concepts

---