Azure Databricks Lead/ Developer

Location: KY and NJ

Consultant either has to be local or need to relocate to any of the locations that is KY or NJ

Mandatory skills: Python, Pyspark, Azure Databricks, ETL,

Key Responsibilities

Designing developing and maintaining scalable data pipelines and ETL processes using Azure Databricks Data Factory and other Azure services

Implementing and optimizing Spark jobs data transformations and workflows within Databricks

Developing and maintaining data models and data dictionaries

Developing and maintaining data quality checks governance policies and security procedures

Designing and creating ETL processes to supply data to various destinations including data warehouses

Integrating data from various sources into Azure Databricks

Working with other teams including data engineers data scientists and analysts to ensure data quality and consistency

Implementing effective monitoring processes to track performance and optimize workflows

Contributing to the design and implementation of data lakehouse solutions using Databricks

Skills and Qualifications

• Azure Databricks Proficiency Experience with Azure Databricks PySpark and Spark

• Programming Languages Proficiency in languages like Python SQL and Scala

• ETL and Data Warehousing Strong understanding of ETL processes data warehousing concepts and data modeling

• Cloud Platforms Experience with cloud platforms particularly Microsoft Azure

• Data Engineering Experience in data engineering data pipelines and data integration

• Data Governance and Security Knowledge of data governance policies and procedures

• Problemsolving and Debugging Excellent problemsolving and debugging skills

• Communication and Teamwork Strong communication and teamwork skills