Databricks

Roles & Responsibilities: Develop and maintain data pipelines and workflows using PySpark on Databricks Write Python scripts for data processing, transformation, and integration with Spark Utilize PySpark and related libraries (e.g., pyspark-ai) to incorporate AI and machine learning functionalities into Spark workflows Perform data analysis, transformations, and aggregations on large datasets Deploy and optimize machine learning models within the Databricks environment Collaborate with data engineers, data scientists, and business teams to understand requirements and deliver solutions Use the Databricks platform to manage, monitor, and optimize Spark jobs and workflows Work with the English SDK for Apache Spark to facilitate natural language interactions with Spark DataFrames Ensure code quality, performance, and scalability of data pipelines and Spark jobs.