Data Engineer

Job Title: Data Engineer Location: Pune Job Type: Full-Time ( WFO ) About TVARIT TVARIT GmbH specializes in developing and delivering cutting-edge artificial intelligence (AI) solutions for the metal industry, including steel, aluminum, copper, cast iron, and more. Our software products empower customers to make intelligent, data-driven decisions, driving advancements in Predictive Quality (PsQ) , Predictive Maintenance (PdM) , and Energy Consumption Reduction (PsE) , etc. With a strong portfolio of renowned reference customers, state-of-the-art technology, a talented research team from prestigious universities, and recognition through esteemed awards such as the EU Horizon 2020 AI Prize , TVARIT is recognized as one of the most innovative AI companies in Germany and Europe. We are seeking a self-motivated individual with a positive "can-do" attitude and excellent oral and written communication skills in English to join our team. Job Description We are looking for a Data Engineer with strong expertise in Azure Databricks , PySpark , and distributed computing to develop and optimize scalable ETL pipelines for manufacturing analytics. The role involves working with high-frequency industrial data to enable real-time and batch data processing . Key Responsibilities Build scalable real-time and batch processing workflows using Azure Databricks , PySpark , and Apache Spark . Perform data pre-processing, including cleaning, transformation, deduplication, normalization, encoding, and scaling to ensure high-quality input for downstream analytics. Design and maintain cloud-based data architectures , including data lakes , lakehouses , and warehouses , following Medallion Architecture . Deploy and optimize data solutions on Azure (preferred) , AWS , or GCP , with a focus on performance, security, and scalability . Develop and optimize ETL/ELT pipelines for structured and unstructured data from IoT, MES, SCADA, LIMS, and ERP systems . Automate data workflows using CI/CD and DevOps best practices , ensuring security and compliance with industry standards. Monitor, troubleshoot, and enhance data pipelines for high availability and reliability . Utilize Docker and Kubernetes for scalable data processing. Collaborate with the automation team, data scientists, and engineers to provide clean, structured data for AI/ML models. Desired Skills and Qualifications Bachelor’s or Master’s degree in Computer Science , Information Technology , or a related field. Minimum 2 years of experience in data engineering , with a strong focus on cloud platforms such as Azure (preferred) , AWS , or GCP . Proficiency in PySpark , Azure Databricks , Python , and Apache Spark . Expertise in relational databases (e.g., SQL Server, PostgreSQL), time-series databases (e.g., InfluxDB), and NoSQL databases (e.g., MongoDB, Cassandra). Experience in containerization (Docker, Kubernetes). Strong analytical and problem-solving skills with attention to detail. Good to have knowledge of MLOps , DevOps , and model lifecycle management . Excellent communication and collaboration skills, with a proven ability to work effectively as a team player. Comfortable working in a dynamic, fast-paced startup environment , adapting quickly to changing priorities and responsibilities.