Lead Data Engineer

Role Overview: As a Data Engineer, you will be responsible for owning the architecture and maintenance of data pipelines within the company. Your main task will involve orchestrating complex workflows using Apache Airflow to ensure seamless data flow from various on-site sources into the central data warehouse. Your role is crucial in transforming raw construction data into actionable insights that drive project timelines and efficiency. Key Responsibilities: - Design, build, and maintain robust data pipelines using Apache Airflow. This includes writing and optimizing Directed Acyclic Graphs (DAGs) to automate Extract, Transform, Load (ETL) processes. - Develop and manage end-to-end ETL lifecycles, ensuring accurate extraction from APIs and databases, transformation for business logic, and loading into the data warehouse. - Maintain and enhance data architecture to support high-volume data ingestion and processing. - Implement monitoring and alerting for data pipelines to ensure high availability and accuracy. Promptly troubleshoot and resolve data issues. - Collaborate closely with construction tech team, data analysts, and software engineers to understand data requirements for Digital Twin and operational dashboards. Qualification Required: - Minimum of 5 years of professional experience in Data Engineering. - Strong hands-on experience with Apache Airflow, including writing custom operators, managing DAG dependencies, and troubleshooting scheduling issues. - Proven track record in building and optimizing ETL/ELT pipelines. - Strong proficiency in Python (for scripting and Airflow) and SQL (for database querying and transformation). - Experience with relational databases (PostgreSQL, MySQL) and Data Warehousing concepts. Additional Details: The company is looking for candidates with experience ranging from 5 to 8 years. The educational qualification required is B.Tech/B.E. The skills required for this role include AWS, Airflow, MySQL, and ETL. Nice-to-have skills include experience with the AWS ecosystem (S3, Redshift, Glue, or EMR), familiarity with containerization tools like Docker or Kubernetes, and knowledge of Real Estate/Construction domain data or IoT data streams. Role Overview: As a Data Engineer, you will be responsible for owning the architecture and maintenance of data pipelines within the company. Your main task will involve orchestrating complex workflows using Apache Airflow to ensure seamless data flow from various on-site sources into the central data warehouse. Your role is crucial in transforming raw construction data into actionable insights that drive project timelines and efficiency. Key Responsibilities: - Design, build, and maintain robust data pipelines using Apache Airflow. This includes writing and optimizing Directed Acyclic Graphs (DAGs) to automate Extract, Transform, Load (ETL) processes. - Develop and manage end-to-end ETL lifecycles, ensuring accurate extraction from APIs and databases, transformation for business logic, and loading into the data warehouse. - Maintain and enhance data architecture to support high-volume data ingestion and processing. - Implement monitoring and alerting for data pipelines to ensure high availability and accuracy. Promptly troubleshoot and resolve data issues. - Collaborate closely with construction tech team, data analysts, and software engineers to understand data requirements for Digital Twin and operational dashboards. Qualification Required: - Minimum of 5 years of professional experience in Data Engineering. - Strong hands-on experience with Apache Airflow, including writing custom operators, managing DAG dependencies, and troubleshooting scheduling issues. - Proven track record in building and optimizing ETL/ELT pipelines. - Strong proficiency in Python (for scripting and Airflow) and SQL (for database querying and transformation). - Experience with relational databases (PostgreSQL, MySQL) and Data Warehousing concepts. Additional Details: The company is looking for candidates with experience ranging from 5 to 8 years. The educational qualification required is B.Tech/B.E. The skills required for this role include AWS, Airflow, MySQL, and ETL. Nice-to-have skills include experience with the AWS ecosystem (S3, Redshift, Glue, or EMR), familiarity with containerization tools like Docker or Kubernetes, and knowledge of Real Estate/Construction domain data or IoT data streams.