Data Engineering - Aws

Role Overview: You will be the Technical Lead Microsoft Fabric Data Engineering, responsible for designing and implementing advanced data solutions utilizing Microsoft Fabric. Your expertise will be crucial in developing robust and scalable data pipelines to support actionable business intelligence and strategic transformation initiatives at HCLTech. Your role will involve ensuring data integrity, optimizing performance, and promoting a data-driven culture within the organization. Key Responsibilities: - Architect, develop, and implement scalable ETL processes using Microsoft Fabric to integrate data from various sources into cloud-based data warehouses and lakehouses. - Collaborate with data analysts, engineers, and business stakeholders to define requirements and translate them into effective data solutions. - Build, maintain, and optimize Power BI dashboards and reports to transform complex datasets into actionable insights for informed decision-making. - Ensure data quality, integrity, and accuracy throughout the ETL lifecycle by adhering to industry best practices for data management and governance. - Monitor, troubleshoot, and optimize ETL processes to maintain high system reliability by proactively identifying and resolving issues or performance bottlenecks. - Document ETL workflows, data mappings, and process modifications to facilitate transparent communication across technical and non-technical teams. - Implement advanced data transformation and ingestion patterns, including Change Data Capture (CDC), schema evolution, error handling, and robust data operations. - Uphold best practices in data security, privacy, and compliance across all data engineering activities. Qualifications Required: - Bachelors degree in Computer Science, Information Technology, Engineering, or related field. - 7+ years of hands-on experience in ETL development, with recent projects utilizing Microsoft Fabric. - Expertise in Microsoft Fabric Data Pipelines (activities, triggers, parameters) and Notebooks (PySpark/Spark SQL). - Deep working knowledge of Delta Lake, including schema evolution, MERGE/UPSERT, OPTIMIZE/VACUUM, partitioning, and checkpointing. - Strong proficiency in CDC ingestion patterns (watermarks, change tables, log-based ingestion). - Advanced SQL skills (window functions, CTEs) and experience with PySpark for data transformations and performance tuning. - Demonstrated experience in implementing data quality frameworks, advanced error handling, and observability. - Solid understanding of business intelligence concepts with practical experience in Power BI for data visualization and reporting. - Knowledge of data warehousing concepts, architectures, and modeling techniques. - Familiarity with major cloud platforms (Azure, AWS, Google Cloud) and their data storage and processing services. - Excellent analytical, troubleshooting, and problem-solving skills. - Strong communication skills to convey complex technical concepts to non-technical audiences. (Note: Additional details about the company were not provided in the job description) Role Overview: You will be the Technical Lead Microsoft Fabric Data Engineering, responsible for designing and implementing advanced data solutions utilizing Microsoft Fabric. Your expertise will be crucial in developing robust and scalable data pipelines to support actionable business intelligence and strategic transformation initiatives at HCLTech. Your role will involve ensuring data integrity, optimizing performance, and promoting a data-driven culture within the organization. Key Responsibilities: - Architect, develop, and implement scalable ETL processes using Microsoft Fabric to integrate data from various sources into cloud-based data warehouses and lakehouses. - Collaborate with data analysts, engineers, and business stakeholders to define requirements and translate them into effective data solutions. - Build, maintain, and optimize Power BI dashboards and reports to transform complex datasets into actionable insights for informed decision-making. - Ensure data quality, integrity, and accuracy throughout the ETL lifecycle by adhering to industry best practices for data management and governance. - Monitor, troubleshoot, and optimize ETL processes to maintain high system reliability by proactively identifying and resolving issues or performance bottlenecks. - Document ETL workflows, data mappings, and process modifications to facilitate transparent communication across technical and non-technical teams. - Implement advanced data transformation and ingestion patterns, including Change Data Capture (CDC), schema evolution, error handling, and robust data operations. - Uphold best practices in data security, privacy, and compliance across all data engineering activities. Qualifications Required: - Bachelors degree in Computer Science, Information Technology, Engineering, or related field. - 7+ years of hands-on experience in ETL development, with recent projects