Source description
About the role
As a Senior Data Engineer at IDMC, your role will involve designing and developing data pipelines (ETL/ELT) and processes with a thorough understanding of star/snowflake schemas. You will be responsible for hands-on experience in PowerCenter Developer and IDMC Data Integration, as well as providing DW Production Support and troubleshooting data or pipeline issues. Your tasks will include developing complex SQL queries, SQL optimization, and ensuring full Life Cycle experience from business requirements gathering to deployment within Business Intelligence/Data Warehousing Architecture. Key Responsibilities: - Design and Development, maintenance, and enhancement of Data Pipelines (ETL/ELT) and processes - Hands-on experience in PowerCenter Developer and IDMC Data Integration - Experience in DW Production Support and troubleshooting data or pipeline issues - Developing complex SQL queries and SQL optimization - Full Life Cycle experience including business requirements gathering, data sourcing, testing/data reconciliation, and deployment within Business Intelligence/Data Warehousing Architecture - Designing and implementing data security - Monitoring and optimizing data storage and data processing - Use of Object function/object-oriented scripting languages including Scala, C++, Java, and Python - Working with cloud data warehouse and data lake platforms (e.g., Snowflake, Databricks, Redshift, BigQuery, MS Synapse, MS ADLS, Apache Hadoop, etc.) - Working knowledge of message queuing, stream processing, and highly scalable 'big data' data stores - Strong project management and organizational skills - Experience supporting and working with cross-functional teams in a dynamic environment - Understanding of data science concepts - Familiarity with AI/ML frameworks and libraries Qualifications Required: - Advanced SQL knowledge and experience working with relational databases - Experience with various databases and ETL tools - Familiarity with big data tools like Kafka, Spark, and Hadoop - Strong analytical skills related to working with unstructured datasets - Experience building and optimizing 'big data' data pipelines, architecture, and data sets - Experience performing root cause analysis on internal and external data and processes - Strong project management and organizational skills - Familiarity with project management best practices and agile software development methodologies In summary, as a Senior Data Engineer at IDMC, you will play a crucial role in designing, developing, and maintaining data pipelines and processes while ensuring data security and optimization. Your expertise in SQL, data warehouses, and big data tools will be essential in supporting cross-functional teams and driving value from large datasets. As a Senior Data Engineer at IDMC, your role will involve designing and developing data pipelines (ETL/ELT) and processes with a thorough understanding of star/snowflake schemas. You will be responsible for hands-on experience in PowerCenter Developer and IDMC Data Integration, as well as providing DW Production Support and troubleshooting data or pipeline issues. Your tasks will include developing complex SQL queries, SQL optimization, and ensuring full Life Cycle experience from business requirements gathering to deployment within Business Intelligence/Data Warehousing Architecture. Key Responsibilities: - Design and Development, maintenance, and enhancement of Data Pipelines (ETL/ELT) and processes - Hands-on experience in PowerCenter Developer and IDMC Data Integration - Experience in DW Production Support and troubleshooting data or pipeline issues - Developing complex SQL queries and SQL optimization - Full Life Cycle experience including business requirements gathering, data sourcing, testing/data reconciliation, and deployment within Business Intelligence/Data Warehousing Architecture - Designing and implementing data security - Monitoring and optimizing data storage and data processing - Use of Object function/object-oriented scripting languages including Scala, C++, Java, and Python - Working with cloud data warehouse and data lake platforms (e.g., Snowflake, Databricks, Redshift, BigQuery, MS Synapse, MS ADLS, Apache Hadoop, etc.) - Working knowledge of message queuing, stream processing, and highly scalable 'big data' data stores - Strong project management and organizational skills - Experience supporting and working with cross-functional teams in a dynamic environment - Understanding of data science concepts - Familiarity with AI/ML frameworks and libraries Qualifications Required: - Advanced SQL knowledge and experience working with relational databases - Experience with various databases and ETL tools - Familiarity with big data tools like Kafka, Spark, and Hadoop - Strong analytical skills related to working with unstructured datasets - Experience building and optimizing 'big data' data pipelines, architecture, and data sets - Experience performing root cause analysis on i
More at Excelencia