Senior Data Engineer, Data Innovation

As a Data Engineer at Shaw group, you will play a crucial role in developing Web Robots to crawl through the web and retrieve data in various formats. Your responsibilities will include scraping website data, building automated reports, and automating data pipelines. You will be a part of a dynamic team based in Hyderabad, Bengaluru, or Gurugram. Key Responsibilities: - Develop Web Robots to crawl through the web and retrieve data - Scrape website data into a structured format - Build automated and custom reports on the downloaded data - Automate end-to-end data pipelines - Understand data requirements and reverse-engineer websites - Develop web robots to automate data extraction - Build monitoring systems to ensure data integrity - Manage website dynamics and layout changes for clean downloads - Transform raw data into structured form using scraping and parsing systems - Store extracted data in recommended databases - Build scalable data extraction systems Qualifications Required: - 2 to 4 years of experience in website data extraction and scraping - Good knowledge of relational databases and writing complex SQL queries - Proficiency in Python for data operations - Expertise in Python frameworks like Requests, UrlLib2, Selenium, Beautiful Soup, and Scrapy - Understanding of HTML, CSS, XML, JSON, and JavaScript - Experience with debugging tools in Chrome - Academic background in data structures and algorithms - BCA/MCA/BS/MS degree with practical knowledge application Shaw group is a global investment and technology development firm with a significant presence in capital markets. We value diversity and do not discriminate in employment matters based on various factors. Interested candidates can apply through our website. We also encourage applicants with relevant experience looking to restart their careers after a break to apply for this position. The Firm offers excellent benefits, a casual working environment, and an attractive compensation package. For more information about our recruitment process and data processing, please visit our website. As a Data Engineer at Shaw group, you will play a crucial role in developing Web Robots to crawl through the web and retrieve data in various formats. Your responsibilities will include scraping website data, building automated reports, and automating data pipelines. You will be a part of a dynamic team based in Hyderabad, Bengaluru, or Gurugram. Key Responsibilities: - Develop Web Robots to crawl through the web and retrieve data - Scrape website data into a structured format - Build automated and custom reports on the downloaded data - Automate end-to-end data pipelines - Understand data requirements and reverse-engineer websites - Develop web robots to automate data extraction - Build monitoring systems to ensure data integrity - Manage website dynamics and layout changes for clean downloads - Transform raw data into structured form using scraping and parsing systems - Store extracted data in recommended databases - Build scalable data extraction systems Qualifications Required: - 2 to 4 years of experience in website data extraction and scraping - Good knowledge of relational databases and writing complex SQL queries - Proficiency in Python for data operations - Expertise in Python frameworks like Requests, UrlLib2, Selenium, Beautiful Soup, and Scrapy - Understanding of HTML, CSS, XML, JSON, and JavaScript - Experience with debugging tools in Chrome - Academic background in data structures and algorithms - BCA/MCA/BS/MS degree with practical knowledge application Shaw group is a global investment and technology development firm with a significant presence in capital markets. We value diversity and do not discriminate in employment matters based on various factors. Interested candidates can apply through our website. We also encourage applicants with relevant experience looking to restart their careers after a break to apply for this position. The Firm offers excellent benefits, a casual working environment, and an attractive compensation package. For more information about our recruitment process and data processing, please visit our website.