Senior Member, Data Engineer (Data Extraction)

As a Data Engineer at The D. E. Shaw group, you will be part of the product development teams located in Hyderabad, Bengaluru, and Gurugram. Your primary responsibility will be to develop Web Robots or Web Spiders that crawl through the web to retrieve data in various formats such as HTML, plain text, PDFs, Excel, and more, both structured and unstructured. You will also work on scraping website data into structured formats, creating automated and customized reports based on the downloaded data for business insights, and automating end-to-end data pipelines. Key Responsibilities: - Understand data requirements of the business group and reverse-engineer websites and data retrieval processes - Develop web robots to automate data extraction and build monitoring systems for data integrity - Manage website dynamics and layout changes for clean data downloads - Transform raw data into structured form using scraping and parsing systems - Store extracted data in recommended databases and build scalable data extraction systems - Automate data pipelines and provide operations support for high availability and zero data losses Qualifications Required: - 2 to 4 years of experience in website data extraction and scraping - Proficiency in relational databases, writing complex SQL queries, and ETL operations - Expertise in Python for data operations and familiarity with frameworks like Requests, UrlLib2, Selenium, Beautiful Soup, and Scrapy - Understanding of HTML, CSS, XML, JSON, and JavaScript - Proficiency in debugging tools in Chrome for reverse engineering website dynamics - Strong academic background in BCA/MCA/BS/MS with knowledge of data structures and algorithms - Problem-solving, analytical, and debugging skills The D. E. Shaw group is committed to a diverse and inclusive work environment and encourages candidates with relevant experience, including those looking to restart their careers, to apply for this position. To apply, visit the company's website. The firm offers attractive benefits, a friendly working atmosphere, and a competitive compensation package. For more details on the recruitment process and data handling, please refer to the company's official website. As a Data Engineer at The D. E. Shaw group, you will be part of the product development teams located in Hyderabad, Bengaluru, and Gurugram. Your primary responsibility will be to develop Web Robots or Web Spiders that crawl through the web to retrieve data in various formats such as HTML, plain text, PDFs, Excel, and more, both structured and unstructured. You will also work on scraping website data into structured formats, creating automated and customized reports based on the downloaded data for business insights, and automating end-to-end data pipelines. Key Responsibilities: - Understand data requirements of the business group and reverse-engineer websites and data retrieval processes - Develop web robots to automate data extraction and build monitoring systems for data integrity - Manage website dynamics and layout changes for clean data downloads - Transform raw data into structured form using scraping and parsing systems - Store extracted data in recommended databases and build scalable data extraction systems - Automate data pipelines and provide operations support for high availability and zero data losses Qualifications Required: - 2 to 4 years of experience in website data extraction and scraping - Proficiency in relational databases, writing complex SQL queries, and ETL operations - Expertise in Python for data operations and familiarity with frameworks like Requests, UrlLib2, Selenium, Beautiful Soup, and Scrapy - Understanding of HTML, CSS, XML, JSON, and JavaScript - Proficiency in debugging tools in Chrome for reverse engineering website dynamics - Strong academic background in BCA/MCA/BS/MS with knowledge of data structures and algorithms - Problem-solving, analytical, and debugging skills The D. E. Shaw group is committed to a diverse and inclusive work environment and encourages candidates with relevant experience, including those looking to restart their careers, to apply for this position. To apply, visit the company's website. The firm offers attractive benefits, a friendly working atmosphere, and a competitive compensation package. For more details on the recruitment process and data handling, please refer to the company's official website.