Tech Lead : Web Crawling

We are looking for an ambitious individual who never gets tired of handling large pool of unstructured data sources. Data is the most important fuel for the products/solutions we build at Almug. Build tools and frameworks which can scrape data without much developer intervention. Our goal is to build 500K+ of stable data pipelines. This requires a high degree of automation. Must have a strong passion for coding Must be a self-starter Must take quality, security and performance seriously Ability to pair with other engineers and cross-team as needed Be excellent in a crisis driving the product team towards quick actions to positively impact clients. Running zero down time deployments Responsibilities Tech Requirements: 4-8 years of experience in Web Scraping or Crawling through Scrapy or other frameworks and related libraries (like beautifulsoup). Strong Python/Java Programming Capability Your role is to apply your knowledge set to fetch data from multiple online sources, cleanse it and build APIs on top of it. Good understanding of data structure and algorithms and how it impacts the efficiency of the system in real work. Sound Knowledge in bypassing Bot Detection Techniques Web RestFul APIs / Microservices Development Experience Think deeply about developing large scale scraping tools including data integrity, health and monitoring systems. Develop a deep understanding of our vast data sources on the web and know exactly how, when, and which data to scrape, parse and store. Work with SQL and NoSQL databases to store raw data. Develop frameworks for automating and maintaining constant flow of data from multiple sources. Good knowledge of distributed technologies, real-time systems of high throughput, low latency, and highly scalable systems. Work independently with little supervision to research and test innovative solutions.