DATA ENGINEER

Role:Data Engineer

Duration:12 months Location:Remote

Required Skills :

1. ClickHouse- 2+ Years
2. Kubernetes - understanding infra.
3. AWS Additional Skills :

Job Description :

· Data Pipeline and Storage: Detailed the process of collecting logs from various sources, structuring them through a pipeline using vectored dev for rate limiting and security, and storing them in ClickHouse as the database. ·Querying and Visualization: They plan to offer Grafana for standard querying but will support other tools like Databricks, Power BI, and Excel. An open-source tool called Keep will serve as the middle layer for event management and rules engine. ·Event Management: Keep will query the lake for triggering events in ServiceNow, acting as an intermediary due to ServiceNow's inability to handle millions of rules directly. ·Scale and Talent Needs: The platform aims to handle 100 petabytes a month initially, requiring sharp talent, especially in ClickHouse and vectored dev. They are seeking vendor support for architecture and operational expertise. ·Challenges and Solutions: The discussion also covered the challenges of scaling, the need for a multi-tenant solution, and the importance of optimizing SQL queries for efficiency. The goal is to consolidate various data sources into a single, queryable database to facilitate correlation and analysis.

Thanks & Regards, Ramyasree US IT RECRUITER Thought wave Software and Solutions 314 N. Lake St, Suite 6, Aurora IL 60506 Desk: 6302164696 Email:ramyasree@thoughtwavesoft.com LinkedIn: https://www.linkedin.com/in/ramya-sree-kolla-519839221/

A Certified Minority Business Enterprise, Disadvantaged Business Enterprise, SAM.gov, SOC2 & ISO2005, Vendors for: · STATES: IL, PA, TN, AK, OR, CT, GA, VA, ID, IA, UT, FL, MN & CO, · NATIONAL LABS: ARGONNE & FERMI, · COUNTIES: HENNEPIN, MN, FULTON & GA, · PUBLIC SCHOOLS: ATLANTA.