ClickHouse Data Engineer

Required Skills1. ClickHouse- 2+ Years 2. Kubernetes - understanding infra. 3. AWS Additional Skills Job Description

· Data Pipeline and Storage: Detailed the process of collecting logs from various sources, structuring them through a pipeline using vectored dev for rate limiting and security, and storing them in ClickHouse as the database.

· Querying and Visualization: They plan to offer Grafana for standard querying but will support other tools like Databricks, Power BI, and Excel. An open-source tool called Keep will serve as the middle layer for event management and rules engine.

· Event Management: Keep will query the lake for triggering events in ServiceNow, acting as an intermediary due to ServiceNow's inability to handle millions of rules directly.

· Scale and Talent Needs: The platform aims to handle 100 petabytes a month initially, requiring sharp talent, especially in ClickHouse and vectored dev. They are seeking vendor support for architecture and operational expertise.

· Challenges and Solutions: The discussion also covered the challenges of scaling, the need for a multi-tenant solution, and the importance of optimizing SQL queries for efficiency. The goal is to consolidate various data sources into a single, queryable database to facilitate correlation and analysis.