Senior Real-time Data Engineer

As a Senior Real-time Data Engineer at our company, your primary mission will be to lead the architecture and development of our customer-facing analytics engine. You will play a crucial role in transforming raw data into actionable insights for our users, ensuring a seamless experience across all dashboards, reports, and APIs. Key Responsibilities: - Architecture & Scaling: Design and maintain a low-latency analytics stack capable of handling high-concurrency queries from thousands of concurrent SaaS users. - Real-time Ingestion: Build and optimize ingestion pipelines to move data from transactional databases (PostgreSQL) and event streams (Kafka/Kinesis) into Apache Pinot. - Semantic Layer Development: Utilize Cube (Cube.js) to model complex sales metrics and ensure consistency across the application. - Performance Engineering: Optimize Apache Pinot tables, indexing strategies, and Cube pre-aggregations to ensure fast dashboard widget loading times. - API Strategy: Expose data models via REST/GraphQL APIs in collaboration with Frontend Engineers to create top-notch data visualizations. - Data Governance: Implement multi-tenant security logic within the semantic layer to enforce strict data isolation between different customer accounts. Technical Requirements: - OLAP Expertise: 3+ years of experience with Apache Pinot or similar technologies like ClickHouse/StarRocks in a production environment. - Semantic Modeling: Deep experience with Cube (Cube.js) including advanced features like pre-aggregations and multi-tenant configurations. - Data Store Mastery: Expert-level knowledge of PostgreSQL, especially in analytical query optimization and Change Data Capture (CDC). - Streaming & Ingestion: Hands-on experience with real-time data movement tools like Debezium, Kafka, or Flink. - Software Craftsmanship: Proficiency in Node.js or Python with a focus on building scalable backend services. - Language Skills: Mastery of complex SQL and the capability to translate business logic into code-based data schemas. If you have experience in building analytics for CRM or Sales Tech ecosystems, contributed to open-source projects (specifically in the Pinot or Cube communities), or worked with Infrastructure as Code tools like Terraform and Kubernetes for managing data clusters, it would be considered a bonus. (Note: The additional details of the company were not explicitly mentioned in the provided job description.) As a Senior Real-time Data Engineer at our company, your primary mission will be to lead the architecture and development of our customer-facing analytics engine. You will play a crucial role in transforming raw data into actionable insights for our users, ensuring a seamless experience across all dashboards, reports, and APIs. Key Responsibilities: - Architecture & Scaling: Design and maintain a low-latency analytics stack capable of handling high-concurrency queries from thousands of concurrent SaaS users. - Real-time Ingestion: Build and optimize ingestion pipelines to move data from transactional databases (PostgreSQL) and event streams (Kafka/Kinesis) into Apache Pinot. - Semantic Layer Development: Utilize Cube (Cube.js) to model complex sales metrics and ensure consistency across the application. - Performance Engineering: Optimize Apache Pinot tables, indexing strategies, and Cube pre-aggregations to ensure fast dashboard widget loading times. - API Strategy: Expose data models via REST/GraphQL APIs in collaboration with Frontend Engineers to create top-notch data visualizations. - Data Governance: Implement multi-tenant security logic within the semantic layer to enforce strict data isolation between different customer accounts. Technical Requirements: - OLAP Expertise: 3+ years of experience with Apache Pinot or similar technologies like ClickHouse/StarRocks in a production environment. - Semantic Modeling: Deep experience with Cube (Cube.js) including advanced features like pre-aggregations and multi-tenant configurations. - Data Store Mastery: Expert-level knowledge of PostgreSQL, especially in analytical query optimization and Change Data Capture (CDC). - Streaming & Ingestion: Hands-on experience with real-time data movement tools like Debezium, Kafka, or Flink. - Software Craftsmanship: Proficiency in Node.js or Python with a focus on building scalable backend services. - Language Skills: Mastery of complex SQL and the capability to translate business logic into code-based data schemas. If you have experience in building analytics for CRM or Sales Tech ecosystems, contributed to open-source projects (specifically in the Pinot or Cube communities), or worked with Infrastructure as Code tools like Terraform and Kubernetes for managing data clusters, it would be considered a bonus. (Note: The additional details of the company were not explicitly mentioned in the provided job descri