Sr Big Data Engineer

Role Overview: As a Sr. Big Data Engineer at our company, you will be a key player in the development and enhancement of the Enterprise TruRisk Platform, focusing on features that aid in effective management of cyber risks. You will work alongside a team of engineers and architects to prototype, design, develop, and support a highly scalable, distributed SaaS-based Security Risk Prioritization product. This role offers you the opportunity to contribute to Qualys' next-generation platform, utilizing Big Data and Micro-Services based technology to process billions of transactions per day and engage in challenging and impactful initiatives. Key Responsibilities: - Act as a thought leader in data platform and pipeline, particularly in Risk Evaluation. - Provide technical leadership to the engineering organization in designing, rolling out, and evolving data platforms. - Collaborate with product teams, professional services, and sales engineers to review solutions and trade-offs and represent engineering in related discussions. - Drive technology explorations and roadmaps to support the company's goals. - Lead technically demanding, cross-functional departments within the organization. - Ensure the quality of architecture and system design. - Break down complex problems into simple, practical solutions. - Possess expert knowledge in performance, scalability, enterprise system architecture, and engineering best practices. - Research and benchmark cloud technology to enhance the company's systems. - Mentor developers, assist in their career growth, and provide technical guidance. - Share knowledge with and train other team members. Qualifications: - Bachelors degree in computer science or equivalent. - Minimum of 6 years of total experience in relevant roles. - At least 2 years of experience in designing and architecting Big Data solutions using Spark. - 3+ years of experience working with engineering resources for innovation. - 4+ years of experience in understanding Big Data events flow pipeline. - In-depth expertise in various search solutions like solr/elastic. - Extensive experience with Kafka. - Proficiency in Data lakes and related ecosystems. - Strong understanding of messaging queues. - Ability to provide requirements for building scalable architectures for Big Data and Micro-services environments. - Familiarity with Presto technology and Airflow. - Hands-on experience in scripting and automation. - Proficiency in RDBMS/NoSQL, Oracle, Cassandra, Kafka, Redis, Hadoop, lambda architecture, kappa, kappa++ architectures with flink data streaming, and rule engines. - Experience in ML models engineering and deployment. - Design and implementation of secure big data clusters to meet compliance and regulatory requirements. - Experience leading the delivery of large-scale systems focused on managing the infrastructure layer. - Robust experience in performance benchmarking testing for Big Data technologies. - Strong troubleshooting skills. - Knowledge of Agile Management (SCRUM, RUP, XP), OO Modeling, and working on internet, UNIX, Middleware, and database-related projects. - Experience in mentoring/training the engineering community on complex technical issues. - Project management experience. Role Overview: As a Sr. Big Data Engineer at our company, you will be a key player in the development and enhancement of the Enterprise TruRisk Platform, focusing on features that aid in effective management of cyber risks. You will work alongside a team of engineers and architects to prototype, design, develop, and support a highly scalable, distributed SaaS-based Security Risk Prioritization product. This role offers you the opportunity to contribute to Qualys' next-generation platform, utilizing Big Data and Micro-Services based technology to process billions of transactions per day and engage in challenging and impactful initiatives. Key Responsibilities: - Act as a thought leader in data platform and pipeline, particularly in Risk Evaluation. - Provide technical leadership to the engineering organization in designing, rolling out, and evolving data platforms. - Collaborate with product teams, professional services, and sales engineers to review solutions and trade-offs and represent engineering in related discussions. - Drive technology explorations and roadmaps to support the company's goals. - Lead technically demanding, cross-functional departments within the organization. - Ensure the quality of architecture and system design. - Break down complex problems into simple, practical solutions. - Possess expert knowledge in performance, scalability, enterprise system architecture, and engineering best practices. - Research and benchmark cloud technology to enhance the company's systems. - Mentor developers, assist in their career growth, and provide technical guidance. - Share knowledge with and train other team members. Qualifications: - Bachelors degree in computer science or equivalent. - Minimum of 6 year