Data Engineer

As a Data Engineer at Cybage Software Pvt. Ltd., you will play a crucial role in designing and implementing the Audience Database architecture. Your key responsibilities will include: - Integrating multiple data sources such as CRMs, APIs, and web tracking into the database. - Building and managing ETL/ELT pipelines using Airbyte for data ingestion and normalization into storage layers like PostgreSQL, S3, and ClickHouse. - Developing backend components and RESTful APIs using FastAPI to serve audience data for dashboards and campaign systems. - Integrating and optimizing Elasticsearch for indexing, search, and personalized audience filtering. - Applying data transformations, deduplication, and schema management using Python and SQL. - Ensuring high availability, performance, and reliability of the data infrastructure. - Collaborating closely with product and analytics teams to define data requirements and delivery mechanisms. - Leading and mentoring a team of data engineers to ensure project milestones are met. Your qualifications should include: - Experience in Data Engineering with strong Python programming skills. - Proven experience with Airbyte (Cloud or OSS) for connector-based data ingestion. - Hands-on experience building APIs using FastAPI or similar Python frameworks (Flask, Django REST). - Strong experience with Elasticsearch, including index design, query optimization, and integration. - Solid command of SQL and data modeling in PostgreSQL or other relational databases. - Familiarity with cloud storage and processing (e.g., AWS S3). - Experience with OAuth2, webhooks, and API-based integrations. - Ability to lead small technical teams and coordinate with stakeholders across functions. As a Data Engineer at Cybage Software Pvt. Ltd., you will play a crucial role in designing and implementing the Audience Database architecture. Your key responsibilities will include: - Integrating multiple data sources such as CRMs, APIs, and web tracking into the database. - Building and managing ETL/ELT pipelines using Airbyte for data ingestion and normalization into storage layers like PostgreSQL, S3, and ClickHouse. - Developing backend components and RESTful APIs using FastAPI to serve audience data for dashboards and campaign systems. - Integrating and optimizing Elasticsearch for indexing, search, and personalized audience filtering. - Applying data transformations, deduplication, and schema management using Python and SQL. - Ensuring high availability, performance, and reliability of the data infrastructure. - Collaborating closely with product and analytics teams to define data requirements and delivery mechanisms. - Leading and mentoring a team of data engineers to ensure project milestones are met. Your qualifications should include: - Experience in Data Engineering with strong Python programming skills. - Proven experience with Airbyte (Cloud or OSS) for connector-based data ingestion. - Hands-on experience building APIs using FastAPI or similar Python frameworks (Flask, Django REST). - Strong experience with Elasticsearch, including index design, query optimization, and integration. - Solid command of SQL and data modeling in PostgreSQL or other relational databases. - Familiarity with cloud storage and processing (e.g., AWS S3). - Experience with OAuth2, webhooks, and API-based integrations. - Ability to lead small technical teams and coordinate with stakeholders across functions.