Data Engineer

As a Data Engineer at Capco, a global technology and management consulting firm, you will be working on engaging projects with the largest international and local banks, insurance companies, payment service providers, and other key players in the industry. Your innovative thinking, delivery excellence, and thought leadership will help clients transform their business, contributing to disruptive work that is changing energy and financial services. ### Role Overview: - Design, develop, and optimize large-scale data processing applications using Scala, Apache Spark, and Java. - Build and maintain high-performance data pipelines capable of processing 80-90 million records daily with a focus on scalability, reliability, and efficiency. - Develop and integrate Native APIs and data services to support business-critical applications and analytics platforms. - Collaborate with cross-functional teams to gather requirements and deliver robust data engineering solutions. - Optimize Spark jobs, data workflows, and distributed computing processes to ensure high throughput and low latency. - Implement best practices for data quality, monitoring, governance, and operational excellence. - Troubleshoot and resolve performance bottlenecks across data processing and ingestion pipelines. ### Key Responsibilities: - Strong hands-on experience in Scala, Apache Spark, and Java. - Proven experience building and supporting large-scale distributed data processing systems handling tens of millions of records daily. - Strong understanding of Spark architecture, performance tuning, partitioning, caching, and optimization techniques. - Experience with data modeling, ETL/ELT processes, and large-scale batch and streaming data pipelines. - Solid understanding of distributed systems, concurrency, and high-volume data processing. - Experience with SQL and relational/non-relational databases. - Experience working in enterprise-scale data environments within Financial Services, Payments, or FinTech domains. - Exposure to cloud platforms (AWS, Azure, or GCP) and containerized deployments. - Familiarity with Kafka, Airflow, Hadoop ecosystem, or similar big data technologies. - Experience with CI/CD pipelines, DevOps practices, and Agile methodologies. ### Qualifications Required: - You should have 5+ years of experience as a Data Engineer. - Strong hands-on experience in Scala, Apache Spark, and Java. - Proven experience building and supporting large-scale distributed data processing systems. - Experience with SQL and relational/non-relational databases. - Familiarity with cloud platforms (AWS, Azure, or GCP) and containerized deployments. - Exposure to Kafka, Airflow, Hadoop ecosystem, or similar big data technologies. Capco, a Wipro company, has a tolerant, open culture that values diversity, inclusivity, and creativity. With no forced hierarchy, everyone has the opportunity to grow as the company grows, taking their career into their own hands. Capco believes that diversity of people and perspective gives a competitive advantage. As a Data Engineer at Capco, a global technology and management consulting firm, you will be working on engaging projects with the largest international and local banks, insurance companies, payment service providers, and other key players in the industry. Your innovative thinking, delivery excellence, and thought leadership will help clients transform their business, contributing to disruptive work that is changing energy and financial services. ### Role Overview: - Design, develop, and optimize large-scale data processing applications using Scala, Apache Spark, and Java. - Build and maintain high-performance data pipelines capable of processing 80-90 million records daily with a focus on scalability, reliability, and efficiency. - Develop and integrate Native APIs and data services to support business-critical applications and analytics platforms. - Collaborate with cross-functional teams to gather requirements and deliver robust data engineering solutions. - Optimize Spark jobs, data workflows, and distributed computing processes to ensure high throughput and low latency. - Implement best practices for data quality, monitoring, governance, and operational excellence. - Troubleshoot and resolve performance bottlenecks across data processing and ingestion pipelines. ### Key Responsibilities: - Strong hands-on experience in Scala, Apache Spark, and Java. - Proven experience building and supporting large-scale distributed data processing systems handling tens of millions of records daily. - Strong understanding of Spark architecture, performance tuning, partitioning, caching, and optimization techniques. - Experience with data modeling, ETL/ELT processes, and large-scale batch and streaming data pipelines. - Solid understanding of distributed systems, concurrency, and high-volume data processing. - Experience with SQL and relational/non-relational databases. - Experience working in enterprise-scale data enviro