Lead Data Engineer

EPAM has presence across 40+ countries globally with 55,000 + professionals & numerous delivery centers, Key locations are North America, Eastern Europe, Central Europe, Western Europe, APAC, the Middle East & Development Centres in India (Hyderabad, Pune, & Bangalore). Exp: 5-14 years Location: Gurgaon/Pune/Hyderabad/Bengaluru/Chennai/Coimbatore Work Mode: Hybrid (2-3 days office in a week) Job Description: 5+ Years of in Big Data & Data related technology experience Expert level knowledge and experience in Apache Spark ( Advance Concepts i.e. RDDs, DataFrames, fault tolerance, Spark internals, Catalyst optimizer and AQE) Hands on programming with Python ( Advance Concepts i.e. comprehensions, decorators, generators including their use cases and benefits and data structures ) OR Scala Understands exception handling and its practical applications. Proficient in using Python or Scala for data engineering and automation. Experience with building stream-processing systems, using technologies such as Spark-Streaming Experience with integration of data from multiple data sources such as RDBMS (SQL Server, Oracle), ERP, Files Good understanding of SQL queries, joins, stored procedures, relational schemas Knowledge of ETL techniques and frameworks Performance tuning of Spark Jobs Experience with native Cloud data services Azure/AWS/GCP Ability to lead a team efficiently Experience with designing and implementing Big data solutions Practitioner of AGILE methodology What you'll do Create and maintain optimal cloud-based Operational Datasource (ODS) & data pipeline architectures capable of ingesting structured and unstructured data using both batch & kafka streaming methods Assemble large, complex data sets that meet functional and technical requirements. Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc. Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources. Develop data flows that can leverage both on premise and cloud architectures. Work with stakeholders including the business analytics teams and IS architecture teams to assist with data-related technical issues and support their data infrastructure needs. WE OFFER Opportunity to work on technical challenges that may impact across geographies Vast opportunities for self-development: online university, knowledge sharing opportunities globally, learning opportunities through external certifications Opportunity to share your ideas on international platforms Sponsored Tech Talks & Hackathons Possibility to relocate to any EPAM office for short and long-term projects Focused individual development Benefit package: • Health benefits, Medical Benefits• Retirement benefits• Paid time off• Flexible benefits Forums to explore beyond work passion (CSR, photography, painting, sports, etc. About EPAM: https://vimeo.com/729865941