Source description
About the role
JOB RESPONSIBILITY Mandatory Skills: 5+ years experience in Spark ecosystem, Python/Scala programming, MongoDB data loads, Snowflake and AWS platform (EMR, Glue, S3) 6+ years IT experience and good expertise in SDLC/Agile 6+ years experience in SQL, complex queries, and optimization Coaches developers on technical skills and problem-solving. Helps team members grow in their technical careers. Hands on experience in writing advanced SQL queries, familiarity with variety of databases. Experience in coding solutions using Python/Spark and performing performance tuning/optimization. Experience in building and optimizing Big-Data pipelines in Cloud. Experience in handling different file formats like JSON, ORC, Avro, Parquet, CSV. Hands on experience in data processing with NoSQL databases like MongoDB. Familiarity and understanding of jobs scheduling. Hands on experience on working with APIs to process data. Understanding of data streaming, such as Kafka services Desired Skills: 2+ years experience in Healthcare IT projects Certification on Snowflake (Snow PRO certification), and AWS (Cloud Practitioner/Solution Architect) Hands-on experience on Kafka streaming pipelines implementation. Responsibilities: Enforces coding standards, reviews code, and ensures maintainability. Coaches developers on technical skills and problem-solving. Helps team members grow in their technical careers. Develop ETL/ELT processes to ingest and transform data from various sources including MongoDB, APIs, Snowflake and flat files (JSON, ORC, Avro, Parquet, CSV). Implement data loading strategies into Snowflake and other data warehouses. Write and optimize complex SQL queries for data extraction, transformation, and analysis across multiple database platforms. Works with Product Managers, Architects, and other teams to align technical goals with business needs. Leverage AWS services to manage and orchestrate data workflows, ensuring high availability and scalability. Implementing the Job in AWS EMR, AWS Glue , AWS Lambda using pyspark Perform performance tuning on Spark jobs and SQL queries to ensure efficient data processing. Participate in Agile ceremonies and contribute to sprint planning, story grooming, and retrospectives. Collaborate with cross-functional teams including data scientists, analysts, and DevOps engineers to deliver data solutions. Ensure data accuracy, consistency, and integrity through validation and quality checks. Maintain documentation for data pipelines, schemas, and data flow processes. Work with Kafka or similar technologies to build and maintain real-time data streaming solutions. Develop and maintain data ingestion processes from external/internal APIs, ensuring secure and reliable data flow. QUALIFICATION Bachelors or Masters in Computer Science/ IT or equivalent. EXPERIENCE 8 - 13 years of experience in SQL, PySpark, Big-Data, Spark, Python, AWS, Snowflake, Agile methodologies. SKILLS AND COMPETENCIES Possess strong analytical and programing skills, attention to detail, good verbal, and written communication skills Strong Analytical skills in relating multiple data sets and identify issues, root cause and patterns
More at Legato