AWS Data Engineer

Job Description We are seeking an experienced AWS Data Engineer to join our team. In this role, you will be responsible for developing and managing AWS Glue ETL processes to convert data from AWS RDS (SQL Server and Oracle) into Parquet format in S3, enabling efficient querying and storage. You will also leverage AWS Glue Crawlers to catalog the data in S3 and make it easier for querying using Amazon Athena. Key Responsibilities: Develop and implement Glue ETL jobs to convert data from AWS RDS (SQL Server and Oracle) to Parquet format in S3. Execute Glue Crawlers to catalog data stored in S3, ensuring an organized data catalog for ease of querying. Create and manage an AWS Glue Data Catalog to facilitate structured and schema-aware querying of data. Write and optimize SQL queries in Athena for complex business use cases and efficient data analysis. Define and implement data lifecycle management strategies for data stored in S3. Utilize the advantages of the Parquet file format for optimized storage and querying. Organize and partition S3 data for improved performance and efficient querying. Manage data security using IAM policies, S3 bucket policies, and KMS encryption for securing S3 data. Ensure compliance with regulatory requirements such as GDPR for secure data handling. Optimize Glue ETL jobs for better performance using PySpark or Glue Studio. Required Skills and Qualifications: Strong experience in developing, debugging, and optimizing Glue ETL jobs using PySpark or Glue Studio. Expertise in connecting AWS Glue ETL jobs wit h AWS RDS (SQL Server and Oracle) for data extraction and transformation. Proficiency in creating and managing Glue Crawlers to catalog and organize S3 data. Deep understanding of S3 architecture and best practices for managing large datasets. Expertise in partitioning and organizing data within S3 for efficient querying. In-depth knowledge of the Parquet file format and its benefits for optimized storage and querying. Strong experience with Amazon Athena , including writing complex SQL queries and optimizing query performance. Ability to create views or transformations in Athena for business use cases. Experience in implementing IAM policies, S3 bucket policies, and KMS encryption to secure data in S3. Understanding of data protection and compliance regulations (e.g., GDPR ) and the ability to implement secure data handling practices. Preferred Qualifications: Familiarity with AWS data lifecycle management strategies and best practices. Job Category: Athena Glue Glue Crawlers Glue ETL Job Type: Full Time Job Location: Bangalore(Karnataka) Hyderabad(Telangana) Kochi Mumbai(Maharashtra) Trivandrum(Kerala) Country: India Experince: 6