Data Engineer

Data Engineer Job Description: Position: AWS- Data Engineer Experience: 1-4 Years Location: Sector 43, Gurgaon, Haryana Job Insight: We are seeking a skilled and experienced Data Engineer to join our dynamic team. You will be responsible for Data Ingestion and Migration, ETL Job Scripting(Python, Pyspark, SQL, DBT), Data Modeling and structuring of our data lake infrastructure. You will collaborate with data architects, data scientists, and other stakeholders to ensure efficient data storage, retrieval, and processing capabilities. The ideal candidate will have 2 to 4 years of experience in data engineering with a focus on data lake technologies. Responsibilities: 1. Design and implement salable data lake and data replication/migration architectures using technologies such as Qlik Replicate, AWS DMS, fivetran, Glue, EMR,DBT, Airflow, AWS S3, AWS Redmine, Snowflake. 2. Develop and maintain data ingestion pipelines to efficiently collect and store structured and unstructured data from various sources. 3. Optimize data lake performance and reliability by implementing best practices in data partitioning, indexing, and compression. 4. Collaborate with data scientists and analysts to understand data requirements and implement data transformations and aggregations as needed. 5. Ensure data lake security and compliance with regulatory requirements by implementing access controls, encryption, and auditing mechanisms. 6. Monitor data lake performance and troubleshoot issues to ensure high availability and reliability. 7. Document data lake architecture, processes, and procedures for knowledge sharing and future reference. 8. Stay updated on emerging trends and technologies in data engineering and contribute to continuous improvement initiatives. Required Skills: 1. Bachelor's degree in Computer Science, Engineering, or a related field. 2. Minimum 1 year of experience in data engineering with a focus on data lake technologies. 3. Proficiency in programming languages such as Python, Pyspark, SQL, DBT. 4. Hands-on experience with data lake technologies and cloud data storage solutions like AWS S3, Glue, Athena. 5. Experience with data ETL tools and frameworks such as Apache NiFi, Apache Kafka, AWS Glue, AWS DMS, Qlik Replication, AWS Athena, AWS Redshift. 6. Familiarity with data governance, security, and compliance requirements. 7. Excellent problem-solving skills and ability to work effectively in a fast-paced environment. 8. Strong communication and collaboration skills to work effectively with cross-functional teams. Good to have skills: 9. Understanding of data modeling concepts and experience with schema design for structured and semi-structured data. 10. Experience with Agile process methodology 11. Exposure to Quicksight, Apache Superset will be a plus