Big Data Engineer

As an experienced Data Engineer, your responsibilities will include: - Designing, building, and maintaining scalable ETL/ELT pipelines using AWS Glue, Lambda, Step Functions, and Apache Spark. - Developing and optimizing data lake and data warehouse architectures using AWS S3, Redshift, and Athena. - Implementing data ingestion, transformation, and storage solutions using AWS Glue, Kinesis, and Kafka. - Working with structured and unstructured data to develop effective data pipelines. - Optimizing query performance in SQL-based and NoSQL databases (Redshift, DynamoDB, Aurora, PostgreSQL, etc.). - Ensuring data security, compliance, and governance using IAM roles, KMS encryption, and AWS Lake Formation. - Automating data workflows using Terraform, CloudFormation, or CDK. - Monitoring and troubleshooting data pipelines, ensuring high availability and performance. - Collaborating with Data Scientists, Analysts, and Engineers to provide robust data solutions. Qualifications required for this role: - Strong experience with AWS Data Services (Glue, Redshift, S3, Lambda, Kinesis, Athena). - Proficiency in Python, SQL, and Scala. - Experience with ETL/ELT pipeline design and data warehousing concepts. - Solid knowledge of relational and NoSQL databases (PostgreSQL, MySQL, DynamoDB). - Familiarity with DevOps practices and CI/CD tools (Jenkins, GitHub Actions). - Experience with Kafka or Kinesis for real-time data streaming. - Knowledge of Terraform, CloudFormation, or CDK for infrastructure automation. - Hands-on experience with data security and governance. - Robust analytical and problem-solving skills. As an experienced Data Engineer, your responsibilities will include: - Designing, building, and maintaining scalable ETL/ELT pipelines using AWS Glue, Lambda, Step Functions, and Apache Spark. - Developing and optimizing data lake and data warehouse architectures using AWS S3, Redshift, and Athena. - Implementing data ingestion, transformation, and storage solutions using AWS Glue, Kinesis, and Kafka. - Working with structured and unstructured data to develop effective data pipelines. - Optimizing query performance in SQL-based and NoSQL databases (Redshift, DynamoDB, Aurora, PostgreSQL, etc.). - Ensuring data security, compliance, and governance using IAM roles, KMS encryption, and AWS Lake Formation. - Automating data workflows using Terraform, CloudFormation, or CDK. - Monitoring and troubleshooting data pipelines, ensuring high availability and performance. - Collaborating with Data Scientists, Analysts, and Engineers to provide robust data solutions. Qualifications required for this role: - Strong experience with AWS Data Services (Glue, Redshift, S3, Lambda, Kinesis, Athena). - Proficiency in Python, SQL, and Scala. - Experience with ETL/ELT pipeline design and data warehousing concepts. - Solid knowledge of relational and NoSQL databases (PostgreSQL, MySQL, DynamoDB). - Familiarity with DevOps practices and CI/CD tools (Jenkins, GitHub Actions). - Experience with Kafka or Kinesis for real-time data streaming. - Knowledge of Terraform, CloudFormation, or CDK for infrastructure automation. - Hands-on experience with data security and governance. - Robust analytical and problem-solving skills.