Data Engineer

Qualifications and Skills

• Strong knowledge on Python and Pyspark

• Expectation is to have ability to write Pyspark scripts for developing data workflows.

• Strong knowledge on SQL, Hadoop, Hive, Azure, Databricks and Greenplum

• Expectation is to write SQL to query metadata and tables from different data management system such as, Oracle, Hive, Databricks and Greenplum.

• Familiarity with big data technologies like Hadoop, Spark, and distributed computing frameworks.

• Expectation is to use Hue and run Hive SQL queries, schedule Apache Oozie jobs to automate the data workflows.

• Good working experience of communicating with the stakeholders and collaborate effectively with the business team for data testing.

• Expectation is to have strong problem-solving and troubleshooting skills.

• Expectation is to establish comprehensive data quality test cases, procedures and implement automated data validation processes.

• Degree in Data Science, Statistics, Computer Science or other related fields or an equivalent combination of education and experience.

• 5-7 years of experience in Data Engineer.

• Proficiency in programming languages commonly used in data engineering, such as Python, Pyspark, SQL.

• Experience in Azure cloud computing platform, such as developing ETL processes using Azure Data Factory, big data processing and analytics with Azure Databricks.

• Strong communication, problem solving and analytical skills with the ability to do time management and multi-tasking with attention to detail and accuracy.