Sr. Big Data Engineer

Sr. Big Data Engineer

Position : Sr. Big Data Engineer

Job Duties :

Attending meetings with stakeholders to understand projects requirements and the need for development of applications or enhancements to existing applications. Architecting and Designing the data lake platform for both batch and real-time pipelines on Azure cloud using the Hadoop technologies and Azure services. Building the Data-lake framework, and defining templates and best-practices that would be used across various application. Deploying, monitoring and debugging jobs in production providing production support. Design extract, Transform, Load (ETL) processes that populate HIVE databases which often includes data from multiple sources. Design and develop Data ingestion products to migrate RDBMS data warehouse data into Enterprise Data Lake with frame works programmed using Shell scripts, Pyspark, SparkSQL and Hive Transformation. Manage and schedule the batch and stream jobs on a Hadoop Cluster using Oozie work flow. Work with different data file formats like Avro, Parquet, JSON, CSV and Xml.Performing defect coordination with both development and testing team. Support QA Analysts through validation of defect reporting and tracking.

Work Location :

Various unanticipated work locations throughout the United States; relocation may be required. Must be willing to relocate.

Minimum Requirements:

Education : Bachelor’s degree in Computer Science, Electronics Engineering, or closely related fields.

Experience:

None

Job Order No : 1052341

This position is eligible for a referral bonus through our employee referral program