Data Engineer

Role- Data Engineer Duration- 12 months + extension Loaction- 100% Remote Required Skills This candidate must possess skills in the Azure Data platform. This client is looking for a mid-level Data Engineer. They must be skilled in Azure Data Bricks, Data Factory, and Data Lake. This candidate should also have an exposure to Big Data. Key Responsibilities: Designs and automates deployment of our distributed system for ingesting and transforming data from various types of sources (relational, event-based, unstructured). Designs and implements framework to continuously monitor and troubleshoot data quality and data integrity issues. Implements data governance processes and methods for managing metadata, access, retention to data for internal and external users. Designs and provide guidance on building reliable, efficient, scalable and quality data pipelines with monitoring and alert mechanisms that combine a variety of sources using ETL/ELT tools or scripting languages. Designs and implements physical data models to define the database structure. Optimizing database performance through efficient indexing and table relationships. Participates in optimizing, testing, and troubleshooting of data pipelines. Designs, develops and operates large scale data storage and processing solutions using different distributed and cloud based platforms for storing data (e.g. Data Lakes, Hadoop, Hbase, Cassandra, MongoDB, Accumulo, DynamoDB, others). Uses innovative and modern tools, techniques and architectures to partially or completely automate the most-common, repeatable and tedious data preparation and integration tasks in order to minimize manual and error-prone processes and improve productivity. Assists with renovating the data management infrastructure to drive automation in data integration and management. Ensures the timeliness and success of critical analytics initiatives by using agile development technologies such as DevOps, Scrum, Kanban Coaches and develops less experienced team members Skillset: Hands-on Azure Data experience Tools to have: Databricks, Synapse, Data Factory, etc. High functional knowledge of Big Data Hands-on experience automating the ETL process No specific tool is required: Jenkins preferred Scala or PySpark knowledge (Scala preferred) -- ---- Thanks & Regards, Tejash - Technical Recruiter Thought wave Software and Solutions 314 N. Lake St, Suite 6, Aurora IL 60506 Desk: 6304918494 EXTN: 157 Email :tejash@thoughtwavesoft.com Website:www.thoughtwavesoft.com https://www.linkedin.com/in/tejash-chettipalli-342aa2246 A Certified Minority Business Enterprise, Disadvantaged Business Enterprise, SAM.gov, SOC2 & ISO2005, Vendors for: · STATES: IL, PA, TN, AK, OR, CT, GA, VA, ID, IA, UT, FL, MN & CO · NATIONAL LABS: ARGONNE & FERMI · COUNTIES: HENNEPIN, MN, FULTON & GA · PUBLIC SCHOOLS: ATLANTA. SECURITY NOTICE: This communication, including any accompanying document(s), is for the sole use of the intended recipient and may contain confidential information. Unauthorized use, distribution, disclosure or any action taken or omitted to be taken in reliance on this communication is prohibited, and may be unlawful. If you are not the intended recipient, please notify the sender (admin@thoughtwavesoft.com) by return e-mail or telephone (630-448-6681) and permanently delete or destroy all electronic and hard copies of this e-mail.