Data Engineer

Title: Data Engineer

*Local to MO or IL

Description:

Datastage is our main ETL tool which is on IBM Cloud Pak (platform).

We will eventually shift to a different ETL tool but not at the very near future.

We are also using Fivetran as our data replication tool.

Familiarity with OOP and scripting tools will be helpful.

Familiarity with cloud-based technology is helpful.

Familiarity with concepts in logistics (transportation and warehousing) is a plus since this is DOT's business.

DB2 is our current database platform, but we will be switching to Snowflake soon.

Required

5+ years SQL, data warehousing/analytics, data engineering

Create/Maintain - machine learning/artificial intelligence data pipelines

designing scalable data models like handling slowly changing dimensions (SCDs) and performance tuning.

Cleanse/transform data

ETL/ELT tools (IBM Cloud Pak, DataStage, Fivetran, Talend, or Informatica)

Moving away from DataStage and I believe moving to IBM Cloud Pak)

Very important to have DataStage to maintain old jobs and help transition jobs to new platform

2+ years of Snowflake (Desired Snowflake Data Engineer Certification)

3+ years Python OO programming (also would be useful Java, or C#)

2+ years Azure (or similar Cloud with AWS, GCP)

Bachelor's degree in Business Analytics, Computer Science, Mathematics, Statistics, or related field

Desired

IBM Cloud, DB2, watsonX.ai, Supply C hain

As a Data Engineer, you will design, test, and maintain data pipelines for advanced analytics. You'll transform complex datasets, ensuring alignment with strategic goals, and collaborate with IT and business experts to deliver crucial insights in adherence to information best practices.

Responsibilities

• Automate data wrangling from a wide variety of sources (flat files, remote servers, databases, legacy systems, message queues, APIs and more)

• Identify opportunities to improve system and business process efficiencies

• Cleanse/transform data in preparation for all levels of analytical use

• Leverage many technologies and methodologies as required—includes scripting and programming languages

• Assist in the preparation, deployment, and maintenance of machine learning/artificial intelligence data pipelines

• Work with vendors and consultants as appropriate.

• Assist with designing scalable data models for real-world applications, including handling slowly changing dimensions (SCDs) and performance tuning.

• Applies lean thinking and tools to identify and eliminate waste in all areas of the position

• YOU MUST HAVE

• Bachelor's degree in applicable field

• 5+ years of experience in applied SQL development, data warehousing/analytics, and data engineering (or equivalent experience)

• Proven experience with ETL/ELT tools such as IBM Cloud Pak, DataStage, Fivetran, Talend, or Informatica

• 2+ years of experience working with Snowflake

• Hands-on experience with the full software development lifecycle (SDLC)

• 2+ years of experience in object-oriented programming using Python, Java, or C#.

• Strong interpersonal skills with professional verbal and written communication abilities

• YOU MAY ALSO HAVE

• Bachelor's degree in Business Analytics, Computer Science, Mathematics, Statistics, or related field

• Snowflake Data Engineer Certification

• Ability to work independently as well as within a team environment

• Effective project management and planning skills

Role Specifics

Travel – Occasional overnight travel 1-3 nights per quarter for company meetings. Must have ability to travel independently as needed, without restriction by all modes of transportation, including car, plane, or train.

Schedule – ability to work flexible hours; may include evening & weekend hours as needed