Data Engineer

Data Engineer:

Designs, builds, and maintains the data infrastructure that supports the organization's data-related initiatives. Collaborates with cross-functional teams, including data scientists, analysts, and software engineers, to ensure the efficient and reliable processing, storage, and retrieval of data. Develops scalable data pipelines, optimizes data workflows, and ensures the quality and integrity of the data.

What you will do

Designs scalable and efficient data pipelines to extract, transform, and load data from various sources into data warehouses or data lakes.

Develops data pipelines that enable efficient data storage, retrieval, and analysis.

Implements data validation and quality checks to identify and address data anomalies or errors.

Designs data warehousing and data lake solutions that facilitate data storage, retrieval, and analysis.

Required:

8+ years of experience with SQL, NoSQL

5+ years of experience with Python or a comparable scripting language

5+ years of experience with Data warehouses and infrastructure components

5+ years of experience with ETL/ELT and building high-volume data pipelines.

5+ years of experience with reporting/analytic tools

5+ years of experience with Query optimization, data structures, transformation, metadata, dependency, and workload management

5+ years of experience with Big Data and cloud architecture

5+ years of hands-on experience building modern data pipelines within a major cloud platform (GCP, AWS, Azure)

3-5 years of experience with deployment/scaling of apps on containerized environment

Preferred:

Experience in designing and building data engineering solutions in cloud environments (preferably GCP)

Experience with Git, CI/CD pipeline, and other DevOps principles/best practices

Experience with bash shell scripts, UNIX utilities & UNIX Commands

ML/AI Experience is a plus

Understanding of software development methodologies including waterfall and agile.

Ability to leverage multiple tools and programming languages to analyze and manipulate data sets from disparate data sources.

Knowledge of API development

Experience with complex systems and solving challenging analytical problems.

Strong collaboration and communication skills within and across teams

Knowledge of data visualization and reporting

Experience with schema design and dimensional data modeling

Google Professional Data Engineer Certification would be nice to have.

Knowledge of microservices and SOA

Experience designing, building, and maintaining data processing systems