GCP Data Architect

As a Big Data Engineer, your role will involve working on the collecting, storing, processing, and analyzing of large sets of data. Your primary focus will be on selecting optimal solutions for these purposes, while also maintaining, implementing, and monitoring them. You will be responsible for integrating these solutions with the architecture used by our clients. Key Responsibilities: - Selecting and integrating Big Data tools and frameworks as needed to provide requested capabilities - Developing and maintaining data pipelines implementing ETL processes, monitoring performance, and advising on necessary infrastructure changes - Translating complex technical and functional requirements into detailed designs - Investigating and analyzing alternative solutions for data storing and processing to ensure streamlined approaches are implemented - Mentoring junior staff through technical training sessions and reviewing project outputs Qualifications Required: - Strong understanding of data warehousing and data modeling techniques - Proficient knowledge of distributed computing principles including Hadoop v2, MapReduce, and HDFS - Robust data engineering skills on GCP cloud platforms such as Airflow, Cloud Composer, Data Fusion, Data Flow, Data Proc, and Big Query - Experience with building stream-processing systems using solutions like Storm or Spark-Streaming - Good knowledge of Big Data querying tools like Pig, Hive, and Impala - Experience with Spark, SQL, and Linux - Familiarity with various ETL techniques and frameworks such as Flume, Apache NiFi, or DBT - Experience with messaging systems such as Kafka or RabbitMQ - Understanding of Lambda Architecture, including its advantages and drawbacks This job will provide you with the opportunity to work on challenging projects and collaborate with a team of experienced professionals. As a Big Data Engineer, your role will involve working on the collecting, storing, processing, and analyzing of large sets of data. Your primary focus will be on selecting optimal solutions for these purposes, while also maintaining, implementing, and monitoring them. You will be responsible for integrating these solutions with the architecture used by our clients. Key Responsibilities: - Selecting and integrating Big Data tools and frameworks as needed to provide requested capabilities - Developing and maintaining data pipelines implementing ETL processes, monitoring performance, and advising on necessary infrastructure changes - Translating complex technical and functional requirements into detailed designs - Investigating and analyzing alternative solutions for data storing and processing to ensure streamlined approaches are implemented - Mentoring junior staff through technical training sessions and reviewing project outputs Qualifications Required: - Strong understanding of data warehousing and data modeling techniques - Proficient knowledge of distributed computing principles including Hadoop v2, MapReduce, and HDFS - Robust data engineering skills on GCP cloud platforms such as Airflow, Cloud Composer, Data Fusion, Data Flow, Data Proc, and Big Query - Experience with building stream-processing systems using solutions like Storm or Spark-Streaming - Good knowledge of Big Data querying tools like Pig, Hive, and Impala - Experience with Spark, SQL, and Linux - Familiarity with various ETL techniques and frameworks such as Flume, Apache NiFi, or DBT - Experience with messaging systems such as Kafka or RabbitMQ - Understanding of Lambda Architecture, including its advantages and drawbacks This job will provide you with the opportunity to work on challenging projects and collaborate with a team of experienced professionals.