Big Data Engineer (Cloudera, AWS)

Job Title: Big Data Engineer (Cloudera, AWS)

Location: Primarily Remote/ HYB (Monthly Onsite Meetings in Reston, VA and/or Washington, DC) Employment Type: Contract

GC/USC

8+ years of experience in Cloudera and AWS Cloud environments with strong expertise in Kafka, NiFi, HBase, Solr, MongoDB, automation scripting, and performance optimization for large-scale enterprise data systems.

Overview:

Big Data Engineer with strong Cloudera and AWS expertise to support a high-availability Production Support Environment . This role involves managing big data infrastructure, optimizing Cloudera components, and ensuring seamless data migrations and automation. Occasional weekend releases and late-night troubleshooting may be required.

Key Responsibilities:

Maintain and optimize Cloudera environments (CDP, CDSW, Hive, Spark, NiFi).

Perform critical data migrations from CDH to CDP .

Design and implement Big Data pipelines and automated data flows using Python/R and NiFi .

Automate project lifecycle processes and perform incremental upgrades.

Support analytics, ML, and data science use cases across infrastructure and cluster deployments.

Manage data governance, quality, testing, and documentation .

Use YARN for resource allocation and job scheduling.

Manage workflows via Oozie and Hue .

Enforce security policies across Hadoop clusters using Apache Ranger .

Configure and manage Cloudera Data Science Workbench via Cloudera Manager.

Troubleshoot Kerberos , TLS/SSL , and data science workload issues.

Provide end-to-end Big Data / Hadoop support throughout development and production.

Optimize performance, replication, and connectivity; implement monitoring and automation scripts.

Support and manage Kafka , NiFi , HBase , Solr , and Flume integrations.

Required Skills & Experience:

8+ years of experience supporting Cloudera applications in AWS Cloud environments.

Expertise with MongoDB , Kafka , NiFi , Flume , HBase , and Solr .

Strong knowledge of Cloud Disaster Recovery strategies.

Kafka Administration & Troubleshooting (brokers, Connect, Zookeeper, MQ integration).

Experience with Kafka Streams API , KStreams , and KTables .

NiFi Administration (flow management, registry server, controllers, service management).

Integration experience: NiFi to Kafka/HBase/Solr , Flume to Kafka/NiFi/MQ .

Strong skills in database management , query optimization , and automation scripting .

Hands-on experience supporting large-scale enterprise data environments .

Preferred Skills:

Experience with DB2 UDB , DB2 mainframe , or MongoDB .

Exposure to streaming technologies such as Kafka , Spark , and Kudu .

Familiarity with AI/ML workloads and GenAI testing environments.