Source description
About the role
Role:Senior Spark Architect/Engineer
Location: Remote (core business hours are 9am to 3pm cst) Duration: 6 month contract to start
Key Responsibilities Architect, develop, test, and deploy data lake/warehouse solutions using modern development tools in AWS. Design and implement robust data ingestion pipelines for high-volume datasets. Lead data migration and transformation initiatives with precision and efficiency. Develop and integrate software applications using industry-standard methodologies and architectural patterns, with a strong focus on performance and security. Collaborate with Business Analysts, Architects, and Senior Developers to define and establish the AWS application framework. Automate end-to-end ETL processes for diverse datasets within the big data ecosystem.
Required Skills & Experience Proficiency in SQL, Spark/PySpark, Spark Structured Streaming, Kafka, Linux/RHEL 8 , Python, Java, and REST APIs. Strong interpersonal skills with the ability to communicate complex technical concepts to both technical and non-technical stakeholders. Proven experience in leading, coaching, and mentoring technical teams.
Hands-on experience with: Spark SQL, Spark Structured Streaming, and Java Spark Framework for large-scale data processing. Machine learning capabilities via API Performance tuning and optimization of Spark jobs. Designing high-throughput, low-latency pipelines using Kafka. AWS EMR for distributed data processing. Experience with Kubernetes or like framework to deploy and manage Spark clusters and applications. Experience with Hybrid Cloud Architectures Delta Table Experience
Preferred Qualifications
- Familiarity with HL7 FHIR specifications is a plus. Experience with Serverless Architectures.
More at Thoughtwave Software and Solutions