StreamSets Developer/Engineer

StreamSets Developer/Engineer Designing, developing, and deploying data pipelines using StreamSets Configuring data sources, transformations, and destinations within pipelines. Monitoring and troubleshooting pipeline performance and errors. Collaborating with data architects and stakeholders to understand data requirements and ensure pipeline efficiency. Implementing best practices for data governance, security, and compliance. Basic understanding of Apache Kafka architecture, including topics, partitions, brokers, and consumer groups. Responsibilities: Pipeline Development: Designing and developing data pipelines using StreamSets based on business requirements. Implementing data transformations, validations, and enrichments within pipelines. Testing pipelines for performance, reliability, and data quality. Pipeline Monitoring and Maintenance: Monitoring pipeline performance metrics such as throughput, latency, and error rates. Troubleshooting and resolving issues related to pipeline failures or data inconsistencies. Optimizing pipelines for efficiency and scalability as data volumes grow. Security and Compliance: Ensuring that data handling practices within StreamSets comply with organizational security policies and regulatory requirements. Implementing encryption, access controls, and auditing mechanisms to protect sensitive data. Documentation and Knowledge Sharing: Documenting pipeline configurations, dependencies, and operational procedures. Sharing knowledge and best practices with team members through training sessions, documentation, and collaboration tools. Performance Optimization: Analyzing pipeline performance bottlenecks and optimizing configurations for improved efficiency. Scaling pipelines to handle increasing data volumes and processing requirements. Collaboration and Communication: Collaborating with cross-functional teams including developers, data scientists, and business analysts to understand data requirements and ensure alignment with business objectives. Communicating updates, issues, and resolutions effectively with stakeholders. Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.