Source description
About the role
Software Developer Posted 6 months ago Location: Remote Job Schedule: Full Time (40 Hours/Week) Salary: $86,715.00/Annum Job duties: Design and enhance custom ETL frameworks for data migrations, transformations, and integrations using Snowflake, Databricks, and AWS S3 Develop dynamic tasks to connect with SFTP servers and copy extract files to and from AWS S3. Implement dynamic API configurations for REST API integration, supporting incremental data extraction, authentication, and multithreaded processing. Build functionalities to process encrypted data using GPG/PGP keys, ensuring secure handling of sensitive information during transfers. Design parallel query execution and implement features for error logging and metrics capture during Snowflake curation operations. Integrate the Great Expectations library into frameworks for robust data validation, including file size checks, data type enforcement, and custom rule processing. Design and implement capabilities to read and write data from and to Kafka, including support for schema transformations and real-time data streams. Automate the handling of large files (10GB+), including encryption, custom headers/footers, and multi-sheet output for enhanced usability using Boto3 streams. Develop multithreaded processes for archival tasks and data processing to improve performance and scalability. Create advanced watermarking features to support incremental and historical data processing, ensuring accurate data tracking. Build deployment validation scripts to synchronize configurations, validate cluster setups, and ensure environment compliance. Develop user-defined functions (UDFs) in Spark for tasks like geolocation extraction and dynamic parameter handling. Implement observability tools to monitor ELT jobs and address issues in regression and production environments. Design global variables and reusable configurations to standardize processes across multiple data products and pipelines. Minimum Education Requirement: This position requires minimum of Bachelor’s degree in computer science, computer information systems, information technology, or a combination of education and experience equating to the U.S. equivalent of a Bachelor’s degree in one of the aforementioned subjects. Apply Online Attach Resume * Submit
More at SDH Systems