Source description
About the role
Job DescriptionData Pipeline DevelopmentBuild and maintain ETL/ELT pipelines using PythonIngest data from multiple sources (APIs, databases, files, streaming systems)Optimize pipelines for performance and scalabilityClean, transform, and validate raw datasetsHandle structured and unstructured data Use frameworks like: PandasPySparkDaskDatabase & Data Warehousing SQL (PostgreSQL, MySQL, SQL Server)NoSQL (MongoDB, Cassandra)Design schemas and optimize queriesBuild data warehouses using:SnowflakeRedshiftBigQueryBig Data Technologies Apache Spark Hadoop Process large-scale datasets efficiently Workflow Orchestration Apache Airflow Cloud Platforms Work on cloud environments: AWS (S3, Glue, Lambda, EMR) Azure (Data Factory, Synapse) GCP (Dataflow, BigQuery) Data Quality & Monitoring Implement data validation checks Monitor pipeline failures and fix bugs Ensure data reliability and integrity Job DescriptionData Pipeline DevelopmentBuild and maintain ETL/ELT pipelines using PythonIngest data from multiple sources (APIs, databases, files, streaming systems)Optimize pipelines for performance and scalabilityClean, transform, and validate raw datasetsHandle structured and unstructured data Use frameworks like: PandasPySparkDaskDatabase & Data Warehousing SQL (PostgreSQL, MySQL, SQL Server)NoSQL (MongoDB, Cassandra)Design schemas and optimize queriesBuild data warehouses using:SnowflakeRedshiftBigQueryBig Data Technologies Apache Spark Hadoop Process large-scale datasets efficiently Workflow Orchestration Apache Airflow Cloud Platforms Work on cloud environments: AWS (S3, Glue, Lambda, EMR) Azure (Data Factory, Synapse) GCP (Dataflow, BigQuery) Data Quality & Monitoring Implement data validation checks Monitor pipeline failures and fix bugs Ensure data reliability and integrity
More at Cognizant