Sr. Data Engineer/Scientist

As a Python Ninja with 5-8 years of experience, you will be showcasing your expertise in a stimulating environment that focuses on building cutting-edge products and services. Your primary responsibilities will include: - Collaborating across an agile team to continuously design, iterate, and develop big data systems. - Supporting custom solutions offered to the product development by working with the engineering team. - Extracting, transforming, and loading data into internal databases and Hadoop. - Understanding data sets and bringing them together. - Filling the gap between development, engineering, and data ops. - Optimizing current and existing data pipelines for speed and reliability. - Creating, maintaining, and documenting scripts to support ongoing custom solutions. - Deploying new products and product improvements. - Understanding best practices, common coding patterns, and positive practices around storing, partitioning, warehousing, and indexing of data. - Documenting and managing multiple repositories of code. Mandatory Requirements: - Hands-on experience in data pipelining and ETL (Any one tools: Hadoop, BigQuery, RedShift, Athena) and in AirFlow. - Familiarity with pulling and pushing files from SFTP and AWS S3. - Familiarity with AWS Athena and Redshift. - Experience in SQL programming to query and transform data from relational databases. - Experience in reading data from Kafka topic (both live stream and offline). - Experience with any Cloud solutions including GCP / AWS / OCI / Azure. - Familiarities with Linux (and Linux work environment). - Excellent written and verbal communication skills. - Experience in PySpark and Data frames. - Experience with SQL and NoSQL databases (MySQL, Cassandra). Preferred Requirements: - Knowledge of REST APIs. - Ability to integrate (not necessary to publish). - From a product experience background. Qualities: - Excellent organizational skills, including attention to precise details. - Strong multitasking skills and the ability to work in a fast-paced environment. Eligibility Criteria: - 5 years of experience in database systems. - 5+ years of experience with Python to develop scripts. - Bachelor's in IT or related field. In addition to the responsibilities and qualifications, the job offers compensation as per industry standards or based on experience and last CTC, along with paid leave. The job location is Hyderabad, with work from office 5 days a week. Please note that the company's vision includes providing employment and career opportunities for millions of job-ready interns, freshers, and professionals in the Industry Academia Community (IAC) through their flagship event 'IAC VISION 2030'. By applying for this position, you confirm your membership in IAC or give consent to be added to the IAC platform as a member of Industry Academia Community. As a Python Ninja with 5-8 years of experience, you will be showcasing your expertise in a stimulating environment that focuses on building cutting-edge products and services. Your primary responsibilities will include: - Collaborating across an agile team to continuously design, iterate, and develop big data systems. - Supporting custom solutions offered to the product development by working with the engineering team. - Extracting, transforming, and loading data into internal databases and Hadoop. - Understanding data sets and bringing them together. - Filling the gap between development, engineering, and data ops. - Optimizing current and existing data pipelines for speed and reliability. - Creating, maintaining, and documenting scripts to support ongoing custom solutions. - Deploying new products and product improvements. - Understanding best practices, common coding patterns, and positive practices around storing, partitioning, warehousing, and indexing of data. - Documenting and managing multiple repositories of code. Mandatory Requirements: - Hands-on experience in data pipelining and ETL (Any one tools: Hadoop, BigQuery, RedShift, Athena) and in AirFlow. - Familiarity with pulling and pushing files from SFTP and AWS S3. - Familiarity with AWS Athena and Redshift. - Experience in SQL programming to query and transform data from relational databases. - Experience in reading data from Kafka topic (both live stream and offline). - Experience with any Cloud solutions including GCP / AWS / OCI / Azure. - Familiarities with Linux (and Linux work environment). - Excellent written and verbal communication skills. - Experience in PySpark and Data frames. - Experience with SQL and NoSQL databases (MySQL, Cassandra). Preferred Requirements: - Knowledge of REST APIs. - Ability to integrate (not necessary to publish). - From a product experience background. Qualities: - Excellent organizational skills, including attention to precise details. - Strong multitasking skills and the ability to work in a fast-paced environment. Eligibility Criteria: - 5 years of experience in database systems. - 5+ years of exp