Source description
About the role
Key Responsibilities -Design and implement scalable data platforms leveraging Data Lake, Lakehouse, and Data Mesh architectures -Build and optimize data pipelines for batch and real-time processing using tools like Databricks, Spark, DBT, and cloud-native services· -Develop robust data ingestion frameworks for structured, semi-structured, and unstructured data (APIs, files, streaming sources) -Design and develop scalable Python-based micro services to enable secure and efficient data sharing across systems and applications- Build RESTful APIs and/or event-driven services for exposing curated datasets from Data Lake/Lakehouse platforms -Work extensively with Python, PySpark, and SQL for data transformation and processing -Implement streaming pipelines using Kafka / Kinesis and integrate with downstream analytics systems -Design and manage large-scale datasets using formats such as Parquet, JSON, CSV, and sensor/IoT data -Optimize data storage, partitioning, and query performance for high-volume analytical workloads· -Collaborate with cross-functional teams (Data Architects, Analysts, BI teams) to operationalize data lake solutions -Contribute to data modeling in Lakehouse environments (medallion architecture, dimensional modeling) -Ensure data quality, reliability, and observability across pipelines -Implement serverless data processing solutions where applicable Required Skills & Experience -4–6 years of experience in Data Engineering or related roles -Strong expertise in Python, PySpark, andSQL -Hands-on experience with Databricks, Delta Lake, or Snowflake -Experience with ETL / ELT frameworks and orchestration tools -Practical exposure to streaming technologies like Kafka or Kinesis -Strong understanding of batch processing frameworks such as Spark, AWS Glue, orDBT -Proficiency in handling structured, semi-structured, and unstructured data -Experience with modern data modeling techniques (Star Schema, Snowflake Schema, 3NF) -Good understanding of Data Lakehouse concepts and implementation patterns -Familiarity with serverless architectures (e.g., Python-based serverless pipelines)
More at Bebo Technologies