Technical Architect - Data Platform Engineering

Experience: 12 15+ years Location: Pune (onsite) About the Role We are looking for a Principal Architect to lead the design and development of a Lakehouse solution end to end. This is an opportunity to architect and build a reusable data platform framework from the ground up. You will own the technical architecture, write code for the core framework, mentor a growing engineering team, and work closely with leadership on the overall technology direction. The ideal candidate is a hands-on architect with a proven track record of building complex frameworks and platforms using open-source big-data technologies such as Spark, Hadoop, and modern Lakehouse stacks (e.g., Iceberg, Trino, Databricks, etc). Key Responsibilities Architecture (40%) Own the end-to-end architecture of the Lakehouse platform and its reusable framework components Design a metadata-driven data engineering framework built on Spark SQL and PySpark Architect a pipeline engine supporting dynamic SQL generation, incremental loads, CDC, SCD implementations, and schema evolution Design Kubernetes-native deployment with multi-tenancy, resource isolation, scaling strategy, high availability, and disaster recovery Define API contracts, plugin/extension architecture, and integration patterns Drive and document key architecture decisions (ADRs) Hands-on Engineering (40%) Build core framework components: pipeline engine, dynamic SQL generation, data quality checks, error handling and retry mechanisms Implement observability across the platform Optimize Spark job performance (partitioning, shuffle tuning, memory management, query plans) Establish and implement CI/CD pipelines, automated testing (unit and integration), and release practices Embed data governance and security into the platform (access control, auditability, cataloging) Engineering Leadership (20%) Mentor and guide junior engineers; conduct design and code reviews Define coding standards, engineering best practices, and documentation norms Contribute to broader technical direction and team growth Continuously raise the bar on engineering quality and delivery practices Required Experience 12 15+ years of software engineering experience overall 8+ years in data platform engineering 5+ years as an Architect on Spark/Hadoop-based platforms Proven success building complex, reusable frameworks or platforms (not just applications) using open-source technologies Expert-level proficiency in Python, PySpark, Spark SQL, and Spark internals Deep knowledge of Apache Iceberg , distributed systems, Kubernetes, Docker , REST APIs, microservices, Linux, Git, and CI/CD Technical Stack Data Platform: Apache Spark, Apache Iceberg, Apache Ozone, Apache Ranger, Trino, Kafka, Airflow, Platform Engineering: Kubernetes-native deployments, Helm, Terraform, Prometheus, Grafana, containerization, multi-tenant architecture, storage management, HA/DR Framework Capabilities Youll Build: JSON-driven pipelines, metadata-driven ETL, dynamic SQL generation, incremental loads CDC, SCD handling, schema evolution, data quality, lineage, plugin architecture, unit testing, performance optimization What Were Looking For A hands-on architect who designs systems and writes the code that proves the design Strong architecture and design instincts with the judgment to keep frameworks simple and extensible Complex problem-solving ability and speed in picking up new technologies Experience with Agile and spec-driven development Clear communication able to explain architecture decisions to engineers and leadership alike Nice to Have Contributions to open-source data projects (Spark, Iceberg, Trino, etc.) Experience with Databricks or comparable managed Lakehouse platforms Prior experience taking a platform or framework from concept to production Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.