Source description
About the role
Databricks Technical Architect Experience : 10- 20 yrs Location: PAN India Required skills: 10 + Years of deep expertise in Databricks Lakehouse, Delta Lake, Unity Catalog and • Strong proficiency in Python, PySpark, Scala SQLand experience with relevant libraries and frameworks (e.g., NumPy, pandas, Fast API). • Design, build, optimize, and operate scalable data pipelines and lakehouse architectures on Databricks, enabling analytics, reporting, and AI/ML use cases in an enterprise environment. Lakehouse Architecture: Design scalable and secure Databricks solutions using Delta Lake patterns like Medallion Architecture. Platform Modernization: Migrate legacy data systems to Databricks Lakehouse on cloud platforms. Domain-Oriented Design: Develop data models and enable cross-platform access with Data Mesh and Lakehouse Federation. Technology Roadmap: Define roadmaps and assess new Databricks and AI features for modernization. Spark Optimization: Optimize PySpark/Spark SQL jobs for better performance and cost efficiency. ETL/ELT Pipelines: Build robust streaming and batch pipelines with Databricks tools. CI/CD and DevOps: Use IaC and DevOps tools to automate deployments. Unity Catalog: Manage centralized governance, access control, and data lineage. Compliance and Security: Set data security standards to meet regulatory requirements. Data Visualization: Knowledge of data visualization tools and libraries like Matplotlib, Power BI or Tableau for presenting insights derived from data analysis.
More at Tata Consultancy Services