DE Architect

Role description Job Description – Databricks Manager Experience Range: 9–12 Years Locations: Pune, Bangalore, Chennai, Gurgaon, Hyderabad, Kolkata Role Levels: Databricks Engineer / Associate Manager – Data Engineering / Manager – Data Engineering Role Overview We are looking for experienced Data Engineering professionals with strong expertise in Databricks , Apache Spark , and cloud-based data platforms . The role involves building scalable data pipelines, optimizing big-data workloads, implementing data governance, and driving best practices across Data Engineering teams. Key Responsibilities For Databricks Engineer (9–12 Years) Design, develop, and maintain ETL/ELT pipelines using Databricks (Python/Scala/Spark SQL). Work extensively on Delta Lake , including ACID transactions, schema evolution, time-travel, and performance optimizations. Build and optimize workflows using Databricks Workflows , DLT , and Unity Catalog . Configure and optimize Databricks clusters and Spark jobs for performance and cost efficiency. Work with cloud storage systems (S3/ADLS/GCS), IAM/RBAC, and networking/security components. Implement Medallion Architecture (Bronze Silver Gold). Lead small teams in delivering Databricks-based data solutions. Own end-to-end design of data pipelines and distributed systems. Implement DevOps/CI-CD practices using Git, Azure DevOps, Terraform, Databricks CLI. Ensure data governance compliance, security controls, and best engineering practices. For Manager – Data Engineering (10–12 Years) All responsibilities of Associate Manager plus : Drive architectural decision-making for large-scale data platforms on Databricks. Partner with business, product, analytics, and data science teams to align solutions with business goals. Lead multiple project teams, mentor engineers, and ensure delivery excellence. Define long-term data engineering roadmap, standards, and best practices. Core Technical Skills (All Roles) Databricks Platform Expertise : Workspace, notebooks, DLT, Workflows, Unity Catalog. Apache Spark : PySpark, Spark SQL, Scala/Java for performance tuning. Delta Lake : ACID transactions, schema enforcement, Z-ordering, optimization. Programming : Python and/or Scala; SQL for analytics and data validation. Cloud Platforms : AWS / Azure / GCP — storage, IAM, networking, security. ETL/Data Architecture : Batch & streaming pipelines, Medallion Architecture. Performance Optimization : Debugging data skew, memory issues, cluster tuning. DevOps & CI/CD : Azure DevOps, Git, Terraform, Databricks CLI. Security & Governance : Row-level/column-level security, encryption, access controls. Soft Skills & Leadership Competencies Strong analytical and problem-solving ability. Effective communication with stakeholders, analysts, and cross-functional teams. Ability to mentor juniors and enable team growth (Associate Manager/Manager). Strategic thinking with the ability to influence design and technology decisions. Ownership mindset with strong execution focus. Preferred Certifications Databricks Data Engineer Associate / Professional Databricks Lakehouse Fundamentals Cloud certifications: AWS/Azure/GCP (Associate or above)