Sharan II SUBCON Requirement/518472

ECMS ID# 518472 Number of Openings* 3 Duration of contract* 6 months Total Yrs. of Experience* 5 to 8 yrs Domain* Financial Services Detailed JD We’re looking for a Big Data Lead Engineer to: • engineer reliable data pipelines for sourcing, processing, distributing, and storing data in different ways, using cloud (Azure) data platform infrastructure effectively. • transform data into valuable insights that inform business decisions, making use of our internal data platforms and applying appropriate analytical techniques. • develop, train, and apply data engineering techniques to automate manual processes, and solve challenging business problems. • ensure the quality, security, reliability, and compliance of our solutions by applying our digital principles and implementing both functional and non-functional requirements. • build observability into our solutions, monitor production health, help to resolve incidents, and remediate the root cause of risks and issues. • understand, represent, and advocate for client needs. • Codify best practices, methodology and share knowledge with other engineers in UBS • have a continuous improvement mindset, who is always on the look out for ways to automate and reduce time to market for deliveries. Your expertise • Extensive experience in building Data Processing pipelines using Apache Spark/Databricks with Python and PySpark • Good Knowledge of inner working on Apache Spark. Structured streaming is a plus. • Deep understanding of Python and its ecosystem, principles and tooling that helps to write production grade applications e.g. PEP8, MyPy, PyLint, Pytest. • Good knowledge of data design patterns and methodologies to build a data lake, based Azure cloud stack e.g. ADLSv2. • Experience in creating data structures optimized for storage and various query patterns for DeltaLake, Parquet, Avro • Deep understanding of the SDLC using Gitlab, Github and knowledge of CI/CD is a plus. • Good to have working experience in cloud (Azure is preferrable). Knowledge of (Kafka or Event Hub) is plus • Datalakehouses using medallion architecture. Knowledge of DataMesh principles is a plus. • Ability to debug using tools Spark UI, Ganglia UI, expertise in Optimizing Spark Jobs • The ability to work across structured, semi-structured, and unstructured data, extracting information and identifying linkages across disparate datasets. • Experience of building applications using Polars, Pandas, Numpy is plus. • Experience of building microservices on Kubernetes is plus • Experience in traditional data warehousing concepts (Kimball Methodology, Star Schema, SCD2). • Experience in orchestration tools like Azure Databricks Workflow, Apache Airflow is plus. • Ability to clearly communicate complex solutions. • Strong problem solving and analytical skills. • Working experience in Agile methodologies (SCRUM) • A proven team player with strong leadership skills, who can work in a collaborative way across business units, teams and regions Mandatory skills Azure databricks, Azure data factory, Python, Pyspark BGCheck (Pre onboarding Or Post onboarding) Before Onboarding Any client prerequisite BGV Agency* FADV Is there any working in shifts from standard Daylight (to avoid confusions post onboarding) * Regular timings