Data Engineer

Location: Remote Exp: 5 to 8 years BDatabricks Data Engineer- We are seeking a skilled and experienced SQL to Databricks Migration Specialist to join our team. The primary responsibility of this role is to migrate SQL-based Crystal Reports to Databricks, ensuring seamless integration and optimal performance. The ideal candidate should have a strong background in SQL, Databricks, and Crystal Reports, along with excellent problem-solving skills and attention to detail. Responsibilities: As a SQL to Databricks Migration Specialist for Crystal Reports, your key responsibilities will include: 1. Analyzing Existing Crystal Reports: - Reviewing existing Crystal Reports that utilize SQL queries. - Identifying the underlying data sources and dependencies. - Assessing the complexity of each report and its compatibility with Databricks. 2. Designing Data Models on Databricks: - Collaborating with stakeholders (business analysts, developers) to understand reporting requirements. - Translating existing SQL queries into optimized SparkSQL or Pyspark code for execution on Databricks. - Designing efficient data models using Delta Lake or other appropriate storage formats. 3. Migrating Crystal Reports: - Developing migration strategies based on the analysis of existing reports. - Rewriting or modifying existing reports using Apache Spark APIs within the Databricks environment. - Ensuring data integrity during the migration process. 4. Performance Optimization: - Profiling and optimizing migrated reports for improved performance on Databricks clusters. - Implementing best practices for query optimization, caching mechanisms, partitioning techniques, etc. 5. Documentation and Training: - Documenting the migration process including steps taken, challenges faced, and solutions implemented. Providing training sessions or documentation for end-users on how to use migrated reports in Databricks. Qualifications: - 1. Bachelor's degree in Computer Science or related field (or equivalent work experience). – 2. Strong proficiency in SQL programming language. - In-depth knowledge of Apache Spark ecosystem including SparkSQL , Pyspark and Python programming languages. 3. Familiarity with cloud-based platforms such as Azure Databricks would be advantageous 4. Familiarity with Crystal Reports or any reporting tools would be advantageous 5. Excellent problem-solving skills with an ability to analyze complex systems efficiently 6. Strong communication skills with an ability to collaborate effectively across teams