Data Engineer - Python

Data Engineer

What You’ll Do

• • Data Engineering & Pipeline Development Design and maintain backend data ingestion and embedding pipelines. Work hands-on with JupyterHub to set up environments, clone repositories, and run complex workflows.

• • Graph Database & Knowledge Management Use Neo4j to build graph-based solutions that map knowledge articles and workflows. Contribute to schema design, data integration, and performance tuning.

• • Data Quality & Optimization Troubleshoot data quality issues, optimize Spark jobs, and manage partition sizes. Convert ECS direct connections to IO Meet using Spark and Python.

• • Authentication & Security Solve JWT authentication challenges, research Kerberos for secure cluster connectivity, and manage credentials for Oracle DB and API clients.

• • DevOps & Collaboration Work with GitLab for source control and Jira for project tracking. Participate in migrations from Azure DevOps to modern tools.

• Your Tech Playground

• • Programming & Tools: Python, JupyterHub, GitLab

• • Databases: Oracle DB, PostgreSQL + PGVector

• • Big Data: Spark, Iceberg, S3, Parquet, CSV

• • Graph & AI: Neo4j, LangChain, LLM integration

• • Security: HashiCorp Vault, Kerberos

• • Visualization: Power BI for dashboards and reporting

Skills That Make You Stand Out

• • Strong experience in data engineering, profiling, and ETL processes

• • Expertise in Spark job tuning and large-scale data exports

• • Familiarity with embedding models like BGEM 3 and Nomic

• • Ability to manage high-volume workloads and optimize compute resources

• • Knowledge of cloud and on-prem integration (Azure + on-prem systems)