Architect (H/(U/M/a/N)* Data Engineer Azure Databricks Kubernetes Docker

- Good communicator, but great at independent work. Solution oriented. Analytical thinking.
- Experience in machine learning engineering, exploratory data analysis, and software development and writing of ETL-Pipelines. Optimally, candidate should have a degree in mathematics, physics, computer sciences or in a related field.
- Experience in Python programming is mandatory, especially with PySpark, whereas XGBoost, Seaborn, Matplotlib and dbutils (Databricks) are nice-to-have.
- Expertise in Git, GitLab, and CI/CD are beneficial (including Azure CLI and Azure-Cloud specific APIs).
- Experience in working with Azure, Databricks, Kubernetes and Docker.
- Candidate should be familiar or inclined to working in an agile environment. Prior experience with predictive maintenance tasks is a plus.

IHRE AUFGABEN

Our customer has five different machine learning-based solutions that identify weak points in the medium voltage grid. While targeting the same purpose, they differ in their data source systems, ML features, ML algorithms and especially the Distribution System Operator (DSO) for which they have been developed. Currently, none of these five solutions is easily applicable to a different target DSO or implemented in a scalable way. A newly developed data platform (iPEN) that collects, prepares, and provides data from all DSOs now allows to develop a new solution that consumes all needed data from a single source system. Based on unified data preparation, feature extraction, and machine learning steps, this new solution should be applicable to all customers DSOs. Particular attention will be paid to the scalability, stability, and maintainability of the resulting software. Data Science tasks related to projects, e.g. Predictive Maintenance Solutions.

- Data Engineering tasks related to projects, e.g. Predictive Maintenance Solutions
- Advice and designing business-critical data engineering use cases, from the business problem to delivery and operation.
- Programming of data ingestion pipelines from various sources, e.g Graph Database or Data Lakehouse.
- Writing of production feature engineering code in Python and PySpark on a Databricks Tech-Stack.
- Build and maintain data pipelines for static, mixed, and time-series data.
- Design, implementation and maintenance of data infrastructure for ML algorithms in Azure Databricks.
- Responsible for the design and implementation of the CI/CD pipelines.
- Data modeling and architecture (schemes, sources, optimizations).

IHR ANSPRECHPARTNER

Andreas Baxmann
Telefon: +49 221 16020 15
E-Mail: [email protected]