Role in one line
Build and operate the self-hosted AI inference platform — Kubernetes, NVIDIA H100 infrastructure, vLLM, observability, GitOps, and production reliability.
Context
We are building a self-hosted LLM inference platform running on NVIDIA H100 GPUs and bare-metal Kubernetes. The platform provides vLLM inference workers and an OpenAI-compatible API for internal services.
There is no managed cloud Kubernetes layer — the engineer will work directly with the infrastructure and own the platform from cluster bootstrap through production operations.
What you will work on
Bootstrap and maintain self-hosted Kubernetes clusters with HA control plane, etcd backup/restore and upgrades.
Set up and maintain NVIDIA GPU infrastructure, including GPU Operator, DCGM and the CUDA/container runtime stack.
Configure GPU sharing and scheduling using MPS or time-slicing.
Deploy and operate vLLM inference workers with zero-downtime rolling updates and graceful connection draining.
Build Prometheus/Grafana monitoring and alerting for inference SLIs such as TTFT, queue depth, KV-cache utilisation and rejection rate.
Implement IaC for bare-metal provisioning and cluster bootstrap using Terraform or Pulumi.
Set up GitOps workflows with ArgoCD or Flux for controlled rollouts and rollbacks.
Configure API Gateway, token-aware rate limiting, NetworkPolicy and mTLS.
Build automation and runbooks for GPU-specific production incidents.
Must-have
5+ years in DevOps, Platform Engineering or SRE with production Kubernetes.
Hands-on experience with self-hosted Kubernetes (kubeadm, RKE2 or k3s), including bootstrap, upgrades, etcd and HA.
Practical experience with NVIDIA GPUs in Kubernetes, GPU Operator/Device Plugin and the CUDA driver stack.
Strong understanding of GPU sharing and scheduling: MPS, time-slicing and multi-process workloads.
Strong Linux/system administration skills, including NUMA, huge pages, CPU management and bare-metal troubleshooting.
Experience with zero-downtime deployments of long-running or stateful workloads.
Strong Prometheus and Grafana skills, including recording and alerting rules.
Experience with Terraform or Pulumi.
Bash/Python scripting for automation, health checks and monitoring.
Basic understanding of LLM inference metrics and concepts: TTFT, token throughput, KV-cache and batching.
Nice-to-have
Experience with vLLM, Triton Inference Server, KServe or similar.
Production GitOps experience with ArgoCD or FluxCD.
Experience with Kong, Envoy, APISIX or similar API gateways.
Experience with Vault or External Secrets Operator.
CKA, CKS or NVIDIA certifications.
Experience benchmarking GPU workloads under mixed-model load.
Tech stack you will touch
Kubernetes, NVIDIA H100, NVIDIA GPU Operator, DCGM, CUDA, containerd, vLLM · Prometheus, Grafana · Terraform / Pulumi · ArgoCD / Flux · Linux, Bash, Python · API Gateway, NetworkPolicy, mTLS · Docker, Git, CI/CD.
Ways of working
Remote, distributed engineering team.
Production-focused environment with ownership of the infrastructure and inference stack.
The role involves building the platform from the ground up and keeping it reliable in production.
Important: As this is a Germany-based project, we are primarily seeking candidates based in Western Ukraine, with Vinnytsia and Lviv being our preferred locations. Other secure western-region locations can be considered case by case.
