Waleed Malik

Project

Kubernetes LLM Inference Platform

A GitOps-managed Kubernetes platform for serving models on your own GPUs, with shared access, usage budgets, inference-aware routing, and observability.

Read the architecture article GitHub Documentation

Self-hosting a model is easy to demo and hard to operate. The missing layer is everything around the model server: who can call it, which model they can see, how much they can spend, where the request should route, which signal should scale capacity, how secrets enter the cluster, and how you know the system is healthy.

This project turns those decisions into a runnable Kubernetes stack. It is not a chatbot app. The model server is one workload inside a platform and it scales, from a single GPU to multi-GPU nodes, as configuration rather than a rewrite.

The platform as a layered stack: experience, tenant edge (keys, budgets, guardrails, SSO), inference-aware routing, serving, GPU scheduling, and cloud substrate, on an Argo CD, External Secrets, and Prometheus foundation.
View full-size inference stack

Architecture decisions

Measure the serving workload

Start with raw vLLM on GPU-backed Kubernetes. Measure time to first token, inter-token latency, throughput, queue depth, and KV-cache pressure with Prometheus, Grafana, and DCGM.

Separate platform from substrate

OpenTofu owns the substrate; Argo CD owns the in-cluster platform. GKE consumes managed GPU plumbing; Hetzner validates the self-managed GPU Operator path.

Route and scale on demand

GIE selects serving endpoints; KEDA scales on queue depth. The serving options are raw vLLM, KServe, and llm-d, including disaggregated prefill/decode with KV-aware routing for multi-GPU workloads.

Keep applications independent

Open WebUI, Tabby, n8n, and an MCP tool gateway use the shared inference platform. Applications can change without taking over model serving, routing, or tenant controls.

Governance & cost

SSO via Dex + oauth2-proxy, PII and prompt-injection guardrails at the tenant edge, Kyverno and default-deny NetworkPolicy enforcement, and OpenCost for per-tenant cost allocation.

Request path

The request path is layered on purpose:

Client / developer tool
  -> LiteLLM tenant edge
  -> Gateway API + GIE routing
  -> vLLM / KServe serving layer
  -> GPU substrate

The important decision is not the component list. It is the ownership boundary: LiteLLM owns tenant economics, GIE owns endpoint selection, vLLM owns token serving, KEDA owns queue-depth autoscaling, OpenTofu owns the cloud substrate, Argo CD owns the in-cluster platform, and External Secrets owns runtime secret delivery.

LiteLLM owns economics while GIE owns inference-aware routing.
View full-size routing diagram

Selected results

On one NVIDIA L4 serving Qwen2.5-0.5B-Instruct with vLLM:

Signal Observed result
GPU utilization 100%
KV-cache usage 1.1%
Waiting requests 0
Throughput at concurrency 32 3,773 output tok/s
TTFT p50, concurrency 1 to 32 27 ms to 135 ms

View benchmark setup and results for the workload, measurements, and test conditions.

The lesson: GPU utilization alone was the wrong scaling signal. Queue depth carried the useful backpressure signal.

These L4 numbers are the cheapest slice, not the ceiling. The same stack runs across multi-GPU nodes with zero-downtime rolling updates; adding capacity is replica count and GPU budget, not rework.

Built for production

Highly available by design; zero-downtime rolling updates across multi-GPU nodes, with SSO and policy-enforced tenancy, per-tenant budgets and cost attribution, and disaggregated prefill/decode serving for larger models.

Related writing