llm-d 1.0
Google Cloud, Red Hat and IBM released llm-d 1.0, the first stable version of an open-source Kubernetes-native distributed inference stack for vLLM and SGLang, at PyTorch Conference.
llm-d 1.0 turns vLLM or SGLang model servers into one coordinated pool, using cache-aware routing, tiered KV cache offload and prefill/decode disaggregation. It adds multi-tenant flow control, zero-downtime upgrades, session-aware routing for agents and a stable v1 API. It runs on NVIDIA, AMD or Intel GPUs and Google TPUs.
- Date
- Tuesday 20 October 2026
- Lab
- Google Cloud
- Kind
- infra
- Access
- open weights
Figures
| Measure | Value | Measured by |
|---|---|---|
| Median time to first token reduction, Ironwood TPU coding workload | more than 85% with 2.4x throughput | company |
| Accelerator duty cycle in RL time-slicing | above 70% from a 40% baseline | company |
| Gemini platform prefix cache hit rate | 35% to 70% on GKE Inference Gateway using the llm-d router | company |
| Contributors | more than 500 from over 25 organizations | company |
This is open-source software under Apache 2.0 and a CNCF Sandbox project, not model weights. All performance figures are vendor-reported.
Sources
- marktechpost.com/2026/10/20/llm-d-1-0-open-source-distributed-inference-stack/?amp
- forkast.news/llm-d-1-0-ships-the-serving-layer-deepseek-withheld-the-sponsors-sell-the-man
This record was checked against its sources on 21 October 2026. How we check