llm-d 1.0 brings tenant and agent-session controls to AI inference

llm-d project contributors released version 1.0 of the Kubernetes-native distributed inference framework on October 20. The update adds operational controls for shared model-serving environments and agent workloads. Announced at PyTorch Conference North America, the release marks the project’s shift from individual performance techniques toward day-two operations.

For teams running models on shared Kubernetes infrastructure, that shift means stable router interfaces alongside tenant-aware traffic controls, session routing and upgrade support. The router 1.0 release moves EPPConfig and InferenceObjective to the stable llm-d.ai/v1 API, lets an InferenceObjective target multiple pools and freezes the configuration surface for core EPP plugins.

llm-d provides a distributed routing and operations layer around inference engines such as vLLM and SGLang, according to IBM Research, which participates in the project. IBM Research calls llm-d vendor-neutral across models, accelerators and clouds, although the reviewed release materials do not independently establish that characterization.

The flow-control layer is designed to protect overloaded deployments, enforce strict priority and allocate capacity fairly among tenants. It remains behind the flowControl feature gate and must be enabled explicitly. The project’s multi-tenant configuration guide shows how operators can set reserved and overflow quotas for teams, rank service tiers by priority and apply fairness within a priority band. Requests receive an HTTP 429 response when a band’s configured capacity is exhausted.

Agent-oriented routing targets repeated computation across multi-turn interactions. Router 1.0 adds a session-state producer, a thunder-agent session manager and scorer, and a per-pod admission gate. The project says these components can direct subsequent turns to the pod holding a session’s key-value cache, making it more likely that earlier context can be reused. The reviewed sources do not include an independent performance result for that behavior.

The router also adds active-active EPP load sharing, fail-open routing when EPP cannot be reached, horizontal autoscaling support and graceful draining after leader-lease loss or SIGTERM. Project materials describe the broader release as supporting zero-downtime operations through high availability, hot standby and rolling updates. The reviewed sources, however, do not include an independent availability test for the full stack.

Operational telemetry now includes W3C trace propagation through EPP, OpenTelemetry tracing in the coordinator, Prometheus trace exemplars and tool-calling telemetry. These features give operators more ways to follow requests across the routing path and inspect agent-related activity.

Adopting the router release requires migration work. Version 1.0 updates the Gateway API Inference Extension dependency to v1.6.2, removes the legacy pool API group and several deprecated plugins, flags and metrics, deprecates InferenceObjective v1alpha2, and standardizes routing headers under the x-llm-d- prefix. The project’s release tracker groups those changes under its broader day-two operations milestone.