llm-d 1.0 Advances Open, Cost-Efficient AI Inference for Production Scale

Chris Aniszczyk, CTO, Cloud Native Computing Foundation

“AI inference is quickly becoming a core cloud native workload. llm-d 1.0 helps organizations meet this shift at scale by bringing together open source inference capabilities with the requirements that matter in scaling production to then meet the demands of agentic workloads. It's an important step toward making high performance open source inference more operationally mature and accessible across the cloud native ecosystem.”

Brian Stevens, senior vice president and AI chief technology officer, Red Hat

“As organizations move beyond AI experimentation and into full-scale production, the bottleneck shifts from generating answers to managing the economics, operational reliability, and extreme infrastructure demands of complex agentic workloads. Open source collaboration is essential to solving these day-two operational challenges at scale. With llm-d 1.0, the community is delivering a standardized, hardened core for distributed inference to empower enterprises to optimize hardware performance, address operational costs, and build high-density AI architectures on an open foundation.”

Priya Nagpurkar, vice president, AI native systems, IBM Research

“At IBM, we see firsthand that the challenge of enterprise AI is no longer just building powerful models but running them efficiently and reliably at scale. llm-d 1.0 represents an important milestone in creating an open foundation for distributed inference to help organizations get more value from their AI infrastructure and more control on the AI capabilities they depend on. As agentic workloads become more prevalent, open technologies like llm-d will be critical to improving performance, lowering costs, and accelerating innovation across the AI ecosystem.”

Mark Lohmeyer, VP & GM, AI & Computing Infrastructure, Google Cloud

“Agentic AI is unlocking tremendous value for enterprises and consumers across a broad range of use-cases. However, as these workloads scale rapidly in production, they place unique demands on the infrastructure and require new approaches to deliver the performance, reliability, and cost-effectiveness our customers need. The release of llm-d 1.0 is an important milestone for the industry to address these needs. As a founding contributor to llm-d, Google Cloud is integrating llm-d 1.0 into Google Kubernetes Engine, and enabling it across our Google TPU and NVIDIA GPU portfolio, enabling customers to scale multi-tenant environments, optimize their inference performance, and dramatically lower their overall serving costs on a hardened, open-source foundation.”

Shadi Saba, vice president of engineering, AI/ML, CoreWeave

“Every team running inference at scale hits the same problems: context that often costs more to manage than the output, tenants competing for the same fleet, and upgrades that can't interrupt traffic. llm-d 1.0 gives the community a hardened, open base to solve them once instead of separately. CoreWeave helped found the project, and customers can run llm-d on CoreWeave Kubernetes Service alongside NVIDIA Dynamo, choosing the serving stack that fits their workload.”

Ujval Kapasi, vice president, Engineering AI Frameworks, NVIDIA

“The next wave of agentic AI demands inference that delivers performance and efficiency at scale. Bringing llm-d to NVIDIA Vera Rubin will help enterprises build and scale advanced AI applications on the next generation of NVIDIA accelerated infrastructure. Through our collaboration with the llm-d community and contributions from open source NVIDIA Dynamo, we're helping enterprises get more from their AI infrastructure investments.”

Phil Guido, executive vice president and chief commercial officer, AMD

“With llm-d v1.0, we are advancing open, high-performance distributed inference on AMD Instinct™ GPUs. Agentic workloads require infrastructure that delivers responsiveness and efficiency. Working with the llm-d community, we're delivering the latest inference innovation on an open foundation, giving developers greater flexibility and choice to deploy AI at scale.”

Acyr Locatelli, VP, Modeling, Cohere

“At Cohere, we face the dual challenge of training frontier language models while serving them efficiently at enterprise scale. llm-d helps us optimize for both, whether maximizing throughput for large-scale reinforcement learning workloads or delivering the responsiveness users expect in production. Its flexible, open-source architecture aligns closely with our commitment to sovereign AI, giving organizations greater control over how AI is deployed, operated, and scaled.”

Salman Paracha, SVP AI Engineering, DigitalOcean

“DigitalOcean is excited about the llm-d 1.0 release and relies on llm-d as foundational distributed inference software for DigitalOcean Serverless Inference and Managed Agents, serving trillions of tokens. Its Kubernetes-native design and integration with the Kubernetes Gateway API Inference Extensions provide robust orchestration and endpoint picking, while production-proven paths for dynamic flow control, inference-aware load balancing, and prefill/decode disaggregation help us operate reliably at scale. We collaborate closely with the llm-d community, contributing upstream features and feedback, and look forward to the continued development of llm-d.”

Vladislav Tankov, Director of AI, JetBrains

“JetBrains has been working with llm-d since its early releases, and it is great to see the project reach 1.0. llm-d now runs in production behind our new Junie Lite plan and serves a significant share of its traffic. Our traffic ranges from long agentic coding sessions to short requests, and predicted latency-based scheduling was straightforward to enable and keeps latency consistent across that mix. Junie Lite brings its AI coding agent to developers at no cost, and llm-d helps make that possible by letting us serve more requests per GPU at lower cost and with faster responses. We have enjoyed working with the llm-d team and congratulate them on this release.”

Mathis Felardos, Lucas Fernandes-Martins and Nikhil Thomas, Inference Engineers and llm-d contributors, Mistral

“Serving frontier models across interactive applications and high-volume APIs means handling steep traffic spikes without letting bursty workloads degrade latency for priority customers. With llm-d's prefix-cache-aware routing, flow control and disaggregated serving support we can enforce strict priority bands across our commercial tiers, automatically holding or shedding lower-priority requests when pools saturate so paid traffic meets our latency SLOs. Mistral is a proud supporter of open-source software, and we will continue to contribute back to llm-d and DisaggregatedSet projects.”

Nishchaie Khanna, Sr Director of Engineering, AI Platform, Roblox

“At Roblox, serving large language models efficiently across a mix of cloud and on-premises GPUs is one of our core infrastructure challenges. llm-d gives us a Kubernetes-native foundation for disaggregated serving and cache-aware routing on the hardware we already run, without tying us to a single inference engine. We're excited to build on llm-d 1.0 and to contribute back to the community.”