Scaling LLM Inference with llm-d and NeuReality Inference Serving Stack

Introduction

Multiple open-source frameworks such as vLLM Production Stack, AIBrix, NVIDIA-Dynamo, llm-d, and KServe are reshaping the industry’s approach to Generative AI inference at scale. NeuReality provides AI infrastructure solutions and Inference Serving Software Stack to enable high-efficiency cost-effective inference and training at scale, agnostic to the deployed GPUs/XPUs.

NeuReality successfully deployed llm-d for prefill-decode disaggregation of Meta’s Llama 3.1 70B serving on NeuReality NR1 AI head nodes hosting Qualcomm Cloud AI100 Ultra inference cards.

In this blog post, we share our perspective on the emerging generative-AI-at-scale framework landscape, and describe our experience with the llm-d framework.

Generative AI inference at scale

2025 started with an “explosion” of generative-AI-at-scale frameworks. Within the first five months, five open-source frameworks appeared, each backed by major industry players and open-source community. The first one was vLLM Production Stack, followed by AIBrix from ByteDance (the TikTok owner), and then NVIDIA Dynamo Platform. Shortly after, llm-d was introduced, led by RedHat, IBM Research and Google, with contributors from CoreWeave, NVIDIA, AMD, Cisco, Hugging Face, Intel, Lambda, Mistral AI, University of California, Berkeley, and the University of Chicago.  Just a week after the llm-d launch, Kserve, a popular predictive inference serving framework on top of Kubernetes, announced its own version of a generative-AI-at-scale framework, in a blog post by engineers from Bloomberg, RedHat, SAP and Nutanix.

All the frameworks have rather similar architecture, with some differences, and handle a subset of the following aspects of generative AI inference at scale:

1. Routing by model name 
2. Prefix-aware, KV-cache aware, load-aware routing 
3. Managing disaggregated prefill-decode
4. Multi-tier KV cache management 
5. Advanced autoscaling that takes into account prefill-decode disaggregation
6. Multi-node serving (pipeline/tensor/expert parallelism)
7. Managing models and LORA adapters
8. Benchmark tools
9. OpenAI API server simulation

The frameworks boast up to 57x (this is not a typo, in fifty seven times!) improvements in time-to-first-token (TTFT), inter-token latency, throughput. A major improvement potential is KV-cache management, this article claims that 50% of the tokens in the real-world workloads can be reused.

Reference Architecture of Generative AI inference at scale frameworks

The following diagram shows a reference architecture of generative-AI-inference-at-scale frameworks. Note that the framework components are logical, each component might be implemented by multiple concrete components. The reference architecture is not exhaustive. We focus on the features we believe are most important and provide high-level, simplified explanations.

In the description that follows, the numbers in parentheses relate to the numbers on the arrows. The order of numbers is not important, the numbers do not represent a sequence of events.

Figure 1: Reference architecture of generative-AI-inference-at-scale frameworks

The frameworks orchestrate a pool of instances of vLLM (and of other LLM inference servers), separated into prefill and decode ones. Each decode instance can also perform prefill if needed, for example, for small prompts, since in such a case disaggregating prefill and decode is not worth it. Decode instance will perform prefill itself also in case no prefill instance is available.

The client application sends OpenAI API request (1), for example a chat completion request to Inference Gateway. Here we talk about a logical concept, not specifically Inference Gateway of Kubernetes. In Kubernetes, Inference Gateway and Inference Scheduler components together form the logical concept that we call Inference Gateway. Inference Gateway routes the request to the decode instance (2), or first to prefill instance and then to decode instance (in AIBrix). The decode instance either gets the IP of its prefill instance from the Inference Gateway (in HTTP Header), or it decides itself which prefill instance to use (in Dynamo).

The gateway considers the load of the instances, reported by metrics (3). Metrics are scraped by a Metrics Database such as Prometheus (4) or queried directly from vLLM instances. The gateway sends the request to a least loaded instance, while the load can be measured by the number of waiting requests in the instance.

In addition, the gateway can store hashes of prefixes of the prompts it encounters and send the request to a vLLM instance it sent a prefix before. This will increase the probability that the vLLM instance already contains KV-cache in its GPU memory (or evicted to host memory). This calculation is internal to the gateway, there is no arrow to other components on the diagram. Note that the gateway can take multiple factors in the decision where to route a request to, for example load, existing active connections, prefix matches. The gateway can be configured to assign different weights to different factors.

If there is no prompt-prefix match and the prompt is not small (above some preconfigured threshold), the gateway must select a prefill/decode pair.  In Dynamo, the gateway selects the decode instance only,  and then the decode instance selects a prefill instance. The prefill instance of the pair will send KV cache to the decode instance of the pair after it finishes KV cache calculation (5). KV transfer can be either push (the prefill instance initiates a connection to the decode instance) or pull (the decode instance initiates a connection to the prefill instance).

Each vLLM instance reports information about KV-cache blocks it has, to a KV Cache Lookup database (for example Redis) (6). The database contains mapping from the tokens to the instances that contain KV-caches for the tokens. KVCacheManager queries KV Cache Lookup database (7) to get information about the KV-cache blocks. The gateway can consult with KVCacheManager (8) to select an instance that contains KV-cache for a prefix of the prompt of the request. Note that this functionality is similar to prefix-aware routing. In prefix-aware routing, the gateway consults its internal map of prefixes (their hashes) to instances. This map is built by the gateway itself according to the requests it encountered. In KV-cache-aware routing, the gateway consults KVCacheManager to route to an instance that contains KV cache for a prefix of the prompt.

KVCacheManager might instruct vLLM instances to evict KV-caches to a global KV-cache store (9), in case there is lack of space in GPU memory. KVCacheManager might instruct vLLM instances to evict KV-cache from GPU memory to CPU memory on host, or to the global KV cache database, or to both. KVCacheManager might ask vLLM instances to perform prefetch of KV cache blocks (10) when requests with the prefixes matching KV cache blocks are routed to these instances.

Finally, Autoscaler is a component, similar to Horizontal Pod Autoscaler (HPA)  of Kubernetes, that performs autoscaling of vLLM instances taking prefill and decode instances into account. Autoscaler performs its decisions based on the metrics in the Metrics DB (11).

The autoscaler sets the number of prefill and decode instances (12), either directly or indirectly, influencing Kubernetes autoscalers. As a more advanced component comparing to HPA, the autoscaler can also calculate the optimal vLLM parameters, for example batch sizes, GPU counts and configure the vLLM instances.

Only AIBrix and Dynamo handle multi-node inference (for example pipeline parallelism) explicitly. In other frameworks, and also optionally in Dynamo, multi-node inference is handled by Ray and Kubernetes using LeaderWorkerSet. Note that we have an interesting case here of an orchestrator (Ray) in another orchestrator (Kubernetes).

Generative-AI-inference-at-Scale frameworks landscape

All the frameworks are Open Source with Apache 2.0 license. AIBrix, llm-d and Kserve are Kubernetes-native, that is to say they are built on Kubernetes from the beginning, according to Kubernetes principles and run on Kubernetes only. vLLM Production Stack and Dynamo can run outside of Kubernetes. vLLM Production Stack implemented the Inference Gateway (they call it Router) in Python. Dynamo implemented the Inference Gateway (they call it KV-aware Router or Smart Router) in Rust. AIBrix, llm-d and Kserve implemented the Inference Gateway using Envoy AI Gateway orEnvoy Gateway that are based on Envoy proxy, a popular proxy in cloud environments. In our opinion, this is an advantage of AIBrix, llm-d and Kserve, while NVIDIA Dynamo introduces its own custom proxy implementation.

vLLM Production Stack, llm-d and KServe use LMCache for KV-cache storage. AIBrix and Dynamo use custom distributed storage implementations.

With such a choice of frameworks, it seems that selecting the best one is not a trivial task. AIBrix seems to be the most advanced and the most mature one. ByteDance claim in their announcement in February 2025 that they successfully deployed AIBrix to support multiple business use cases. All other frameworks are rather new, and it is not clear if any of them are used in production. NVIDIA Dynamo explicitly states that it is in Alpha stage. While multiple accelerator vendors endorsed llm-d (NVIDIA, AMD, Intel, Google), it is not clear if accelerators other than NVIDIA GPU can be orchestrated by NVIDIA Dynamo. NVIDIA Dynamo, AIBrix, Kserve support multiple LLM inference servers, see below.

• The table below shows framework popularity judging by the number of GitHub stars, forks and contributors. Note that Kserve is an existing framework for predictive inference, so its data represents its popularity so far, both predictive and generative inference frameworks.

NVIDIA Dynamo and AIBrix seem to be “single-company open source” projects, for better or for worse, while llm-d and Kserve are driven by multiple organizations. The leading contributors of VLLM Production Stack are PhD students at Chicago University. Interestingly, while NVIDIA is a founding member of llm-d and they expressed intent to contribute parts of Dynamo to llm-d, it continues to develop their framework (Dynamo). Another twist is that Kserveuses llm-d components and there are contributors from RedHat in both llm-d and Kserve. In any case, since all the frameworks are Open Source with a permissive license, the frameworks can use any components or borrow ideas from other frameworks.

Considering all the above, we think that llm-d is poised to become the leading generative-AI-at-scale framework, though it will have tough competition from AIBrix. llm-d is Kubernetes-native, is based on production-tested, popular open-source components, is written in “the Kubernetes programming language” (Go), endorsed by multiple accelerator vendors, and has a large community behind it, with companies that have proven open-source success record. Google managed to build Kubernetes as the leading container orchestration framework. Together with IBM and others, Google succeeded again in building Istio as a leading service mesh (not without some glitches in the process). Now we expect that Google, together with RedHat, IBM and others, will be able to recreate the previous success stories and build the leading generative-AI-at-scale framework.

Deploying llm-d for Prefill-Decode Disaggregation with Qualcomm Cloud AI 100 Ultra

At NeuReality, we run Qualcomm Cloud AI 100 inference cards connected to NeuReality NR1 AI head nodes, ARM-based servers-on-chip that include embedded AI-NIC directly connecting them to the network. On these modules, we launch vLLM instances as part of NeuReality Inference Serving Software Stack, modified to leverage Qualcomm inference cards and NeuReality hardware. NR1 AI head nodes are connected to Kubernetes clusters as worker nodes, in addition to x86 worker nodes and x86 control plane nodes. We used Qualcomm device plugin to manage Qualcomm devices on NR1 AI-head nodes.

We modified llm-d-routing-sidecar to use our high-performance hardware accelerated KV-cache connector.

We deployed llm-d as the prefill-decode disaggregation “well-lit path”, using eight prefill instances and one decode instance deployed on separate server nodes. The llm-d-inference-scheduler was configured to use prefill and decode filters, queue-scorer and active-request-scorer, for optimal load balancing among prefill instances and among decode instances.

Figure 2: 8Px1D vLLM pods, Inference Gateway and Inference Scheduler pods

Using llm-d-benchmark and the inference-perf harness, we achieved TTFT improvement that we expected.

Conclusion & Next Steps

We believe llm-d has the potential to become the leading generative-AI-at-scale framework. NeuReality successfully deployed llm-d on NeuReality Kubernetes clusters and integrated it with NR1 AI head nodes hosting Qualcomm AI100 Ultra inference cards. Having llm-d/vLLM integrated into our software stack, opens the path for our customers to use other existing and in-development features such as: Wide Expert Parallelism well-lit path, LMCache, and Workload Variant Autoscaler.
 We will continue monitoring and leveraging developments in generative-AI-at-scale landscape and frameworks.

🌐 Contact us at https://www.neureality.ai/contact to learn more about NR1 AI-headnode with its embedded AI-NIC loaded with our Inference Serving Stack (NR-ISS) and our ScaleOut Networking Stack (NR-SONS), as well as on NR2 AI-SuperNIC product coming out soon focusing on ScaleOut efficiency.

Legal Statements

© 2025 NeuReality Ltd. All rights reserved.

This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NeuReality Ltd (“NeuReality”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. Some of the information presented herein is based on third-party sources. NeuReality does not verify, endorse, or take responsibility for the accuracy or truthfulness of such third-party information. NeuReality accepts no responsibility or liability for any errors or omissions, or for any losses, damages, or other consequences that may arise from any party’s reliance on such information.

All third-party trademarks or brand names mentioned in this document, including but not limited to NVIDIA®, Qualcomm®, ByteDance®, AIBrix®, and others, are the property of their respective owners. These references are used solely to identify and describe the respective products or services. No association, sponsorship, or endorsement by these entities is implied.