The inference serving problem is not what most people think it is. It was always assumed that AI training required special, very high-performance, large-scale distributed compute, with HPC infrastructure characteristics, while AI inference was the easy part. Embedded in the application, the user required an AI model output, hence getting the result from a specific, bounded compute resource.
But that is not the case. Today, production AI inference at scale requires an AI NIC for inference designed specifically for distributed AI workloads. AI inference is a scaled deployment problem: running large, distributed workloads, handling large-scale context, KV-cache inputs and outputs, and sharing them between sessions and agent tasks.
The industry conversation is fixated on local GPU FLOPS, GPU memory bandwidth, and model optimization and quantization. These all matter. But for production AI inference at scale, the network has become a first-class performance constraint. It determines how efficiently data is exchanged, synchronized, and moved across the cluster, and therefore how much of the deployed GPU capacity can actually be used.
As GPU generations improve, communication can become a larger share of runtime. In large-scale distributed workloads, even small increases in latency and its variance can reduce utilization, increase Time to First Token (TTFT) and Time per Output Token (TPOT), and raise cost per token.
When an organization deploys a large language model at production scale, it may be serving thousands of concurrent requests, running tensor, pipeline, or expert parallelism across multiple GPUs, and moving KV-cache state across memory tiers or between dedicated XPUs, for instance, prefill and decode workers. In that environment, the scale-out network sits on the critical path of inference performance. XPUs denotes dedicated workload accelerators (GPU is a type of XPU)
While inference may be served on multiple types of infrastructure, ranging from AI factories to edge equipment, dedicated AI infrastructure at scale requires dedicated AI networking, where efficiency determines how much usable output can be generated from deployed compute. Yet many data centers still rely on general-purpose networking designs that were optimized for conventional cloud and storage traffic, not high-frequency AI communication.
This article explains why AI inference demands a different networking architecture, what an AI NIC for inference must do, and how NeuReality’s NR2 AI-SuperNIC targeted for AI training and Inference networking, addresses these requirements at the architectural level.
How AI Inference Actually Uses the Network
To understand why standard NICs fall short, it helps to look at the networking patterns that emerge in production AI inference.
Model Parallelism Across GPUs in Inference
Large models, including GPT-class models, Llama-class models, and Mixture-of-Experts architectures, often cannot be served efficiently on a single GPU or a single server, especially under production concurrency, long-context, or latency-target requirements.
Models are partitioned across GPUs using tensor parallelism, pipeline parallelism, expert parallelism, or combinations of these techniques. Scale-up interconnects are typically used inside a node or rack. Beyond that boundary, the workload depends on the scale-out network.
Information like at layer boundaries and synchronization points, intermediate activations, expert-routing data, and collective communication results must move between GPUs. These transfers are often small, frequent, and latency sensitive. In MoE inference, for example, dispatch-combine traffic can be difficult to hide behind compute.
These functions are performed using collective communication operations such as AllReduce across tensor-parallel GPU groups, AllGather, broadcast during dynamic model loading and Expert splitting, and scatter/gather patterns used by advanced serving techniques. Also, when these operations are not done efficiently, are handled by host CPUs, or consume GPU resources, they take cycles away from token generation and degrade further the performance of the system.
This is why distributed inference requires high-throughput direct memory access, low response latency, and predictable transport behavior offloaded from the CPU and GPU. Without that, network overhead appears directly in token latency and GPU idle time.
KV-Cache Mobility
The mobility of the KV-cache as a model context representation, with larger context-dependent sizes, becomes a prime issue in distributed, at-scale AI inference. Modern inference optimizations and agentic AI workloads make KV-cache placement a system-level performance issue.
During prefill, the model computes the KV cache for the input context. During decode, that cache is reused to generate output tokens. In production systems, KV-cache state may be reused across sessions, moved between prefill and decode workers, offloaded to memory tiers, or onboarded from remote nodes.
These memory tiers can include GPU HBM, CPU DRAM, SSDs, and remote memory pools across the cluster. Large-scale serving systems also use KV-aware routing and cache-locality policies to decide where each request should run to avoid or reduce KV-cache movement.
The placement of the KV-cache, in order to either continue prefill work or start Decode, is a binding requirement, as work can’t start before the required KV-cache is available. It impacts both system utilization and AI performance, measured in TTFT and TPOT parameters, both of which are critical in AI scale deployment.
In disaggregated inference architectures, prefill and decode may run on separate XPU pools. That makes the network part of the decode critical path: the decode worker may wait for KV-cache state before it can generate tokens.
This requires network hardware that can move KV-cache data with high throughput, low latency, and low queueing jitter. In representative deployments cited in NeuReality’s scale-out networking white paper, KV-cache onboarding can consume more than 20% of decode time under 400 Gbps networking, while multi-terabit networking can reduce this overhead to single digits.
Figure 1: Onboarding large KV-caches over a network can consume significant execution time,
even in hybrid models with attention windows[1].
Disaggregated Inference Across Heterogeneous XPUs
A growing inference architecture separates different workload stages across dedicated hardware pools. Prefill-decode disaggregation is one example: the prefill phase processes the input prompt, while the decode phase generates output tokens. Attention and feed-forward stages may also be disaggregated in some architectures.
These stages have different computational, memory, and latency profiles. In heterogeneous clusters, they may run on different XPU types, connected through the scale-out network. Vendor-specific scale-up interconnects do not extend cleanly across cluster boundaries or across vendors, so scale-out networking is the more common shared communication layer.
Disaggregation creates a new dependency: intermediate state must move between workers at the right time. Depending on the serving architecture, that state may include synchronization information, KV-cache data, embeddings, activations, or other request-specific data.
Disaggregation only helps when the scale-out network can move that state with low latency, sufficient bandwidth, and predictable performance.
The Limits of Standard NICs in Inference Workloads
To see why standard NICs fall short in distributed inference, it is useful to look at what they were not designed to do.
RDMA is a baseline requirement for modern scale-out AI networking because it reduces host involvement in data movement between GPUs, servers, and memory tiers. But RDMA alone is not enough. At AI-cluster scale, the transport layer must also manage congestion, reliability, and multipath utilization while serving burst behavior, elephant flows, and AI-specific communication patterns without adding jitter.
The core requirement is simple: GPUs need high-throughput data movement and synchronization traffic delivered with minimal CPU interference and predictable low latency. That is why AI infrastructure increasingly depends on RDMA-capable, AI-optimized scale-out NICs rather than conventional Ethernet adapters alone.
CPU and Software Overhead in the Data Path
In conventional networking architectures, the CPU is involved in data movement. RDMA reduces this burden, but the host CPU can still be involved in connection management, completion processing, and coordination. The NIC architecture determines how much latency, jitter, and scheduling overhead remain in the critical data path.
For inference workloads, every microsecond of latency can affect Time to First Token (TTFT) and Time per Output Token (TPOT). Residual CPU involvement or software-driven network coordination can add jitter and reduce GPU utilization.
Lack of Quality of Service for AI Traffic
AI inference traffic has distinct quality-of-service requirements. Collective communication and decode-stage messages need low latency and high priority. KV-cache transfers and model movement need high bandwidth. Control traffic needs reliability. A NIC must provide QoS across these competing traffic types so the system can maintain predictable high performance with low variability.
Standard NICs can provide conventional traffic management, but they generally lack AI-aware scheduling semantics. They can move packets, but they do not inherently understand which transfers sit on the token-generation critical path, which can tolerate delay, and which must be protected from congestion and jitter.
Lack of Collective Operation Support
Standard NICs – even high-bandwidth 400GbE NICs – have limited native understanding of AI collective operations. When AllReduce or expert-parallel inference requires reductions, dispatch-combine, or synchronization across many GPUs , the operation is typically orchestrated above the NIC by software, CPU, or GPU resources. It requires handling of many concurrent flows while trying to minimize data transfer and overlap it with compute time.
That introduces latency and consumes host or GPU resources that could otherwise be applied to inference.
Throughput-Latency Tradeoffs
High-throughput NICs designed for storage or hyperscale web workloads are optimized for bulk transfer efficiency and operation across a large number of flows. They may achieve excellent bandwidth utilization, but they can also introduce buffering, pacing, and scheduling behaviors that add latency variance.
For production inference, variance is a problem. Consistent response times depend not only on bandwidth, but also on predictable low-latency communication. At higher link speeds, fixed per-transaction latency becomes a larger share of total transfer time for short AI messages. Congestion control, multipath utilization, and tail-latency behavior therefore become primary design concerns.
This is why the industry is moving beyond classic single-path assumptions for AI scale-out networks. OpenAI’s MRC work is a useful example from large-scale AI operations: it extends RoCE with multipath reliability, adaptive packet spraying, SRv6 source routing, and fast failure avoidance to reduce congestion and jitter in very large GPU fabrics. The reference[2] is training-focused, but the design lesson applies to inference as well: AI NICs must use the available fabric efficiently, avoid hot spots, and keep short, latency-sensitive transfers predictable under load.
NeuReality’s NR2 AI-SuperNIC: Purpose-Built for AI Factories
NeuReality designed the NR2 AI-SuperNIC as a purpose-built AI NIC for AI scale-out networking across both training and inference. In production-scale inference serving, that means delivering:
- Ultra-low latency for short, high-frequency AI messages in decode, MoE, and collective communication paths
- High scale-out bandwidth for collectives, KV-cache movement, prefill-related data, and model movement
- In-network collective offload for operations such as AllReduce, AllGather, and MoE dispatch-combine patterns
- Native integration with collective libraries and inference frameworks such as vLLM, SGLang, and TensorRT-LLM through xCCL or inference-specific equivalent communication libraries
- Deterministic performance with minimal jitter across latency-sensitive, bandwidth-intensive, and control-plane traffic classes
- Standards-aligned Ethernet transport, including Ultra Ethernet and RoCEv2/MRC approaches for interoperability and scale
- Flexible datapath that can adapt to evolving algorithms, congestion-control approaches, and AI communication patterns
- Vendor-agnostic design that can operate across heterogeneous XPU platforms
To deliver the above, NR2 is built around a deterministic hardware datapath with on-chip connection state, flexible transport engines, and in-network compute support for collective operations. GPU traffic moves through the networking path without placing the CPU or NPU in the critical path.
1.6 Tbps Bandwidth for Multi-GPU Inference Clusters
NR2 delivers 1.6 Tbps of scale-out bandwidth, helping the network keep pace with next-generation GPUs and XPUs . This bandwidth is especially important for disaggregated inference architectures, where KV-cache transfers between prefill and decode workers, KV-cache onboarding from memory tiers, and model movement can create heavy network load during peak serving periods.
NR2 is based on open Ethernet approaches, including Ultra Ethernet and RoCEv2/MRC, and supports efficient multipath load balancing, programmable congestion control, selective retransmission, and out-of-order handling at scale.
Deterministic Low-Latency Hardware Datapath
At higher link speeds, latency becomes a primary limiter of system performance rather than a secondary concern. NR2’s Hardware Transport Engine enables RDMA while improving overlap between communication and computation, reducing idle GPU time, latency variance, and scheduling overhead.
In inference serving, this can translate into lower inter-token latency, higher tokens per second per user, and better GPU utilization. These improvements affect user-facing metrics such as Time to First Token (TTFT) and Time per Output Token (TPOT), especially in real-time AI applications.
In-Network Collectives and xCCL Integration
NR2 includes in-network compute support for collective and math-collective operations, with xCCL integration for heterogeneous environments. This helps move communication-heavy operations closer to the network path, reducing data movement and lowering the amount of GPU work spent on coordination instead of inference.
Flexible Transport for Evolving AI Workloads
NR2 also includes a Flexible Transport Engine (FTE/FleXport) to support evolving communication patterns, congestion-control approaches, and network topologies. This matters because inference workloads and systems are changing quickly, from prefill-decode disaggregation and MoE routing to heterogeneous XPU pools and expanding context windows.
NR2 and NR-NEXUS™ Together
These capabilities become stronger when paired with an optimized inference operating layer. The NR-NEXUS™, developed by NeuReality, coordinates distributed inference workloads, resource pools, routing, KV-cache movement, and transport across GPU and XPU clusters. NR2 provides the scale-out networking layer; NR-NEXUS™ provides the inference control layer above it. Together, they address the same production problem: keeping GPU capacity active, predictable, and usable at high performance and cluster scale.
Frequently Asked Questions (FAQ)
What is an AI NIC for inference?
An AI NIC for inference is a network interface designed for distributed AI workloads. NeuReality’s implementation of this category is the NR2 AI-SuperNIC. It supports high-throughput, low-latency scale-out communication, RDMA-based data movement, collective-operation offload, and efficient movement of data between GPUs, memory tiers, and servers.
Unlike a standard NIC, an AI NIC is optimized for AI scale-out communication with training- and inference-specific traffic patterns such as MoE dispatch-combine, collective communication, KV-cache transfers, and prefill-decode disaggregation.
Why does AI inference need a specialized NIC?
AI inference workloads create networking patterns that standard NICs were not designed to handle at cluster scale. These include frequent high-bandwidth, low-latency collective communication, KV-cache transfers in disaggregated architectures, MoE dispatch-combine traffic, and congestion-sensitive multipath flows.
A specialized AI NIC can reduce latency variance, improve multipath utilization, offload collective operations, and integrate with inference serving frameworks. This can improve token throughput, increase GPU utilization, and reduce cost per token.
Does an AI NIC help with single-GPU inference, or mainly distributed inference?
An AI NIC matters most in distributed inference, where requests, KV-cache state, collective communication, and intermediate data move across GPUs and XPUs, servers, or memory tiers.
For single-GPU inference, the network is usually not the main bottleneck. In those cases, memory bandwidth, batching efficiency, and local KV-cache management tend to dominate performance. The network is still involved in feeding data or KV-cache into and out of the GPU.
Which inference serving frameworks is NR2 compatible with?
NR2 is designed to integrate through xCCL collective communication and inference-specific communication libraries used by major inference serving frameworks and engines, including vLLM, SGLang, TensorRT-LLM, and others.
How does prefill-decode disaggregation affect network requirements?
In disaggregated inference, the prefill phase processes the input prompt, while the decode phase generates output tokens. These phases may run on separate hardware pools optimized for each stage.
This architecture requires KV-cache data and other intermediate state to move between workers or memory/storage components. These transfers require low-latency, high-bandwidth networking with consistent performance. An AI NIC designed for inference must support this traffic pattern as part of the scale-out network for distributed AI workloads.
Can NR2 work with inference clusters that use different XPU types?
Yes. NR2 is designed for heterogeneous XPU environments, including GPUs, FPGAs, and ASICs.
This allows infrastructure teams to build heterogeneous inference clusters without making the scale-out network dependent on a single XPU vendor. Vendor-specific scale-up interconnects do not extend cleanly across cluster boundaries or across vendors, so scale-out networking becomes the shared communication layer.
[1] The plot is calculated based on KV-cache characteristics of GPT-OSS-120B (taking only the layers without attention windows, and increasing context size), combined with generation speeds and concurrent users as interpreted from NVIDIA’s blog post. Set to be fixed across all prompt sizes.
GPT-OS – S. Agarwal et al., “gpt-oss-120b & gpt-oss-20b model card,” arXiv preprint arXiv:2508.10925, 2025.
[2] Open AI MRC – https://openai.com/index/mrc-supercomputer-networking/
