An inference request runs in two stages: processing your prompt, then generating the answer. Those two jobs want different things from a GPU.
Matching each inference task to the right XPU improves overall model computation time and cost per token — but only when XPUs can communicate together, efficiently.
Heterogeneous LLM serving
Disaggregated inference separates prompt processing from token generation. The two stages don’t have to run on the same GPU platform.
Prefill and decode stress an accelerator differently. Prefill processes the whole prompt in parallel and is compute-bound. Decode generates one token at a time, reading model weights and KV-cache state on every step, and is bound by memory bandwidth. These are different hardware problems, and no single accelerator is the best answer to both.
That opens a question worth testing: can one model serve a single request across two vendors’ GPUs, with each stage on the platform that suits it? Splitting across vendors also means capacity can come from more than one product line or supply channel, which matters when specific GPUs are scarce or when different clouds expose different inventory.
The plan
I set out to demonstrate that idea with an NVIDIA H100 as the prefill worker and an AMD MI300X as the decode worker. The H100 would process an incoming prompt and build the KV cache. The MI300X would receive the required cache blocks and continue generation as part of the same request.
Both hosts used one GPU and the same GPT-OSS-20B model with vLLM 0.15.0. Dynamo 0.9.1 supplied the frontend, request routing, and worker discovery. The final form used GPU-backed KV buffers and a two-node Kubernetes deployment.
I built the system in layers. First a CPU-staged path established cross-vendor model compatibility. Then a software RDMA path established GPU memory registration and transfer. Finally a Kubernetes deployment placed that path inside pod network namespaces.
Establishing a baseline
Since the machines were on completely different networks, it was simpler to setup a WireGuard connection so they would see each other as if they were on the same network.
The initial path used the H100 for prefill and the MI300X for decode. After processing the prompt, the H100 copied the KV blocks required by the decode worker into host memory. UCX sent those blocks over TCP through the WireGuard link. The MI300X received them into host memory, copied them into its GPU cache, and continued generation. Dynamo coordinated the workers and presented one endpoint to the client.
This baseline was valuable because it removed RDMA and GPU memory registration from the first compatibility check. It answered the most basic question first: could the CUDA and ROCm workers interpret the same transferred model state?
One obstacle was a generic vLLM compatibility value that included the selected attention backend. The CUDA and ROCm workers naturally selected different vendor implementations, so their aggregate values differed even when their KV layouts were compatible. I adjusted the handshake so that a vendor-specific implementation label did not create a false rejection. I kept the concrete compatibility checks: the same model snapshot, cache layout, block geometry, region count, and transfer metadata.
The ROCm worker also needed the correct host-buffer integration for copying KV blocks between GPU memory and CPU staging. After aligning those paths, the complete request could move from H100 prefill to MI300X decode and produce the expected semantic answer.
The container environments needed deliberate alignment. The H100 could use the connector packages already present in its CUDA image, while the MI300X required ROCm-compatible connector builds. I pinned the relevant revisions, made the AMD connector available through the interface expected by Dynamo, and removed an unused CUDA-only dependency that blocked the AMD worker. Dynamo remained the common coordination layer while each worker retained its native vendor stack.
Creating an RDMA-compatible path without RDMA hardware
The next goal was to configure the application with GPU-backed KV buffers and exercise its RDMA interfaces. The available machines had no physical RDMA network devices. I therefore used RXE (Soft-RoCE) to provide the RDMA device seen by UCX and NIXL.
RXE could not be attached directly to WireGuard. WireGuard exposes an IP-only interface, while RXE expects an Ethernet-style device with link-layer addressing. I placed a point-to-point VXLAN over the WireGuard connection and attached RXE to that VXLAN. This preserved the existing host-to-host path while supplying the interface model RXE required.
The network device was only the first part of the problem. Standard RXE expected ordinary system-memory regions. The H100 and MI300X instead exported GPU allocations through their respective driver mechanisms, and the two implementations did not present identical mapping behavior. The MI300X KV allocations were also large enough that a simple whole-object mapping was not suitable.
I patched RXE at a platform level to accept GPU memory from both NVIDIA and AMD, safely handle the large MI300X mappings, and expose those registrations through the corresponding userspace RDMA path. From the application’s perspective, the KV regions were GPU-backed memory registrations rather than CPU staging buffers.
Initialization order then became important. Registering the MI300X KV cache pinned a significant part of GPU memory. When registration happened before model warm-up, the ROCm runtime could no longer obtain the temporary memory needed to finish initialization. I deferred cache registration until warm-up was complete, after the model, kernels, sampler, and output path had performed their setup work.
With this sequence in place, both workers reported GPU-backed KV buffers and NIXL and UCX moved the registered regions through the software RXE path.
Moving from hosts to Kubernetes pods
I next created a two-node k3s cluster and deployed the serving graph with DynamoGraphDeployment. The deployment included the Dynamo frontend, an H100 prefill worker, an MI300X decode worker, and Kubernetes-based validation clients.
The host-network form was a useful intermediate checkpoint because it reused the RXE devices already configured on the machines. The final form needed to work without giving the workers the host network namespace.
This exposed a subtle RXE assumption. A Kubernetes pod can create a network device inside its own namespace, but the original RXE implementation still created its UDP resources and searched for routes from the host namespace. As a result, a pod-local RXE device could exist and appear active while its route lookup occurred in the wrong network.
I patched RXE so the device, sockets, and route lookup followed the network namespace of its backing interface. Each worker pod then created its own VXLAN and RXE device. Flannel carried the pod traffic between the nodes, using WireGuard as the host-to-host connection underneath.
The final DynamoGraphDeployment used normal pod addresses. It did not rely on host networking, host IPC, or host PID sharing. The privileged capability required by the workers was limited to constructing the pod-local network devices needed for this software RDMA environment.
Validation
A cross-vendor response alone would not show where the model state had traveled, so I checked the important boundaries independently.
I validated the system at several layers. At the memory layer, the H100 registered NVIDIA GPU allocations and the MI300X registered AMD GPU allocations through the RDMA path. At the data layer, I traced representative K and V tensors and compared their hashes before and after transfer; the producer and consumer values matched. At the transport layer, the pod-local RXE devices exchanged RDMA traffic across the VXLAN path. At the application layer, the request completed with the expected answer. At the orchestration layer, Dynamo discovered the two workers and routed the request from prefill to decode inside the Kubernetes deployment.
What the experiment demonstrated
The result is a concrete demonstration that H100 prefill and MI300X decode can participate in the same disaggregated inference request. In this environment, software RXE provided the RDMA-compatible transport, while the application used GPU-resident KV buffers and preserved the data correctly across CUDA, ROCm, NIXL, UCX, Dynamo, and Kubernetes.
The scope of the result is equally concrete: it validates the cross-vendor application, memory-registration, data-transfer, and orchestration path implemented in this environment. The two GPU platforms remained native to their own software stacks while participating in a shared inference pipeline.
