HomeCommunityAI blog
June 29, 2026

Accelerating distributed inference on Arm Neoverse

Learn how hierarchical all-reduce and RDMA accelerate distributed LLM inference on Arm Neoverse, delivering up to 7.45× higher decode throughput

By Bolt Liu

Share
Reading time 12 minutes

Introduction

Developers can use distributed CPU inference across multiple systems for several reasons: to run models that do not fit on a single system, increase inference throughput for smaller models, or use idle CPU server clusters for offline AI workloads. However, adding more systems does not automatically improve performance. Communication overhead can quickly become the limiting factor, preventing distributed inference from scaling efficiently even when additional compute and memory resources are available.

Arm Neoverse-based systems combine high memory capacity, strong SIMD execution, and efficient infrastructure-class compute for CPU-only inference. However, communication can become a bottleneck when tensor parallel inference scales across multiple systems. A conventional flat TCP all-reduce operation can make this issue worse by forcing each layer to wait for repeated communication between systems.

This blog explores how to reduce that overhead on 2 NVIDIA Grace systems that run Arm Neoverse V2 CPUs. Gloo is the default communication backend for CPU-only inference in popular inference frameworks such as vLLM and uses TCP by default. We start with a TCP-bound baseline and apply 3 optimizations:

  • Introduce a hierarchical all-reduce (HAR) design that keeps intra-system traffic in shared memory and crosses the network only once using RDMA.
  • Route the logits collective through the same fast path.
  • Tune OpenMP thread behavior for decoding.

Together, these changes increased median decode throughput from 5.47 to 40.8 output tokens/s for Qwen2.5-14B-Instruct at TP = 4. HAR also benefits smaller distributed topologies. At TP = 2 across 2 systems, HAR reached 34.4 output tokens/s. TCP reached 11.74 output tokens/s, and the direct RDMA path reached 18.68 output tokens/s. The 2-system setup also enabled Qwen3-235B-A22B to run when a single Grace system ran out of memory. These results show that distributed CPU inference improves throughput, topology flexibility, and support for larger models.

Test environment

The evaluation used 2 NVIDIA Grace systems connected over Ethernet. Each system has 2 NUMA domains, 144 Arm Neoverse V2 cores, and 960 GB of LPDDR5X memory. The software stack included vLLM with the CPU backend, PyTorch with ACL, Gloo as the baseline collective layer, Ubuntu 24.04.4, and Linux kernel 6.8.0-124-generic. The primary model was Qwen2.5-14B-Instruct in BF16, with tensor parallelism set to TP = 4.

Component

Configuration

Systems

2 × NVIDIA Grace systems

CPU

Arm Neoverse V2, 72 cores per NUMA domain

Memory

960 GB LPDDR5X per system

Model

Qwen2.5-14B-Instruct, BF16

Ranks

4 ranks, TP = 4

Runtime

vLLM CPU backend, PyTorch + ACL

Network

RDMA with ConnectX NICs

Figure 1. Two-system Grace topology used for TP=4 CPU inference, with shared memory inside each system and RDMA between systems.

Figure 1. 2-system Grace topology used for TP = 4 CPU inference, with shared memory inside each system and RDMA between systems.

Performance bottleneck of the baseline: Flat TCP all-reduce

With TP = 4, each rank computes a partial contribution for row-parallel operations. An all-reduce sums the 4 contributions and returns the complete output tensor to every rank. The next dependent operation can then proceed. In the baseline path, Gloo uses a topology-blind ring all-reduce over TCP. The ring treats all links equally, however, communication within a local intra-system is much faster than communication between systems.

With 4 ranks, a ring all-reduce performs reduce-scatter operation followed by an all-gather operation, requiring 2(N−1) = 6 serialized steps. In this topology, 2 of the 4 ring edges cross between systems, so each serialized step waits for a slower TCP communication across the network. Qwen2.5-14B performs roughly 2 all-reduces per layer, or about 96 collectives per output token, which makes decode network-bound rather than compute-bound.

The baseline achieved 5.47 output tokens/s. At this point, the Grace cores were not the limiting factor. The decode path waited for repeated TCP round trips.

Flat TCP ring all-reduce uses 6serialized steps for TP = 4, and each step waits on a cross-system TCP hop.

Figure 2. Flat TCP ring all-reduce uses 6 serialized steps for TP = 4, and each step waits on a cross-system TCP hop.

This is the performance data of running with TCP gloo compared to single TP:

TP 

Throughputs (t/s) 

1 

23.68 

2 

11.74 

4 

5.47 

Hierarchical all-reduce: Keep local traffic local

The key idea is to split the collective according to the physical hierarchy. Intra-system communication uses shared memory, while inter-system communication uses a single RDMA exchange over Ethernet. We refer to this design as HAR.

HAR performs communication with RDMA and completes the final reduction in shared memory. Each rank writes its shard to the matching rank on the remote system by using RDMA Write with Immediate. After the remote shard arrives, a shared-memory barrier synchronizes the local ranks. Each rank then performs the final local reduction across the 2 shared-memory slots. HAR does require a leader relay or a broadcast. Every worker computes the final result locally.

HAR exchanges same-rank shards across systems with RDMA, then performs the final reduction locally through shared memory.

Figure 3. HAR exchanges same-rank shards across systems with RDMA, then performs the final reduction locally through shared memory.

  1. Fan-out copy: each rank stages its shard into its local shared-memory slot and RDMA send buffer.
  2. Cross-system exchange: same-rank peers exchange shards in parallel over RDMA.
  3. Partial reduce: each rank folds the remote shard into its local slot.
  4. Shared-memory barrier: local peers synchronize.
  5. Final reduce: slot0 and slot1 are summed locally to produce the full all-reduce result.

This replaces 6 serialized TCP-paced steps with a single RDMA exchange, lightweight shared-memory synchronization, and a final local shared-memory reduction. The first optimization step increased throughput from 5.47 to 25.4 output tokens/s, which is a 4.6× improvement over the TCP baseline.

Routing logits through the fast path

After HAR removed the dominant hidden-state all-reduce cost, the per-token logits gather became the next bottleneck. This bottleneck already existed in the framework but became more visible after the HAR optimzation. In the original path, Gloo gathered the logits over TCP sockets. This gather transfers a vocabulary-wide tensor during every decode step, so it became the remaining cross-system bottleneck.

The optimized path zero-pads each rank’s logits shard to the full vocabulary size and uses HAR all-reduce to reconstruct the final logits. This transfers more bytes than a root gather, but it replaces kernel TCP sockets with RDMA verbs and the hierarchical path. The route is topology-aware. When tensor parallel ranks span systems, logits use HAR over RDMA; when all ranks are on one system, the original shared-memory path remains in use.

Fast logits routing replaces the root Gloo/TCP gather with a topology-aware HAR all-reduce over RDMA when tensor-parallel ranks span systems.

Figure 4. Fast logits routing replaces the root Gloo/TCP gather with a topology-aware HAR all-reduce over RDMA when tensor-parallel ranks span systems.

Routing logits through HAR increased throughput from 25.4 to 36.2 output tokens/s, adding 42% on top of the RDMA hierarchical all-reduce improvement.

Keeping OpenMP threads hot for decode

Once the network bottleneck was reduced, CPU-side dispatch overhead became the next bottleneck. Decoding consists of many small parallel regions for normalization, QKV projection, attention, and MLP operations. If OpenMP worker threads sleep between these short regions, the next region incurs futex wake-up, OS scheduling, and cache warm-up costs.

Setting GOMP_SPINCOUNT=100M keeps idle OpenMP worker threads spinning instead of sleeping. On dedicated, pinned cores, those CPU cycles would otherwise be idle, so the trade-off is favorable for latency-bound decoding. This tuning increased throughput from 36.2 to 40.8 output tokens/s, adding another 13% improvement.

Increasing GOMP_SPINCOUNT keeps OpenMP worker threads ready between short decode kernels, reducing wake-up overhead on the critical path.

Figure 5. Increasing GOMP_SPINCOUNT keeps OpenMP worker threads ready between short decode kernels, reducing wake-up overhead on the critical path.

Note: this tuning is best suited to reserved cores. On shared or oversubscribed systems, busy-waiting can waste power and interfere with neighboring workloads.

Measured results

The full optimization ladder shows how each step removed a different bottleneck from the decode path. The final configuration reached 40.8 output tokens/s, or 7.45× faster than the TCP Gloo baseline.

 

Step

Median output tokens/s

Improvement

Baseline TCP Gloo

5.47

Baseline

HAR RDMA all-reduce

25.4

4.6× over baseline

Logits routing

36.2

+42% over HAR

GOMP spincount tuning

40.8

+13% over logits routing; 7.45× over baseline

Across topology comparisons, TP = 4 with HAR and tuning also exceeded TP = 2 single-system shared-memory performance. The optimized TP = 4 result achieved 40.8 output tokens/s, compared with the 32.49 output tokens/s for TP = 2 shared memory, which represents a 26% improvement. This result shows that distributed inference can provide both greater memory capacity and higher throughput when the communication path is topology-aware.

TP = 2 also benefits from HAR

The HAR optimization is not limited to TP = 4. TP = 2 across 2 systems uses a simpler topology, with 1 rank on each Grace system. In this case, there is no local pair to reduce within each system, so HAR becomes a streamlined 2-way RDMA verbs all-reduce. This approach still avoids the TCP socket path and keeps the distributed collective on the low-latency RDMA transport.

The TP = 2 results show the same pattern as TP = 4. Topology-aware communication determines whether adding a second system also improves performance. TCP across 2 systems reached 11.74 output tokens/s. A direct RDMA path increased throughput to 18.68 output tokens/s. HAR reached 34.4 output tokens/s, slightly higher than the single-system TP = 2 shared-memory result of 32.49 output tokens/s.

Topology

Output tokens/s

What it shows

TP = 2 shared memory, single system

32.49

Strong local baseline

TP = 2 TCP, two systems

11.74

Cross-system TCP bottleneck

TP = 2 direct RDMA, two systems

18.68

RDMA helps but does not fully recover local performance

TP = 2 HAR, two systems

34.4

Distributed path exceeds the single-system TP = 2 result

This matters because TP = 2 is often the first distributed configuration that users try when a model or workload no longer fits on one system. HAR makes that step useful. It preserves the benefit of distribution without making cross-system communication the dominant cost.

Scaling to larger models

The same distributed setup also enabled Qwen3-235B-A22B to run with TP = 4 across 2 Grace systems. A single Grace system could not fit the model in memory, while the 2-system configuration reached 4.21 output tokens/s on the 10-prompt ShareGPT benchmark.

The performance analysis above focuses on Qwen2.5-14B-Instruct because it provides a clear way to compare communication paths and measure each optimization step. However, distributed inference is also valuable for a second reason. It enables models that do not fit on a single system.

Qwen3-235B-A22B demonstrates this benefit. A single Grace system ran out of memory, but distributing the model across 2 Grace systems with TP = 4 made inference possible. The 2-system configuration reached 4.21 output tokens/s on the 10-prompt ShareGPT benchmark. The result demonstrates that multi-system Arm Neoverse deployments can support larger models while maintaining practical CPU inference performance.

Model

Single Grace system

Two Grace systems, TP = 4

Result

Qwen3-235B-A22B

Out of memory

4.21 output tokens/s

Distributed inference makes the model viable

This highlights the broader value of topology-aware distributed inference. Communication optimization improves latency and throughput, while memory aggregation across systems enables much larger model deployments than a single system can support.

Upstream contribution

The HAR implementation was prepared as an opt-in contribution to vLLM with PR Link. A CMake cache variable, VLLM_CPU_RDMA_HAR, gates the RDMA path. The default is OFF, which keeps stock CPU builds unchanged and avoids a libibverbs dependency on systems without RDMA.

  • When HAR is enabled, the build links libibverbs, compiles RDMA operations, registers the CPU RDMA collectives.
  • When HAR is disabled, RDMA symbols are compiled out and the CPU path remains identical to the upstream shared-memory-only implementation.
  • The contribution adds hierarchical all-reduce for multi-node tensor parallelism, topology-aware logits routing.

Key takeaways

  • Topology matters: a flat ring all-reduce can make distributed decode wait on repeated cross-system TCP hops.
  • HAR RDMA removes the dominant bottleneck: shared memory handles local communication, while one RDMA exchange handles cross-system traffic.
  • Residual bottlenecks shift: after hidden-state collectives move to RDMA, logits gathering and CPU dispatch overhead become visible.
  • Distributed CPU inference can beat single-system inference: TP = 4 across 2Grace systems reached 40.8 output tokens/s, exceeding TP = 2 shared-memory performance on one system.
  • Scale enables larger models: 2 Grace systems were able to run Qwen3-235B-A22B where a single system ran out of memory.

Conclusion

Distributed inference on Arm Neoverse is not only about adding more systems. It also requires communication algorithms that match the hardware topology. The optimized configuration replaced topology-blind TCP all-reduce with a hierarchical RDMA design, routed logits through the fast path, and kept CPU worker threads ready for latency-sensitive decoding. Together, these changes delivered a measured 7.45× throughput improvement. The same approach also improves smaller distributed topologies such as TP = 2. HAR exceeded the single-system TP = 2 shared-memory result and it increased model capacity. Qwen3-235B-A22B ran across 2 Grace systems at TP = 4, while a single Grace system ran out of memory. This approach provides a practical way to scale CPU-only LLM inference across Arm Neoverse systems while preserving build-time portability for upstream users.


Log in to like this post
Share

Article text

Re-use is only permitted for informational and non-commercial or personal use only.

placeholder