Accelerating DiskANN vector search with Arm Neon
Explore how Arm and Microsoft optimize DiskANN using Arm Neon to deliver 8x throughput in tests on Azure Cobalt 200.
By Patrick Fox

This blog post is co-authored by Patrick Fox, Senior Engineer, Arm and Mark Hildebrand, Principal Software Engineer, Microsoft.
AI applications need more than model inference. Retrieval-augmented generation (RAG) and agentic workflows that use retrieved context also depend on finding relevant information efficiently. Vector search supports this retrieval process, making it an important part of the CPU software stack to optimize for AI applications.
Microsoft’s DiskANN enables developers to integrate fast, accurate and scalable vector search into databases and other applications. Developed through Microsoft’s DiskANN research project, it uses graph-based approximate nearest-neighbor algorithms to search large collections of vectors. This article focuses on its open-source Rust library and the distance calculations used to identify vectors most similar to a query.
Arm and Microsoft engineers have collaborated to optimize these calculations for Arm CPUs using Arm Neon. The contributions are merged into the upstream DiskANN project, making the implementation available to developers through the open-source library.
On Azure Cobalt 200, DiskANN v0.61.0 achieves 8x the geometric-mean in-memory search throughput of v0.56.0 across our tested 2-bit and 4-bit workloads.
We explain how the new Neon kernels process packed values efficiently, then examine the release-level benchmark results and their limitations.
Optimizing DiskANN Distance Calculations with Arm Neon
During graph-based search, DiskANN repeatedly evaluates candidate vectors against a query to determine which candidates to explore. Depending on the search configuration, these comparisons use inner product, euclidean distance (squared-L2), or cosine distance. The kernels discussed here accelerate inner-product and squared-L2 calculations for packed unsigned integer vectors.
Low-bit quantization packs multiple vector elements into each byte. For example, a 768-dimensional FP32 vector contains 3,072 bytes of element data, while a packed 4-bit representation uses 384 bytes. The challenge is extracting packed values and performing arithmetic on these efficiently.
Arm Neon provides single instruction, multiple data (SIMD) operations that process multiple elements of the input vectors in parallel. For packed low-bit integer data, we use masks and shifts to unpack the values, then UDOT (Unsigned Dot Product) instructions to multiply and accumulate their products. UDOT is available on AArch64 processors that implement the Advanced SIMD dot-product extension. In its 128-bit form, UDOT multiplies 16 pairs of unsigned 8-bit values, adds groups of four products, and accumulates the results into four 32-bit lanes. Arm Neoverse V2 and Neoverse V3 can perform up to four UDOT instructions in parallel, offering high throughput per cycle.
The 4-bit x 4-bit Inner Product Kernel
Let’s take a closer look at how we use Arm Neon to accelerate the inner product when DiskANN uses vectors with 4-bits for each element.
The inner product multiplies vector elements and sums the results:

In v0.56.0, the AArch64 4-bit inner-product path used a generic implementation. The newer release adds a dedicated Neon implementation that unpacks and processes quantized values using UDOT.
A 128-bit Neon register can hold 16 packed bytes, in this instance each byte representing two 4-bit values. The lower four bits hold the first element, while the upper four bits hold the second.
To compute the inner product, we use masks and shifts to extract the lower and upper 4-bit values into separate byte-sized lanes.
// Extract the lower 4-bit values
lower_x = x_vec & 0x0F;
lower_y = y_vec & 0x0F;
// Extract the upper 4-bit values
upper_x = (x_vec >> 4) & 0x0F;
upper_y = (y_vec >> 4) & 0x0F;
Each operation here processes all 16 bytes in parallel. We now have two pairs of Neon registers containing the original 4-bit values in byte-sized lanes, ready for the inner product calculation.
Each UDOT instruction multiplies 16 pairs of unsigned 8-bit values. It sums groups of four products and adds them to four 32-bit accumulator lanes.
We apply one UDOT instruction to the lower 4-bit values and another to the upper values. Together, they process 32 elements from each input vector per iteration.
The kernel uses a separate accumulator for each stream.
s0 = s0.dot_simd(lower_x, lower_y);
s1 = s1.dot_simd(upper_x, upper_y);
Using separate accumulators reduces the dependency chain between successive UDOT instructions. This exposes instruction-level parallelism, allowing the processor to overlap independent accumulation operations.
After processing the vector, we combine the accumulators and sum their four 32-bit lanes to produce the inner product.

The 2-bit kernel follows the same approach. Each packed byte contains four 2-bit values, which we extract using shifts and a 0x03 mask. The kernel uses four UDOT instructions and four independent accumulators to process 64 elements per full SIMD iteration.
The 4-bit x 4-bit Squared L2 Kernel
We use a similar approach to accelerate euclidean distance (squared-L2) calculations for vectors with 4-bit elements.

We start by unpacking the 4-bit values into byte-sized lanes using the same masks and shifts described earlier.
Instead of multiplying corresponding elements directly, we first calculate their unsigned absolute differences. The Rust intrinsic vabdq_u8 performs this operation across all 16 byte-sized lanes using the AArch64 UABD instruction.
We then use UDOT with the difference vector as both operands. This squares each difference and accumulates the results into four 32-bit lanes. Squaring the absolute difference gives the same results as squaring the signed difference, while keeping the operands suitable for unsigned dot-product instructions.
We repeat these operations for the upper 4-bit values, using a second independent accumulator. After unpacking, the arithmetic stage processes 32 elements per iteration using two UABD and two UDOT instructions. Finally, we combine the accumulators and sum their lanes to produce the squared-L2 distance.

The 2-bit squared-L2 kernel follows the same approach, extracting four 2-bit streams using masks and shifts. Each full SIMD iteration processes 64 elements using four UABD and four UDOT instructions for the distance arithmetic.
Benchmarking methodology and results
The new kernel implementations mentioned above accelerate individual vector comparisons. To evaluate search-level performance, we benchmark in-memory search using diskann-benchmark, comparing DiskANN v0.56.0 with v0.61.0 on Azure Cobalt 200. We also compare Cobalt 100 and Cobalt 200 with both running v0.61.0. These are search-level, release-to-release measurements rather than isolated kernel benchmarks. Other changes between v0.56.0 and v0.61.0 may also contribute to the measured speedups.
The workload contains one million Wikipedia embeddings with 768 dimensions, generated using the Cohere multilingual-22-12 model. We use 2-bit and 4-bit scalar quantization and search 5,000 queries, requesting ten neighbors per query. Graph construction and quantizer training precede the search measurements and are excluded from search QPS.
The graph-build configuration uses a maximum degree of 32, a build search-list size of 50, and alpha=1.2. Search runs use 16 or 32 threads and SearchL values from 10 to 80 in increments of ten. SearchL controls search breadth; it is distinct from the ten neighbors requested.
For each configuration, we report the arithmetic mean of ten recorded QPS measurements. We calculate speedups from matched configuration means, then take the geometric mean of those ratios.
Cobalt 200 and Cobalt 100 refer to Azure VMs D32plds_v7 and D32plds_v6, respectively, both running Windows on Arm. We build DiskANN using Rust 1.97.1 with RUSTFLAGS="-C target-feature=+neon,+dotprod".
For example, with 16 threads and SearchL=40, DiskANN throughput increases from 13,509 QPS in v0.56.0 to 87,036 QPS in v0.61.0, a 6.4x improvement. SearchL controls the size of the candidate list during search. Recall, which measures the percentage of true nearest neighbours retrieved, also increases from 84.58% to 85%, demonstrating improved throughput while maintaining search accuracy.
The figures below reports geometric means of QPS performance across the tested configurations.

DiskANN v0.61.0 achieves 8.0x QPS performance on average over x0.56.0 in our tested configuration. Similarly Cobalt 200 compared with Cobalt 100 gives the following:

Conclusion
Packed low-bit vectors need efficient unpacking as well as fast arithmetic. These Neon kernels combine masks and shifts with unsigned dot products, using independent accumulators to expose parallel work. Across our tested low-bit workloads on Azure Cobalt 200, DiskANN v0.61.0 achieves 8.0× the geometric-mean search throughput of v0.56.0.
The upstream work also extends beyond the examples shown in this blog, adding mixed-precision and FP16 inner-product kernels, 8-bit squared-L2 kernels, and AArch64 software prefetching.
Acknowledgements
We would like to thank Mark Hildebrand at Microsoft for his collaboration and support throughout this work. We also thank Arm engineers Utsav Goel and Mustafa Idris for their contributions.
These optimizations reflect the value of collaboration between Arm and Microsoft in improving the performance of open-source software. We look forward to building on these contributions and continuing our work on DiskANN.
By Patrick Fox
Re-use is only permitted for informational and non-commercial or personal use only.
