Compressing an 11B VLM to 2.7-bit weights for Arm mobile CPUs

As multimodal AI moves from the cloud to personal devices, developers face a fundamental constraint. Capable vision-language models require far more memory and bandwidth than most mobile systems can provide.

Mobile phones are a natural place to run multimodal AI. They already have cameras, microphones, sensors, and personal context on a single device. Running models locally also reduces reliance on cloud services, helping keep personal data on-device while reducing latency and improving responsiveness.

The challenge is fitting a model with billions of parameters within the memory and power constraints of a mobile device. Llama 3.2 Vision 11B needs 21.3 GB for its bfloat16 weights alone. Even with 4-bit quantization, the weights still require more than 5 GB. Reducing that footprint introduces two significant challenges. Developers need to preserve useful model accuracy and ensure that the target hardware can decode the compressed weights efficiently.

Llama-Mobile, developed by Graphcore Research in collaboration with Arm, introduces two key ideas:

• S3D8, a novel 2.7-bit weight format for efficient decoding on Arm CPUs
• Quantization-aware training that does not require the model's original training data

Applied to Llama 3.2 Vision 11B, the approach reduces the weight footprint by more than 80%, from 21.3 GB to 3.7 GB. It also quantizes activations to INT8 while retaining an average visual question-answering score of 66.1%, compared with 74.4% for the original bfloat16 model.

Designing a low-bit format around Arm SIMD

S3D8 is a vector quantization format that stores 3 weights in a single byte, resulting in an average of 2.7 bits per parameter. Each byte contains a shared 5-bit centroid index and 1 sign bit for each weight. The index selects one of 32 learned centroids, stored as a 32x3 table of positive INT8 values. A bfloat16 scale for each output channel preserves the weight range.

S3D8 is a compact storage format, not a new arithmetic format. During inference, Arm Advanced SIMD (Single Instruction Multiple Data), also known as Arm Neon, decodes multiple S3D8 values in parallel. It uses table-lookup instructions to convert the values into channel-scaled INT8. The CPU processes those weights using existing INT8 matrix multiplication instructions.

Separating storage from computation reduces memory traffic without requiring dedicated 2.7-bit arithmetic hardware.

Matrix multiplication throughput on Google Pixel 8a

Autoregressive token generation processes one token at a time, requiring the model weights to be read for each new token. This makes generation memory-bandwidth-bound, which is where the more compact S3D8 representation provides its clearest advantage over INT8, as shown in Table 1.

Text prefill and vision encoding process many tokens or patches at once. These operations are compute-bound: S3D8 must be decoded to INT8, so it performs similarly to INT8.

Pixel 8a workload	bfloat16 GMAC/s	INT8 GMAC/s	S3D8 GMAC/s
Token generation	13.6	26.5	33.8
Text prefill	76.3	201.1	197.7
Vision encoding	116.5	206.1	211.8

Table 1: Throughput for selected matrix multiplication operations in Llama 3.2 Vision 11B on five Pixel 8a CPU cores. 

For the token-generation workload, S3D8 reaches 33.8 GMAC/s, approximately 28% higher throughput than INT8 in this test. For the more compute-intensive prefill and vision workloads, S3D8 performance remains close to INT8.  

Using a custom C++ runtime, the S3D8 quantized model generates a median 3.8 tokens per second on the Pixel 8a, excluding model loading and prefill. 

Recovering accuracy with quantization-aware distillation

Llama-Mobile uses quantization-aware training (QAT) to recover much of the accuracy lost from quantizing weights to less than 3 bits. Specifically, it uses quantization-aware distillation (QAD), where a quantized student learns from the original bfloat16 teacher model.

During fine-tuning, the student uses quantized weights and INT8 activations in the forward pass while learning to match the teacher’s next-token distribution. This approach exposes the model to quantization effects during training rather than applying compression only after training is complete.

A practical challenge with QAT is obtaining suitable fine-tuning data. The datasets used to train the original model are often unavailable. Llama-Mobile addresses this challenge by creating a synthetic dataset. The teacher model generates responses for images sampled from ImageNet, and the resulting image-text examples are then used to fine-tune the quantized student.

This distillation approach avoids dependence on the original model-training corpus, which may be proprietary, inaccessible, or too costly to reproduce.

Figure 1 compares QAT with direct casting and GPTQ. Direct casting maps each weight to its nearest quantized value without calibration or training. GPTQ is a post-training quantization method that uses calibration data and second-order information to reduce quantization error.

Figure 1: Average performance across VQAv2, ChartQA, DocVQA, and AI2D across bit widths. Direct cast and QAT use INT8 activations, while GPTQ is weight-only with bfloat16 activations.

Direct-cast task performance degrades sharply below 4 bits per parameter, falling below 40% at around 3.5 bits. GPTQ retains more accuracy across the sweep, but QAT performs best under extreme compression.

At approximately 2.7 bits per parameter, S3D8 with GPTQ scores 34%, compared with 1.8% for rate-matched INT with GPTQ. Both results use the same quantization method, so the comparison isolates the benefit of the S3D8 representation.

Applying QAT to S3D8 increases the score to 66.1%, compared to 74.4% for the original bfloat16 model. Quantization-aware training recovers most of the accuracy that would otherwise be lost at this bit width.

What Llama-Mobile shows for edge inference on Arm CPUs

Llama-Mobile shows how an 11B parameter VLM can be compressed for an Arm mobile CPU. S3D8 reduces the model weights from 21.3 GB to 3.7 GB. With QAT, the model retains an average task score of 66.1%, compared with 74.4% for the original model.

For Arm CPU developers, the key takeaway is the value of flexible weight formats that separate storage from computation. S3D8 stores weights compactly at 2.7 bits per parameter, then decodes them into INT8 values for use with existing matrix multiplication instructions. This approach reduces memory traffic without requiring dedicated low-bit arithmetic support.

S3D8 also demonstrates the benefit of designing a quantization format around the target hardware. The layout works efficiently with Arm Advanced SIMD table-lookup instructions, enabling fast parallel decoding during inference. The result is higher throughput during memory-bandwidth-bound token generation, while maintaining performance similar to INT8 for compute-bound workloads such as text prefill.

The broader lesson extends beyond one model or quantization format. As edge AI systems become more capable, practical deployment will increasingly depend on co-design. This includes model architecture, numerical representation, software kernels, and processor capabilities.

This study covers just one VLM, but it demonstrates a practical approach to multimodal inference at the edge. Future work can explore how the same principles apply to different model sizes and architectures, device classes, Arm CPU generations, and multimodal workloads. This work could expand the range of AI experiences that can run privately and responsively on-device.

Ready to go deeper? Explore how Llama-Mobile was built, benchmarked, and trained, including the full technical methodology and implementation details:

Explore the Llama-Mobile paper  Read the Graphcore Research blog