Llama-3.2-3B-Instruct INT4

Cloud CPUAWS Graviton G4

Quantized instruction-following causal language models for efficient text generation, using INT4 weights with dynamic INT8 activations plus INT8 embeddings and LM head. The family targets compact storage and low-memory operation while preserving chat-oriented capabilities, long-context handling, and decoder-only transformer behavior.View on Hugging Face

Accuracyvs base model

MMLU (5-shot)60.35%↓1.06 pp

Tokens / sec

23.42tok/s

Time to First Token

About this version

Original Model:: Llama-3.2-3B-Instruct by Meta — Repository
Optimization/conversion:: Arm-Optimized version for execution on Arm-based platforms.
Converted/optimized by:: Arm
License:: Not specified in the model metadata.

This repository contains a converted or optimized version of the Original Model (the “Optimized Model”). The Original Model has been converted or optimized as described above for execution on Arm-based platforms.

No retraining or fine-tuning of the Original Model was performed as part of the conversion or optimization. The conversion or optimization was not intended to change the Original Model’s behavior or intended use.

Original Model and Documentation

For information about the Original Model, including its development, training data, intended uses, limitations and other relevant information, please refer to the Original Model repository. Information in that repository was provided by the original developer or other third parties and, unless expressly stated otherwise, has not been independently verified by Arm.

Licenses and Third-Party Terms

Use of the Original Model and the Optimized Model is subject to the applicable licenses, usage restrictions and other terms identified above and in the relevant repositories. Publication of the Optimized Model does not grant any rights beyond those provided under the applicable license terms.

You are responsible for reviewing those terms and ensuring that your use of the Original Model and the Optimized Model is permitted.

Purpose of this Release

The Optimized Model is provided as a reference implementation to demonstrate and evaluate execution and performance on Arm-based systems. It is not a production-ready or supported solution.

About this version

Original Model:: Llama-3.2-3B-Instruct by Meta — Repository
Optimization/conversion:: Arm-Optimized version for execution on Arm-based platforms.
Converted/optimized by:: Arm
License:: Not specified in the model metadata.

Original Model and Documentation

Licenses and Third-Party Terms

You are responsible for reviewing those terms and ensuring that your use of the Original Model and the Optimized Model is permitted.

Purpose of this Release

The Optimized Model is provided as a reference implementation to demonstrate and evaluate execution and performance on Arm-based systems. It is not a production-ready or supported solution.

Metric	Resultsvs baseline	Change
End to end latency p50	3674.20ms13176.06ms	−9501.86 ms
End to end latency p90	3707.23ms13494.73ms	−9787.50 ms
End to end latency p99	3764.38ms13742.23ms	−9977.84 ms
Peak memory	2631.86MB12492.45MB	−9860.59 MB
Average memory	2620.54MB12469.00MB	−9848.46 MB
Time to first inference	3639.43ms13121.56ms	−9482.13 ms
Tokens per second	23.42tok/s5.78tok/s	+17.64 tok/s
Time to first token	1108.14ms2795.21ms	−1687.07 ms
MMLU (5-shot)	60.35%61.41%	−1.06%

Configuration	Baseline	Arm Optimized
Dataset	MMLU (test)	MMLU (test)
Sample count	14,042	14,042
Batch size	1	1
Prompt length	128	128
Generation length	60	60
Runs	50	50
Warmup runs	10	10

Llama-3.2-3B-Instruct INT4

Llama-3.2-3B-Instruct INT4

Get Started with this Model

Text Generator (ONNX Runtime GenAI)

Model Evaluation

Configuration

Comparability