Llama-3.2-1B-Base INT4

Cloud CPUAWS Graviton G4

Compact decoder-only causal language models optimized for efficient text generation on ARM CPUs, using ONNX Runtime GenAI with INT4 weights, dynamic INT8 activations, and INT8 embeddings. Designed for low-memory, low-latency inference on Graviton and edge-style aarch64 systems, they support long-context continuation and in-context learning workflows while preserving the behavior of a base, non-instruction-tuned pretrained model.View on Hugging Face

Accuracyvs base model

MMLU (5-shot)30.04%↓2.5 pp

Tokens / sec

54.99 tok/s

Time to First Token

About this version

Original Model:: Llama-3.2-1B by Meta — Repository
Optimization/conversion:: Arm-Optimized version for execution on Arm-based platforms.
Converted/optimized by:: Arm
License:: Not specified in the model metadata.

This repository contains a converted or optimized version of the Original Model (the “Optimized Model”). The Original Model has been converted or optimized as described above for execution on Arm-based platforms.

No retraining or fine-tuning of the Original Model was performed as part of the conversion or optimization. The conversion or optimization was not intended to change the Original Model’s behavior or intended use.

Original Model and Documentation

For information about the Original Model, including its development, training data, intended uses, limitations and other relevant information, please refer to the Original Model repository. Information in that repository was provided by the original developer or other third parties and, unless expressly stated otherwise, has not been independently verified by Arm.

Licenses and Third-Party Terms

Use of the Original Model and the Optimized Model is subject to the applicable licenses, usage restrictions and other terms identified above and in the relevant repositories. Publication of the Optimized Model does not grant any rights beyond those provided under the applicable license terms.

You are responsible for reviewing those terms and ensuring that your use of the Original Model and the Optimized Model is permitted.

Purpose of this Release

The Optimized Model is provided as a reference implementation to demonstrate and evaluate execution and performance on Arm-based systems. It is not a production-ready or supported solution.

About this version

Original Model:: Llama-3.2-1B by Meta — Repository
Optimization/conversion:: Arm-Optimized version for execution on Arm-based platforms.
Converted/optimized by:: Arm
License:: Not specified in the model metadata.

Original Model and Documentation

Licenses and Third-Party Terms

You are responsible for reviewing those terms and ensuring that your use of the Original Model and the Optimized Model is permitted.

Purpose of this Release

The Optimized Model is provided as a reference implementation to demonstrate and evaluate execution and performance on Arm-based systems. It is not a production-ready or supported solution.

Metric	Resultsvs baseline	Change
End to end latency p50	1544.99ms5220.67ms	−3675.67 ms
End to end latency p90	1591.85ms5250.71ms	−3658.85 ms
End to end latency p99	1637.17ms5442.96ms	−3805.79 ms
Peak memory	1243.42MB5928.29MB	−4684.87 MB
Average memory	1173.57MB5920.31MB	−4746.74 MB
Time to first inference	1592.75ms5183.99ms	−3591.24 ms
Tokens per second	54.99tok/s14.50tok/s	+40.49 tok/s
Time to first token	449.08ms1076.85ms	−627.77 ms
MMLU (5-shot)	30.04%32.54%	−2.50%

Configuration	Baseline	Arm Optimized
Dataset	MMLU (test)	MMLU (test)
Sample count	5,000	5,000
Batch size	1	1
Prompt length	128	128
Generation length	60	60
Runs	50	50
Warmup runs	10	10

Llama-3.2-1B-Base INT4

Llama-3.2-1B-Base INT4

Get Started with this Model

Text Generator (ONNX Runtime GenAI)

Model Evaluation

Configuration

Comparability