Llama-3.1-8B-Base INT4

Cloud CPUAWS Graviton G4

A quantized 8B-parameter base causal language model for text completion, using INT4 weights with dynamic INT8 activations plus INT8 embeddings and LM head. It is a decoder-only Transformer with grouped-query attention, a large native context, and no chat or instruction tuning, making it suited to prompt completion and few-shot pattern-following tasks.View on Hugging Face

Accuracyvs base model

MMLU (5-shot)65.01%↓1.08 pp

Tokens / sec

11.57tok/s

Time to First Token

About this version

Original Model:: Llama-3.1-8B by Meta — Repository
Optimization/conversion:: Arm-Optimized version for execution on Arm-based platforms.
Converted/optimized by:: Arm
License:: Not specified in the model metadata.

This repository contains a converted or optimized version of the Original Model (the “Optimized Model”). The Original Model has been converted or optimized as described above for execution on Arm-based platforms.

No retraining or fine-tuning of the Original Model was performed as part of the conversion or optimization. The conversion or optimization was not intended to change the Original Model’s behavior or intended use.

Original Model and Documentation

For information about the Original Model, including its development, training data, intended uses, limitations and other relevant information, please refer to the Original Model repository. Information in that repository was provided by the original developer or other third parties and, unless expressly stated otherwise, has not been independently verified by Arm.

Licenses and Third-Party Terms

Use of the Original Model and the Optimized Model is subject to the applicable licenses, usage restrictions and other terms identified above and in the relevant repositories. Publication of the Optimized Model does not grant any rights beyond those provided under the applicable license terms.

You are responsible for reviewing those terms and ensuring that your use of the Original Model and the Optimized Model is permitted.

Purpose of this Release

The Optimized Model is provided as a reference implementation to demonstrate and evaluate execution and performance on Arm-based systems. It is not a production-ready or supported solution.

About this version

Original Model:: Llama-3.1-8B by Meta — Repository
Optimization/conversion:: Arm-Optimized version for execution on Arm-based platforms.
Converted/optimized by:: Arm
License:: Not specified in the model metadata.

Original Model and Documentation

Licenses and Third-Party Terms

You are responsible for reviewing those terms and ensuring that your use of the Original Model and the Optimized Model is permitted.

Purpose of this Release

The Optimized Model is provided as a reference implementation to demonstrate and evaluate execution and performance on Arm-based systems. It is not a production-ready or supported solution.

Metric	Resultsvs baseline	Change
End to end latency p50	7640.80ms28947.46ms	−21306.66 ms
End to end latency p90	7664.59ms29060.25ms	−21395.66 ms
End to end latency p99	7707.41ms29158.13ms	−21450.72 ms
Peak memory	5634.72MB28891.88MB	−23257.16 MB
Average memory	5612.54MB28891.64MB	−23279.10 MB
Time to first inference	7226.02ms28410.24ms	−21184.22 ms
Tokens per second	11.57tok/s2.67tok/s	+8.90 tok/s
Time to first token	2443.74ms6423.90ms	−3980.16 ms
MMLU (5-shot)	65.01%66.09%	−1.08%

Configuration	Baseline	Arm Optimized
Dataset	MMLU (test)	MMLU (test)
Sample count	14,042	14,042
Batch size	1	1
Prompt length	128	128
Generation length	60	60
Runs	50	50
Warmup runs	10	10

Llama-3.1-8B-Base INT4

Llama-3.1-8B-Base INT4

Get Started with this Model

Text Generator (ONNX Runtime GenAI)

Model Evaluation

Configuration

Comparability