Qwen3-1.7B-Base INT4 Dynamic

Mobile CPUvivo X300SME2

This is a 4-bit quantized version of Qwen/Qwen3-1.7B-Base optimized for edge deployment on ARM devices using ONNX Runtime. The model was quantized using GPTQ (INT4 weights, group size 32, asymmetric, with an INT8 LM head and tied INT8 token embeddings) and exported to the ONNX format with a fused key-value-cache attention graph for efficient autoregressive inference on ARM CPUs (Android smartphones, AWS Graviton). Activations are quantized to INT8 per token dynamically at kernel time (W4A8), so no activation calibration is required.View on Hugging Face

Accuracyvs base model

MMLU (5-shot)60.14%↓2.54 pp

Tokens / sec

21.66 tok/s

Time to First Token

About this version

Original Model:: Qwen3-1.7B-Base by Alibaba Cloud — Repository
Optimization/conversion:: Arm-Optimized version for execution on Arm-based platforms.
Converted/optimized by:: Arm
License:: The Original Model and the Optimized Model are subject to Apache-2.0.

This repository contains a converted or optimized version of the Original Model (the “Optimized Model”). The Original Model has been converted or optimized as described above for execution on Arm-based platforms.

No retraining or fine-tuning of the Original Model was performed as part of the conversion or optimization. The conversion or optimization was not intended to change the Original Model’s behavior or intended use.

Original Model and Documentation

For information about the Original Model, including its development, training data, intended uses, limitations and other relevant information, please refer to the Original Model repository. Information in that repository was provided by the original developer or other third parties and, unless expressly stated otherwise, has not been independently verified by Arm.

Licenses and Third-Party Terms

Use of the Original Model and the Optimized Model is subject to the applicable licenses, usage restrictions and other terms identified above and in the relevant repositories. Publication of the Optimized Model does not grant any rights beyond those provided under the applicable license terms.

You are responsible for reviewing those terms and ensuring that your use of the Original Model and the Optimized Model is permitted.

About this version

Original Model:: Qwen3-1.7B-Base by Alibaba Cloud — Repository
Optimization/conversion:: Arm-Optimized version for execution on Arm-based platforms.
Converted/optimized by:: Arm
License:: The Original Model and the Optimized Model are subject to Apache-2.0.

Original Model and Documentation

Licenses and Third-Party Terms

You are responsible for reviewing those terms and ensuring that your use of the Original Model and the Optimized Model is permitted.

Metric	Resultsvs baseline	Change
End to end latency p50	6858.63ms27164.53ms	−20305.89 ms
End to end latency p90	7151.49ms27344.85ms	−20193.36 ms
End to end latency p99	7428.93ms27498.80ms	−20069.87 ms
Peak memory	1941.80MB6843.79MB	−4901.99 MB
Average memory	1863.09MB6761.51MB	−4898.42 MB
Time to first inference	6269.52ms26819.56ms	−20550.04 ms
Tokens per second	21.66tok/s4.92tok/s	+16.73 tok/s
Time to first token	937.14ms1253.34ms	−316.20 ms
MMLU (5-shot)	60.14%62.68%	−2.54%

Configuration	Baseline	Arm Optimized
Dataset	MMLU (test)	MMLU (test)
Sample count	5,000	5,000
Batch size	1	1
Prompt length	127	127
Generation length	128	128
Runs	20	20
Warmup runs	5	5

Qwen3-1.7B-Base INT4 Dynamic

Qwen3-1.7B-Base INT4 Dynamic

Get Started with this Model

Text Generator (ONNX Runtime GenAI)

Model Evaluation

Configuration

Purpose of this Release

Comparability