Phi-3.5 Mini INT8

Cloud CPUAWS Graviton G4

An INT8-quantized Phi-3.5 Mini language model for text generation. Exported as an ExecuTorch program using the XNNPACK backend for inference on Arm CPUs.View on Hugging Face

Accuracyvs base model

HellaSwag66%↓1 pp

Tokens / sec

7.69tok/s

Latency

130.06ms

Peak Memory

5755.88MB

Get Started with this Model

Learning path

Run optimized LLMs from the Arm AI Portal on Arm Neoverse cloud instances

Download and run Arm-optimized text-generation models from the Arm AI Portal on Arm Neoverse cloud instances.

Learning path

Deploy an AI Agent on Arm with llama.cpp and llama-cpp-agent using KleidiAI

Learn how to build and deploy an AI agent application on Arm servers using llama.cpp and llama-cpp-agent with KleidiAI optimization for efficient LLM inference and function calling.

Learning path

Run vLLM inference with quantized models and benchmark on Arm servers

Learn how to run Llama 3.1 8B and Whisper inference with BF16 and W8A8 quantized models on Arm servers, then compare throughput and accuracy with vLLM benchmark tools.

Learning path

Deploy Arcee AFM-4.5B on Arm-based Google Cloud Axion with Llama.cpp

Learn how to build llama.cpp, quantize the Arcee AFM-4.5B model, and run optimized inference on Google Cloud Axion instances with perplexity-based quality evaluation.

Model Evaluation

Base model: Phi-3.5-mini-instructTested on: AWS Graviton G4 (arm64)

Evaluation Conditions

Text Generation · AWS Graviton G4 · INT8 · C4

Metric	Resultsvs baseline	Change
End to end latency p50	130.06ms806.30ms	−676.23 ms
Peak memory

About this version

Original Model:: Phi-3.5-mini-instruct by Microsoft — Repository
Optimization/conversion:: Arm-Optimized version for execution on Arm-based platforms.
Converted/optimized by:: Nota AI
License:: The Original Model and the Optimized Model are subject to MIT.

This repository contains a converted or optimized version of the Original Model (the “Optimized Model”). The Original Model has been converted or optimized as described above for execution on Arm-based platforms.

No retraining or fine-tuning of the Original Model was performed as part of the conversion or optimization. The conversion or optimization was not intended to change the Original Model’s behavior or intended use.

Original Model and Documentation

For information about the Original Model, including its development, training data, intended uses, limitations and other relevant information, please refer to the Original Model repository. Information in that repository was provided by the original developer or other third parties and, unless expressly stated otherwise, has not been independently verified by Arm.

Licenses and Third-Party Terms

Use of the Original Model and the Optimized Model is subject to the applicable licenses, usage restrictions and other terms identified above and in the relevant repositories. Publication of the Optimized Model does not grant any rights beyond those provided under the applicable license terms.

You are responsible for reviewing those terms and ensuring that your use of the Original Model and the Optimized Model is permitted.

Phi-3.5 Mini INT8

Cloud CPUAWS Graviton G4

An INT8-quantized Phi-3.5 Mini language model for text generation. Exported as an ExecuTorch program using the XNNPACK backend for inference on Arm CPUs.View on Hugging Face

Accuracyvs base model

HellaSwag66%↓1 pp

Tokens / sec

7.69tok/s

Latency

130.06ms

Peak Memory

5755.88MB

Get Started with this Model

Learning path

Run optimized LLMs from the Arm AI Portal on Arm Neoverse cloud instances

Download and run Arm-optimized text-generation models from the Arm AI Portal on Arm Neoverse cloud instances.

Learning path

Deploy an AI Agent on Arm with llama.cpp and llama-cpp-agent using KleidiAI

Learn how to build and deploy an AI agent application on Arm servers using llama.cpp and llama-cpp-agent with KleidiAI optimization for efficient LLM inference and function calling.

Learning path

Run vLLM inference with quantized models and benchmark on Arm servers

Learn how to run Llama 3.1 8B and Whisper inference with BF16 and W8A8 quantized models on Arm servers, then compare throughput and accuracy with vLLM benchmark tools.

Learning path

Deploy Arcee AFM-4.5B on Arm-based Google Cloud Axion with Llama.cpp

Learn how to build llama.cpp, quantize the Arcee AFM-4.5B model, and run optimized inference on Google Cloud Axion instances with perplexity-based quality evaluation.

Model Evaluation

Base model: Phi-3.5-mini-instructTested on: AWS Graviton G4 (arm64)

Evaluation Conditions

Text Generation · AWS Graviton G4 · INT8 · C4

Metric	Resultsvs baseline	Change
End to end latency p50	130.06ms806.30ms	−676.23 ms
Peak memory

About this version

Original Model:: Phi-3.5-mini-instruct by Microsoft — Repository
Optimization/conversion:: Arm-Optimized version for execution on Arm-based platforms.
Converted/optimized by:: Nota AI
License:: The Original Model and the Optimized Model are subject to MIT.

Original Model and Documentation

Licenses and Third-Party Terms

You are responsible for reviewing those terms and ensuring that your use of the Original Model and the Optimized Model is permitted.

Configuration	Baseline	Arm Optimized
Dataset	C4	C4
Sample count	200	200
Batch size	1	1
Prompt length	512	512

Phi-3.5 Mini INT8

Get Started with this Model

Run optimized LLMs from the Arm AI Portal on Arm Neoverse cloud instances

Deploy an AI Agent on Arm with llama.cpp and llama-cpp-agent using KleidiAI

Run vLLM inference with quantized models and benchmark on Arm servers

Deploy Arcee AFM-4.5B on Arm-based Google Cloud Axion with Llama.cpp

Model Evaluation

Phi-3.5 Mini INT8

Get Started with this Model

Run optimized LLMs from the Arm AI Portal on Arm Neoverse cloud instances

Deploy an AI Agent on Arm with llama.cpp and llama-cpp-agent using KleidiAI

Run vLLM inference with quantized models and benchmark on Arm servers

Deploy Arcee AFM-4.5B on Arm-based Google Cloud Axion with Llama.cpp

Model Evaluation

Configuration

Comparability

Purpose of this Release