Topo llama.cpp WebUI Chat

LLM chat application with Arm CPU inference provided by llama.cpp. This project demonstrates running large language models on CPU with inference provided by the llama.cpp server. The upstream Linux Arm64 image includes architecture-specific CPU backend variants for Armv8.0 baseline, Armv8.2 dot product/FP16/SVE, Armv8.6 int8 matrix multiply/SVE2, and Armv9.2 SME-capable CPUs. The stack includes: - llama.cpp - Quantized SmolLM2 135M default model loaded through llama.cpp - Built-in web chat interface - No GPU required - pure CPU inference Perfect for demos and testing! The default SmolLM2-135M-Instruct model gives the project a small ready-to-use model reference by default. Ideal for testing LLM workloads on Arm hardware without GPU dependencies while avoiding a source build during Topo Project deployment.

View on GitHub

Topo llama.cpp web UI

This is a Topo Project and follows the Topo Project Specification.

Complete LLM chat application with Arm CPU inference provided by llama.cpp.

Overview

This project demonstrates running large language models on CPU with inference provided by the llama.cpp server and a configurable GGUF model.

The upstream Linux Arm64 llama.cpp server image is built with architecture-specific CPU backend variants enabled. llama.cpp can then load a backend variant that matches the Arm CPU features available at runtime.

The stack includes:

llama.cpp
Quantized SmolLM2 135M model bundled in the image
Built-in web chat interface
No GPU required - pure CPU inference

Arm CPU Optimizations

The prebuilt ghcr.io/ggml-org/llama.cpp:server image currently enables llama.cpp CPU backend variants for Linux Arm. This project pins the image digest so these exact variants remain stable.

The Linux Arm backend variants and feature combinations are defined in upstream ggml/src/CMakeLists.txt, and the CPU server image is built with GGML_CPU_ALL_VARIANTS=ON in upstream .devops/cpu.Dockerfile.

Backend variant	Arm features included
`armv8.0_1`	Baseline Armv8.0
`armv8.2_1`	Dot product
`armv8.2_2`	Dot product, FP16 vector arithmetic
`armv8.2_3`	Dot product, FP16 vector arithmetic, SVE
`armv8.6_1`	Dot product, FP16 vector arithmetic, SVE, int8 matrix multiply
`armv8.6_2`	Dot product, FP16 vector arithmetic, SVE, int8 matrix multiply, SVE2
`armv9.2_1`	Dot product, FP16 vector arithmetic, SVE, int8 matrix multiply, SME
`armv9.2_2`	Dot product, FP16 vector arithmetic, SVE, int8 matrix multiply, SVE2, SME

Prerequisites

Arm Hardware: An Arm system (physical or virtual).
Docker: For container orchestration with Topo
LLM Model: Optional when overriding the bundled default; provide a supported single-file GGUF model (e.g., Llama 3.1, Mistral, etc.)

Note: MODEL must point to a supported single-file .gguf model artifact. Use a Hugging Face repo ID to auto-select a CPU-friendly quantization (preferring Q4_K_M), a Hugging Face repo plus quantization suffix as <repo>:<quantization>, or a direct .gguf URL. Sharded GGUFs and multimodal projector files (mmproj) are rejected with a clear error because this project only supports single-file text model GGUFs today. Not all model repos include GGUF quantizations — look for repos with -GGUF in the name. The selected model is downloaded when the service starts and cached inside the container.

Build-Time Parameters

Parameter	Description	Default
`MODEL`	Hugging Face GGUF repo, `<repo>:<quantization>`, or direct `.gguf` URL	`unsloth/SmolLM2-135M-Instruct-GGUF`
`MODEL_ENDPOINT`	Hugging Face API-compatible endpoint for repository model downloads	`https://huggingface.co`

Usage

The easiest way to deploy is using topo. Download and install topo from here

Clone the project:

topo clone https://github.com/Arm-Examples/topo-llama-web-ui.git

Build and Deploy the project:

cd topo-lama-web-ui
topo deploy --target <ip-address-of-target>

Common Model Selection Examples

Use a different model:

topo deploy --target <ip-address-of-target> \
  --arg MODEL=bartowski/Qwen_Qwen3.5-0.8B-GGUF

Select an exact quantization:

topo deploy --target <ip-address-of-target> \
  --arg MODEL=unsloth/SmolLM2-135M-Instruct-GGUF:Q4_K_M

Access the Chat Interface

Open your browser to http://<ip-address-of-target>:8080 to start chatting!

Run this project

Topo project

Use Topo to configure and deploy this container-based application to Arm-based Linux devices over SSH.

Configure project

Model

Deploy project

Install the Arm Topo CLI to manage your model deployments through the terminal

1. Install Topo CLI

2. Run the following command to clone and configure the project.

Clone and Configure Project

topo clone https://github.com/Arm-Examples/topo-llama-web-ui.git#9ef51579218733d97707f23c68189b2c4042e129 MODEL_ENDPOINT='https://ml-models.devplatform.arm.com/' MODEL='Arm/ernie-4-5-0-3b-pt-q4-k-m-llamacpp-raspberrypi5'

Read the Topo docs for installation, troubleshooting and advanced usage.

Loading example…

About the models on this Arm AI Portal

The Arm AI Portal (the “Portal”) provides details of AI models developed, optimized or tested for use on Arm-based platforms. Models listed on the Portal may have been developed by Arm or by third parties. Unless expressly stated otherwise, Arm did not develop or train the underlying third-party model and is not responsible for its original intended behavior or use.

Information accompanies each model listing and describes the model’s provenance, together with relevant information about the original third-party model, its developer and repository, where applicable. Inclusion on the Portal of a model developed by a third party or based on a third-party model does not constitute Arm’s endorsement or certification of that third-party model or any third-party work relating to it.

Except as expressly stated in the information accompanying a specific model, Arm makes no representation as to the accuracy, safety, security, non-infringement, legal compliance, suitability for production use or fitness for any particular purpose of any model, converted or optimized version, or related outputs.

Inclusion of a model on the Portal does not itself grant you any rights to use that model. Use of each model is subject to the applicable license terms, usage restrictions and documentation. You are responsible for reviewing that information and independently evaluating the model, its outputs and its suitability for your intended use, including compliance with applicable legal, regulatory, safety and security requirements. Any use of or reliance on a model, its outputs or related materials is at your own risk. Where the Portal links to repositories or other materials controlled by third parties, those repositories and materials are outside Arm’s control and may change without notice.

Use of the Portal is subject to the Arm Website Terms and Conditions of Use.