Run Local LLMs Efficiently with Llama.CPP on Arm
This page is your technical guide to running open source large language models (LLMs) like LLaMA, Mistral, and Gemma using Llama.CPP on Arm-based CPUs. Learn how to install, optimize, and deploy local LLMs across platforms like AWS Graviton, Microsoft Cobalt, Google Axion, NVIDIA Grace, and Raspberry Pi.
Benefits of Running LLMs on Arm with Llama.CPP
Llama.CPP is a lightweight, high-performance inference engine optimized for Arm CPUs, enabling fast, private LLM execution without accelerators. It supports quantization, multi-threading, and NEON/SVE acceleration for efficient, low-latency deployment across cloud environments. Designed for real-world developer workflows, it integrates seamlessly with CI/CD pipelines and container-based systems. Using PyTorch? Go to the PyTorch Developer Launchpad.
Get Started
Setup
To start developing with Llama.CPP on Arm-based systems, you’ll first need to set up your environment. This includes installing the core C++ library, exploring the high-level Python API, and validating your setup by running a local LLM inference.
Before starting, make sure you have:
What's Next?
- CODE-ALONGS
- ARM DEVELOPER PROGRAM
- COURSES AND LABS
- DEVELOPER RESEARCH
- MORE RESOURCES
Build and Run a RAG Pipeline on Arm – On Demand
Watch the step-by-step code-along and expert Q&A to build a complete RAG app with Hugging Face, LangChain, and FAISS – optimized for Arm-based infrastructure.
Arm Developer Program
Have a technical question about Llama.CPP on Arm cloud or Arm cloud migration?
Join the Arm Developer Program and connect with a global community of developers and Arm engineers to build better apps on Arm. Get early access to tools, technical content, workshops, and support to help you debug, optimize, and ship your projects.
Arm Developer Council
Join the Arm Developer Council to share feedback, help shape the tools and platforms you use — and receive a voucher for your time.
