Scaling On-Device AI Across different Arm backends with ExecuTorch

Taking a PyTorch model from development to an Arm-based edge device involves more than choosing whether to run it on a CPU, GPU, or NPU. Different devices expose different compute resources. Each ExecuTorch backend supports a different set of operators, data types, and optimization techniques.

ExecuTorch provides a common deployment workflow that helps bridge this gap. You can start with the same PyTorch model and export it once through a familiar workflow. You can then specialize its execution for the capabilities of the target device.

If you are new to ExecuTorch, start with Ethos-U and Beyond: How ExecuTorch 1.0 powers AI at the edge for a high-level introduction. You can also follow the ExecuTorch Beginner Pathway for a guided introduction to the deployment workflow. This article builds on those concepts. It explains how backend selection, graph partitioning, operator coverage, and quantization affect execution across Arm compute targets.

This is where the model graph becomes important. Which parts of the graph can a backend execute efficiently? Which operations need to remain on the CPU or another execution path? When several compute resources are available, how should you divide the workload between them?

In this article, we explore these questions by comparing five ExecuTorch execution paths across the Arm compute continuum:

• XNNPACK for optimized CPU inference, including on Cortex-A processors 
• Vulkan for GPU compute, with a current focus on Android GPUs 
• Arm VGF for graph-based execution through TOSA and ML SDK for Vulkan 
• CMSIS-NN for quantized inference on Cortex-M CPUs 
• Ethos-U for dedicated embedded NPU acceleration

We look at how the same PyTorch model is transformed for each target. We also examine how partitioning, operator coverage, quantization, and non-delegated execution affect the resulting deployment.

The common workflow and where it branches

The top part of this flow is largely shared. At each branch, the capabilities of the target hardware determine how the program is specialized.

That distinction matters. Portability here means a common PyTorch and ExecuTorch programming model. It does not mean that one .pte file or one runtime binary is optimal or valid for every target. Quantization, compilation, memory planning, linked kernels, registered delegates, and the final program can differ by device.

What an ExecuTorch partitioner actually does

A backend cannot assume that it supports the entire exported graph. Its partitioner examines the graph and tags the nodes or connected subgraphs that the backend can accept. ExecuTorch then lowers each tagged region into a backend-specific representation.

At runtime, ExecuTorch treats each delegated region as a delegate call. The backend initializes and executes the compiled region. Operations that were not delegated remain visible to the ExecuTorch runtime and must have an appropriate kernel implementation available.

Consider a deliberately simplified graph:

Suppose a backend supports Conv and ReLU, but not CustomOp. The exported program might become:

This is partial delegation. It increases model coverage, but it is not free. Every delegate boundary can introduce tensor-layout conversions, synchronization, copies, or quantization and dequantization work. Delegating more operators is useful only when the resulting regions are large and efficient enough to repay their boundary costs.

The typical API shape for delegate-based backends is compact:

import torch  
from executorch.exir import to_edge_transform_and_lower  
  
exported_program = torch.export.export(model.eval(), example_inputs)  
  
edge_program = to_edge_transform_and_lower(  
    exported_program,  
    partitioner=[TargetPartitioner()],  
)  
  
executorch_program = edge_program.to_executorch()  
executorch_program.save("model_target.pte") 

Changing TargetPartitioner changes which parts of the graph are claimed and how they are lowered. Backend-specific quantization and compile specifications may need to run before or as part of this step.

Path 1: Cortex-A CPU execution with XNNPACK

XNNPACK is a strong starting point for broad CPU inference coverage. In ExecuTorch, the XnnpackPartitioner identifies supported graph regions. It converts them to XNNPACK’s graph representation and serializes those regions into the program. It does not necessarily claim every operation in the model. For example, unsupported or custom operations may remain in the ExecuTorch graph. Later in the article, we look at how those non-delegated operations are handled. At runtime, the XNNPACK delegate initializes and executes the lowered graph using XNNPACK’s optimized kernels.

On Arm CPUs, XNNPACK can make use of optimized microkernels, including Arm KleidiAI integrations. Availability depends on the operator, data type, and hardware support them. Kernel selection can then use CPU features without requiring a different model architecture.

from executorch.backends.xnnpack.partition.xnnpack_partitioner import (  
    XnnpackPartitioner,  
)  
  
edge_program = to_edge_transform_and_lower(  
    exported_program,  
    partitioner=[XnnpackPartitioner()],  
) 

Choose this path when broad CPU availability matters, when no suitable accelerator is present. CPU execution can also provide the best balance of latency, memory, and integration cost. It is also a valuable baseline. Accelerator results are most useful when compared with a tuned CPU path.

XNNPACK is more than a fallback option. For many models and devices, it is the primary execution backend.

Path 2: GPU compute with the Vulkan backend

The ExecuTorch Vulkan backend lowers supported operators to compute-shader implementations included within ExecuTorch. It then executes them through the Vulkan API. The backend supports Vulkan 1.1 and is developed with a focus on Android GPUs. Its documented capabilities include dynamic shapes, FP32 and FP16 inference, and selected low-bit and dynamically quantized linear paths.

from executorch.backends.vulkan.partitioner.vulkan_partitioner import (  
    VulkanPartitioner,  
)  
  
edge_program = to_edge_transform_and_lower(  
    exported_program,  
    partitioner=[VulkanPartitioner()],  
) 

Vulkan is a good option for sufficiently parallel workloads and for devices whose GPU is available to the application. However, device support for Vulkan does not mean that the whole model runs efficiently on Vulkan. The partitioner also considers operator support, tensor properties, data types, and backend constraints. A graph fragmented into many small GPU regions can lose its theoretical advantage to dispatch, synchronization, and data-movement overhead. 

The practical questions are therefore: 

• How much of the graph is delegated? 
• Are the largest compute-intensive regions delegated? 
• How many CPU–GPU boundaries remain? 
• Does the chosen precision improve performance without unacceptable accuracy loss? 
• Does the target GPU and driver behave consistently for this workload? 

Path 3: graph-based execution with Arm VGF 

Vulkan and VGF both use Vulkan technology, but they are not interchangeable names for the same path. 

The ExecuTorch Vulkan backend and Arm VGF are separate execution paths. The ExecuTorch Vulkan backend maps supported operators to its own compute-shader implementations. Arm VGF uses a graph-based compilation flow. The Arm VGF backend lowers supported graph regions through TOSA (Tensor Operator Set Architecture), a standardized representation for tensor operations. It then uses the ML SDK Model Converter to generate VGF (Vulkan Graph Format) content. At runtime, the VGF graph is executed through ML SDK for Vulkan on a platform with the required Vulkan ML support.

This difference affects more than packaging. A graph-level compiler can reason about and optimize a supported region as a unit. An operator-oriented shader backend exposes a different set of capabilities and trade-offs. The two paths can therefore support different operators, shapes, precisions, and target environments. 

Choose VGF when the target platform explicitly supports the VGF and ML SDK stack. Use this path when you want graph-based compilation. To check compatibility, start with the Arm VGF backend target requirements and the ML SDK for Vulkan platform documentation. The target must provide ML SDK for Vulkan and a Vulkan 1.3-or-later driver. The driver capabilities can also be inspected using tools such as vulkaninfo.

Choose the ExecuTorch Vulkan backend when you want the direct ExecuTorch GPU compute path on a supported Vulkan device. In either case, compare the current operator-support tables against the exported graph. Do not choose by label alone.

Path 4: Cortex-M CPU execution with CMSIS-NN 

This path targets microcontrollers and deeply embedded devices, where memory, power, and compute resources are tightly constrained. Typical examples include IoT endpoints, sensors, wearables, and embedded control systems based on Arm Cortex-M processors.

The Cortex-M path is architecturally different from XNNPACK, Vulkan, VGF, and Ethos-U. The current ExecuTorch Cortex-M backend is an operator-library backend rather than a delegate-based backend.

After PT2E (PyTorch 2 Export) quantization, ExecuTorch runs a sequence of lowering passes over the graph. PT2E quantization prepares and converts the exported model for lower-precision execution. For supported quantized ATen operations, these passes replace the original operations with optimized CMSIS-NN kernel calls. Operations that this path does not support remain on portable FP32 kernels.

The integrated quantization path currently uses symmetric INT8. It uses per-channel quantization for convolution and per-tensor quantization for other supported operators.

This path targets memory- and power-constrained systems where every kernel and byte in the runtime matters. It is currently marked beta. The documented and validated instruction-set path is MVE, also known as Helium, on processors such as Cortex-M55 and Cortex-M85.

CMSIS-NN and Ethos-U both appear in embedded AI systems, but they solve different problems. CMSIS-NN accelerates neural-network operators on the Cortex-M CPU. It does not turn the CPU into an NPU. 

Path 5: embedded NPU execution with Ethos-U 

Ethos-U is a dedicated NPU path. The EthosUPartitioner finds supported, TOSA-compatible subgraphs. The backend serializes these subgraphs to TOSA. The Vela graph compiler then produces the command stream consumed by the NPU.

Ethos-U execution is integer-only. Operators intended for delegation must be quantized. The current primary flow using symmetric INT8 weights and asymmetric INT8 activations. Partial quantization is possible, but non-quantized operators are not delegated to Ethos-U. 

This makes quantization part of partitioning. It is possible for an operator to be structurally supported but remain outside the NPU region. Its data type, parameters, or surrounding graph might not meet backend constraints. 

Choose Ethos-U when the system includes an Ethos-U NPU and the workload benefits from its power- and performance-efficient quantized execution. Treat Vela configuration and memory placement as part of model optimization. On constrained systems, memory placement can be as important as the arithmetic. This includes whether weights and intermediate tensors use SRAM, flash, or external memory.

Unsupported operators: fallback without the mythology 

Saying that unsupported operators fall back to the CPU is useful shorthand, but it can hide the real mechanism. 

The key decisions are normally made ahead of time: 

1. A partitioner claims the regions it believes the backend supports. 
2. Those regions are lowered into backend-specific data. 
3. Unclaimed operations remain in the ExecuTorch graph or can be offered to a later partitioner. 
4. The runtime executes the serialized plan using the delegates and kernels linked into the target application. 

This is not universal, dynamic failover. If a delegate has already claimed and compiled a region, the runtime does not generally move a failing operation from inside that opaque region to the CPU. Likewise, a non-delegated operator can execute only if the target runtime contains a compatible implementation. 

ExecuTorch can apply more than one partitioner. For example, a deployment might offer GPU-compatible regions to Vulkan. It can then offer eligible remaining regions to XNNPACK:

edge_program = to_edge_transform_and_lower(  
    exported_program,  
    partitioner=[  
        VulkanPartitioner(),  
        XnnpackPartitioner(),  
    ],  
) 

Ordering expresses priority. The earlier partitioner sees the unclaimed graph first. It does not guarantee support for every backend combination, data type, graph boundary, or runtime build. Validate the exact combination on the target and inspect the lowered graph before relying on it. 

You can inspect the result immediately after lowering:

from executorch.devtools.backend_debug import get_delegation_info
from executorch.exir.backend.utils import format_delegated_graph

graph_module = edge_program.exported_program().graph_module

# Summarize what was and was not delegated.
delegation_info = get_delegation_info(graph_module)
print(delegation_info.get_summary())

# Inspect delegate boundaries and the lowered subgraphs.
print(format_delegated_graph(graph_module))

The output shows which parts of the graph were replaced by delegate calls. It also shows which operations remain in the ExecuTorch graph. For a multi-backend configuration, this is useful for confirming which partitions each backend claimed. It also shows where execution moves to another backend or ExecuTorch kernels.

Comparison of the five paths

Execution path	What it represents	Main preparation	Best fit	Treatment of remaining operations
XNNPACK	CPU delegate and optimized operator library	Optional backend-specific quantization; XNNPACK partitioning	Broad CPU inference, especially mobile and application-class processors	Remain as ExecuTorch operations if not claimed
Vulkan	GPU delegate using compute shaders	Vulkan partitioning; select supported precision and shapes	Parallel workloads on supported Android/Vulkan GPUs	Remain for another partitioner or ExecuTorch kernels
Arm VGF	Graph-based backend using TOSA and ML SDK for Vulkan	VGF partitioning and graph conversion; optional quantization	VGF-compatible Arm platforms	Remain outside VGF-delegated regions
Cortex-M / CMSIS-NN	CPU operator-library backend	PT2E INT8 quantization and Cortex-M lowering passes	Constrained Cortex-M systems	Portable FP32 kernels for unsupported operations
Ethos-U	Dedicated embedded NPU delegate	Target-specific quantization, TOSA lowering, and Vela compilation	Quantized embedded inference on Ethos-U55/U65/U85	Non-delegated runtime path, subject to available kernels

The table also highlights an important distinction. These names describe different abstraction levels. Cortex-A and Cortex-M are processor families. XNNPACK and CMSIS-NN are optimized software libraries. Vulkan is a graphics and compute API as well as the name of an ExecuTorch backend. VGF is a graph format/backend path. TOSA is an intermediate representation. Vela is a compiler. Ethos-U is NPU hardware with an ExecuTorch backend. 

Treating all of these terms as equivalent accelerators can quickly make architecture discussions confusing. 

A practical backend-selection workflow 

Start with the hardware you can actually deploy, but do not stop there:

1. Export the real model. Operator names in the source module are not enough; decompositions and graph transformations affect what a partitioner sees. 
2. Establish a CPU baseline. On application processors, measure XNNPACK. On Cortex-M, test the appropriate CMSIS-NN and portable-kernel build. 
3. Apply target-specific quantization. Quantization support differs by backend. Calibrate with representative data and check task accuracy, not only tensor similarity. 
4. Inspect delegation. Measure both the percentage of operations delegated and, more importantly, whether the expensive regions were delegated. 
5. Count boundaries. A high delegation percentage can still perform poorly if the graph alternates repeatedly between backends. 
6. Build the target runtime deliberately. Register every required delegate and kernel, and avoid carrying unused components on constrained systems. 
7. Profile on the device. Driver behavior, memory bandwidth, thermal limits, and delegate overhead do not appear in a desktop export log. 
8. Generate a separate artifact per target. Keep the PyTorch source common while allowing quantization, lowering, and runtime configuration to specialize.

After lowering, `get_delegation_info()` provides an immediate view of delegated and non-delegated operators: 

graph_module = edge_program.exported_program().graph_module  
delegation_info = get_delegation_info(graph_module)  
print(delegation_info.get_summary()) 

For runtime profiling, ExecuTorch provides ETRecord, ETDump, and the Inspector API. ETRecord captures model graphs and metadata during export. This data allows runtime events to be related back to operators in the model. ETDump captures profiling and debugging data produced while the model runs on the ExecuTorch runtime. The Inspector API combines these artifacts to analyse execution times and identify performance bottlenecks. These can include expensive operators, delegate execution, or non-delegated operations.

The real portability win 

The strongest reason to use ExecuTorch across Arm targets is not the promise of one universal binary. It is the ability to preserve a common model-development architecture while making target-specific decisions explicit. 

The same PyTorch model can be exported and analyzed through a consistent toolchain. One device might run most of the model through XNNPACK on a Cortex-A CPU. Another might send large regions to a Vulkan GPU. A VGF-capable platform can use graph-based lowering. A Cortex-M build can replace quantized operators with CMSIS-NN kernels. An Ethos-U system can compile supported quantized regions through TOSA and Vela for the NPU. 

The model source remains familiar. The execution plan differs by target, as it should. 

A useful way to think about heterogeneous on-device AI is to focus on the development workflow. Use one workflow to understand, partition, and optimize the model for the compute resources that are available.

Continue exploring 

• ExecuTorch backend overview 
• Understanding ExecuTorch backends and delegates 
• XNNPACK backend 
• Vulkan backend 
• Arm VGF backend 
• Arm Cortex-M backend 
• Arm Ethos-U backend 
• ExecuTorch profiling and debugging tools 
• ExecuTorch, XNNPACK, KleidiAI, and SME2 

Technical details and API names in this article were checked against the ExecuTorch 1.3 documentation. Backend capabilities and maturity change over time. Verify operator support and target requirements against the version used in your project.