On-device profiling

Measuring directly on physical hardware gives you a real measurement, but the mechanics differ by target: an XNNPACK device runs a single binary directly, but an Ethos-U device requires a build-flash-capture flow. This is a manual process you run yourself (outside np profile) using the artifacts Arm AI Model Optimizer provides.

There are 2 profiling paths, depending on the backend of your device:

ExecuTorch + XNNPACK (RPi5, Graviton4, vivo X300 Pro): run the provided executor_runner binary against your .pte. See ExecuTorch + XNNPACK.
ExecuTorch + Ethos-U (Alif E8): the model is compiled into firmware, so you build → flash → capture over RTT. See ExecuTorch + Ethos-U.

If you do not have physical access to the target device, np profile --method mlia gives a static estimate using the Arm Machine Learning Inference Advisor. See Profiler for what it produces for each backend.

ExecuTorch + XNNPACK: `executor_runner`

This profiling path applies to RPi5, AWS Graviton4, and vivo X300 Pro. A single C++ binary, executor_runner, loads the .pte and runs it. Latency and peak memory are read from its output.

The binary differs according to the platform. Android is driven over adb from a host PC instead of a shell on the device. Everything else (the command, and how you read the numbers) is the same on all 3 devices. For information on device-specific steps, see Per-Device Notes.

Prerequisites

The executor_runner binary for your device architecture. See the Arm AI Model Optimizer profiling binaries. The binary must be built from the same ExecuTorch version as the .pte to load the model.
The .pte model from Arm AI Model Optimizer.
/usr/bin/time for peak-memory measurement (GNU time on Linux).
For vivo X300 Pro only: Android platform-tools (adb) on the host PC.

Device	Binary
RPi5 (aarch64 Linux)	`executor_runner-linux-aarch64`
AWS Graviton4 (aarch64 Linux)	`executor_runner-linux-aarch64-graviton`
vivo X300 Pro (Android arm64)	`executor_runner-android-arm64`

note

The use and download of the Arm AI Model Optimizer Performance Evaluation Software from Arm Artifactory is subject to the terms of the End User License Agreement (EULA) for Arm AI Model Optimizer Performance Evaluation Software, a copy of which is included in the software download package.

Command

/usr/bin/time -v ./executor_runner \
    --model_path   model.pte \
    --num_executions 10 \
    --etdump_path  profile.etdump

Argument	Description
`--model_path`	Path to the `.pte` model.
`--num_executions`	Number of inference iterations (used to compute the average latency).
`--etdump_path`	ETDump (op-level profiling) output path. Omit if you do not need op-level data.

Example output: Raspberry Pi 5

Running the command above on a Raspberry Pi 5 with a classification model produces the following output:

I executorch:cpuinfo_utils.cpp:87] Failed to open midr file /sys/devices/soc0/image_version
I executorch:cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu0/regs/identification/midr_el1
I executorch:executor_runner.cpp:203] Resetting threadpool with num threads = 4
I executorch:executor_runner.cpp:352] Model file /tmp/model.pte is loaded.
I executorch:executor_runner.cpp:361] Using method forward
I executorch:executor_runner.cpp:412] Setting up planned buffer 0, size 9633856.
I executorch:executor_runner.cpp:440] Method loaded.
I executorch:executor_runner.cpp:498] Model executed successfully 10 time(s) in 426.452859 ms.
I executorch:executor_runner.cpp:502] 1 outputs:
Output 0: tensor(sizes=[1, 3], [-0.225492, 0.129735, 0.0447894])
I executorch:executor_runner.cpp:151] ETDump written to file '/tmp/profile.etdump'.
        Command being timed: "executor_runner --model_path model.pte --num_executions 10 --etdump_path profile.etdump"
        User time (seconds): 1.45
        Percent of CPU this job got: 325%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.45
        Maximum resident set size (kbytes): 43600
        Exit status: 0

How to read the output

Average latency: from Model executed successfully 10 time(s) in 426.452859 ms: divide total by iterations → 426.45 / 10 ≈ 42.65 ms per inference.
Peak memory: from /usr/bin/time -v's Maximum resident set size (kbytes): 43600 → ≈ 43.6 MB.
Model memory: Setting up planned buffer 0, size 9633856 is the activation arena (≈ 9.6 MB).
Success check: Exit status: 0 plus the printed output tensor. The Failed to open midr file / midr_el1 lines at the top are harmless CPU-identification probes and do not affect results.

The same 3 lines carry the same meaning on Graviton4 and vivo X300 Pro.

Per-device notes

AWS Graviton4

Graviton4 is a cloud instance rather than a desktop board, so "on-device" means inside the instance: launch an Arm instance (the r8g family is Graviton4), copy the binary and the .pte in with scp, and run the previous command over SSH.

Instance image

Launch the instance with Ubuntu 26.04. executor_runner-linux-aarch64-graviton is built and validated on that image. An older image ships an older glibc than the binary requires, and it will not start.

vivo X300 Pro on Android

The phone is driven over adb from a host PC. Enable Developer options → USB debugging on the phone, connect it, and confirm it is visible:

adb devices

Push the binary and the model into /data/local/tmp. This is the directory a shell user can execute from on Android. Then run:

adb push executor_runner-android-arm64 /data/local/tmp/executor_runner
adb shell chmod 755 /data/local/tmp/executor_runner
adb push model.pte /data/local/tmp/

adb shell "cd /data/local/tmp && ./executor_runner \
    --model_path model.pte \
    --num_executions 10 \
    --etdump_path profile.etdump"

adb pull /data/local/tmp/profile.etdump .

ExecuTorch + Ethos-U: Alif E8

This profiling path applies to Alif E8. Unlike the XNNPACK path, the Ethos-U model is compiled into the firmware, so there is no single binary. The flow is build → flash → capture over RTT:

.pte  ──(MLEK cmake build, model embedded)──▶  mram.bin + ext_flash.bin
      ──(J-Link)──▶ upload flasher ──▶ ext_flash.bin (XMODEM) ──▶ write mram.bin
      ──(SWD reset)──▶ capture inference time over RTT

Prerequisites

Alif DevKit-E8 board + Segger J-Link for Serial Wire Debug (SWD) and Real-Time Transfer (RTT).
An x86_64 Linux host tested on Ubuntu 22.04.
Provided with Arm AI Model Optimizer: the nota-github/mlek_experimental build environment, the executorch_fpic.patch file included at the root of every extracted executor_runner-<type>-<version>.tar.gz package, and the SE Tools TOC configs (asr.json and flasher-cfg.json, delivered directly).
From Alif (download with your Alif account): Alif Security Toolkit (SETOOLS, SE_FW 1.109.00), and alifsemi/alif_usb-to-ospi-flasher.
Toolchain: arm-none-eabi GCC, CMake and Make, Python 3.11 with pylink-square, pyserial, and xmodem, and J-Link software.
The .pte from Arm AI Model Optimizer, with its memory_mode and recommended TMP_MEM_SIZE and ACTIVATION_BUF_SZ values.

Build the firmware (`.pte` → `mram.bin` + `ext_flash.bin`)

git clone https://github.com/nota-github/mlek_experimental
cd mlek_experimental
git submodule update --init --recursive          # pulls executorch + dependencies

# Set up ML-framework resources (creates a venv and installs ExecuTorch)
./set_up_default_resources.py --ml-frameworks executorch

# Apply the fPIC patch (required for bare-metal build — without it, inference hangs)
git -C dependencies/executorch apply /path/to/executorch_fpic.patch

mkdir -p build_benchmark && cd build_benchmark
PYTHONPATH=<mlek>/dependencies/executorch cmake \
  -DUSE_CASE_BUILD=executorch_inference_runner \
  -DTARGET_PLATFORM=alif -DTARGET_SUBSYSTEM=RTSS-HP -DTARGET_BOARD=DevKit-e8 \
  -DML_FRAMEWORK=ExecuTorch \
  -DCMAKE_TOOLCHAIN_FILE=scripts/cmake/toolchains/bare-metal-gcc.cmake \
  -DCMAKE_BUILD_TYPE=Release \
  -DETHOS_U_NPU_ID=U85 -DETHOS_U_NPU_CONFIG_ID=Z256 \
  -DETHOS_U_NPU_MEMORY_MODE=Dedicated_Sram \
  -DETHOS_U_NPU_CACHE_SIZE=0x00060000 \
  -DML_FWK_TMP_MEM_SIZE=0x00040000 \
  -Dexecutorch_inference_runner_ACTIVATION_BUF_SZ=0x00600000 \
  -DEXECUTORCH_ENABLE_EVENT_TRACER=ON \
  -DOSPI_RAM_SUPPORT=ON \
  -DOSPI_RAM_TEST=ON \
  -Dexecutorch_inference_runner_MODEL_PATH=<path/to/model.pte> \
  ..

make -j$(($(nproc)/2)) mlek_executorch_inference_runner

Outputs:

build/bin/sectors/executorch_inference_runner/mram.bin        # firmware + weights
build/bin/sectors/executorch_inference_runner/ext_flash.bin   # external flash data

memory_mode

The default is Dedicated_Sram: activations live in external Octal Serial Peripheral Interface (OSPI) pseudo-static random-access memory (PSRAM) with the NPU cache in on-chip SRAM (shown above via -DETHOS_U_NPU_CACHE_SIZE and -DOSPI_RAM_TEST=ON). To run a small model entirely in on-chip SRAM, use Shared_Sram instead: set -DETHOS_U_NPU_MEMORY_MODE=Shared_Sram, drop the -DETHOS_U_NPU_CACHE_SIZE and -DOSPI_RAM_TEST flags, and size ACTIVATION_BUF_SZ for SRAM. The build's memory_mode must match the one used when the .pte was lowered (Vela), or the device will fault. Arm AI Model Optimizer provides the per-model values.

warning

After changing the executorch source or the patch, delete build_benchmark before rebuilding (rm -rf build_benchmark). The per-build is incremental and will otherwise reuse stale objects; the benchmark reports success but runs the old binary.

Flash the device

From the SE Tools directory, write the images over J-Link using Serial Wire Debug (SWD) in order:

# (1) upload the flasher
cp <flasher>/out/flasher/E8-HE/debug/flasher.bin  build/images/flasher_he.bin
./app-gen-toc -f build/config/flasher-cfg.json && ./app-write-mram -p

# (2) transfer ext_flash.bin to the flasher over XMODEM (via RTT)

# (3) write mram.bin
cp <build>/.../mram.bin  build/images/mram.bin
./app-gen-toc -f build/config/asr.json && ./app-write-mram -p

# (4) issue an SWD reset over J-Link to boot the new firmware

note

app-write-mram writes magnetoresistive random-access memory (MRAM) but does not reset the chip. A J-Link SWD reset (a software action issued over the debug probe, not a physical button) is required to boot the v1.1.0 firmware and initialize the inference RTT channel, otherwise the host sees 0 bytes.

Capture results over RTT

Read the inference time from the J-Link RTT channel:

Per-inference latency: parse the Inference: 12.345 ms lines from the RTT log.
Optionally, output tensors and ETDump are streamed over RTT as base64 (output.bin, etdump.bin).

ExecuTorch + XNNPACK: executor_runner​

Prerequisites​

Command​

Example output: Raspberry Pi 5​

How to read the output​

Per-device notes​

AWS Graviton4​

vivo X300 Pro on Android​

ExecuTorch + Ethos-U: Alif E8​

Prerequisites​

Build the firmware (.pte → mram.bin + ext_flash.bin)​

Flash the device​

Capture results over RTT​

Next steps​

ExecuTorch + XNNPACK: `executor_runner`

Prerequisites

Command

Example output: Raspberry Pi 5

How to read the output

Per-device notes

AWS Graviton4

vivo X300 Pro on Android

ExecuTorch + Ethos-U: Alif E8

Prerequisites

Build the firmware (`.pte` → `mram.bin` + `ext_flash.bin`)

Flash the device

Capture results over RTT

Next steps