Concepts
Workspace hierarchy
Work is organized in a 4-level hierarchy: Workspace → Project → Experiment → Job.
| Level | Represents | Created by |
|---|---|---|
| Workspace | Root container for all projects and datasets | np workspace init |
| Project | 1 model and 1 target device | np projects create |
| Experiment | 1 optimization configuration (run.yaml) | np experiments create |
| Job | 1 step execution within the pipeline, run asynchronously on the backend | np run (1 job for each step) |
Directory layout
Locally, an experiment directory holds its configuration and the artifacts pulled from np export. The backend is the source of truth for everything else.
workspace/<project>/<experiment>/
├── run.yaml # pipeline configuration
├── run.detail.yaml # fine-grained overrides (optional)
├── meta.json
├── report.json # raw np report output
├── logs/ # execution logs (written automatically)
└── NN_<step>/ # e.g. 01_graph_optimize, 02_graph_quantize
├── job.json
└── artifacts/ # populated by np export --level result or higher
See Log reference for the content of logs/, and np export for what each --level pulls into artifacts/.
How a job runs
np installs directly using pip and connects to the Arm AI Model Optimizer backend over HTTPS. np creates and submits experiment configurations and retrieves results, but optimization, evaluation, and profiling run on the backend.
By default, np run submits and returns immediately. To follow progress in the terminal, use --wait. To resume a failed or cancelled run, use --retry.
Every command falls into 1 of 3 tracks
| Track | Execution | Example commands |
|---|---|---|
| Synchronous | A single direct HTTP call. The command does not return until the server completes the processing. There is no GPU worker involved. | experiments create, validate, datasets register, model and dataset listing |
| Asynchronous | Queued and run on a GPU worker. Model registration is an exception: it runs a synchronous upload, then verifies in the background. Pass --wait to block until processing finishes. Omit --wait to regain control immediately and check the status later. | run, evaluate, profile, models register |
| Local | Read-only, local only | Workspace inspection |
Connecting to the backend
np communicates with 2 servers: a backend server that handles everything except login, and a separate auth server. Both addresses come from environment variables. You must be logged in to run commands, including np workspace init:
np health ready # verify the backend is reachable
np auth login <credential> # log in once; the CLI refreshes your session automatically after that
See CLI reference: np auth for how credentials, environments, and auto-refresh work.
Context-aware commands
The CLI detects the current project and experiment automatically from your working directory.
# Inside experiment_00 → runs that experiment
user@host:.../resnet-rpi5/experiment_00$ np run
# Inside project directory → shows all experiments
user@host:.../resnet-rpi5$ np report
Optimization pipeline
Arm AI Model Optimizer uses a modular optimization pipeline. Each module handles a specific stage of the workflow.
| Pipeline | When to use |
|---|---|
Graph Optimize → Graph Quantize (--steps go,gq) | GO applies hardware-appropriate graph transformations at the intermediate representation (NPIR) level, for example, fuse, fold, and replace. GQ then converts weights and activations to INT8 using calibration data. This is the standard pipeline for most models. |
Advanced Quantize → Graph Optimize → Graph Quantize (--steps aq,go,gq) | AQ first applies weight-aware quantization at the PyTorch level before graph conversion, preserving accuracy at aggressive bit-widths (INT4). Use this pipeline when you need higher accuracy at INT4. |
You can also run steps independently for iterative experimentation.
Graph Extractor (GE)
GE converts a model into the intermediate representation (PT2 / NPIR). It traces the model structure once when you run np models register, using the inputs in model_register.yaml. This is also where every tensor shape is resolved to a fixed, static value. There is no shape configuration later in run.yaml or run.detail.yaml. For models already in .pt2 format, this step is skipped.
For more details, see the Graph Extractor page.
Advanced Quantize (AQ)
AQ applies weight-aware quantization at the PyTorch level before graph conversion. AQ preserves accuracy at aggressive bit-widths (INT4) by protecting weights that activations are most sensitive to.
Supported algorithms: AWQ, AutoRound, RTN, SmoothQuant, GPTQ
For more details, see the Advanced Quantizer page.
Graph Optimize (GO)
GO applies hardware-appropriate graph transformations at the NPIR level, for example, fold, fuse, remove, and replace. No configuration is required: the target device determines which patterns are applied.
For more details, see the Graph Optimizer page.
Graph Quantize (GQ)
GQ converts floating-point weights and activations to lower bit-width integers using calibration data. Available schemes depend on the target hardware backend.
| Backend | Supported schemes |
|---|---|
| XNNPACK | W8-A8, W8-A8_dynamic, W4-A8_dynamic |
| Ethos-U | W8-A8, W8-A16 |
For more details, see the Graph Quantizer page.
Iterative optimization
The typical workflow is as follows:
- run an experiment
- check the metrics
- adjust the configuration
- rerun the experiment
- repeat until the signal-to-noise ratio (SNR), accuracy, perplexity (PPL), latency, and memory are acceptable
np run --wait → np report → adjust run.yaml / run.detail.yaml → np run --wait → ...
Each iteration is a new experiment inside the same project. Running np report from the project directory compares all experiments side by side:
Project: resnet-rpi5
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Model ┃ Steps ┃ Status ┃ Memory(MB) ┃ Latency(ms) ┃ MACs ┃ Params ┃ SNR (vs input) ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ input_model │ - │ - │ -- │ -- │ 4.1G │ 25.6M │ inf │
│ experiment_00 │ GO, GQ │ success │ -- │ -- │ 4.1G │ 25.6M │ 1.48 │
│ experiment_01 │ GO, GQ │ success │ -- │ -- │ 4.1G │ 25.6M │ 0.82 │
│ experiment_02 │ AQ, GO, GQ │ success │ -- │ -- │ 4.1G │ 25.6M │ 3.20 │
└───────────────┴────────────┴─────────┴────────────┴─────────────┴──────┴────────┴────────────────┘
RPi5 is an xnnpack-backend device, so mlia reports MACs and parameters only. Memory and latency show as -- (see Profiling output by backend). MACs and parameters stay constant across GO and GQ because quantization changes bit width, not the underlying graph.
What to adjust
| Symptom | What to try |
|---|---|
| SNR and accuracy too low | Use a less aggressive scheme (W4-A8_dynamic → W8-A8_dynamic), or add AQ |
| Latency and memory too high | Use a more aggressive scheme (W8-A8 → W4-A8_dynamic), or enable AMP |
| AQ result is unstable | Try a different algorithm (AWQ → AutoRound), or tune weight.calibration_method in run.detail.yaml |
| Calibration is noisy | Increase num_samples in run.detail.yaml |
Creating the next experiment
# From the project directory
np experiments create --steps go,gq
cd experiment_01
vi run.yaml # adjust scheme, steps, etc.
np experiments detail # optional: generate run.detail.yaml
vi run.detail.yaml # fine-tune calibration, granularity, AMP
np run --wait
# Compare from project directory
cd ..
np report --sort snr
Completed experiments are preserved for later comparison. For a specific walkthrough, see Tutorial 2: Computer vision (CV) model.
Measurement and inspection
These modules measure or inspect the model at each step and provide the metrics that help you make optimization and iteration decisions. Evaluator and Profiler run per step during np run, and you can also run both stages after processing. Visualizer is a separate post-run command.
Evaluate
Evaluator measures the quality of each checkpoint against an evaluation dataset. List the checkpoints that you want to compare in the evaluate section of run.yaml. Evaluator compares them during np run. You can also run Evaluator after processing with np evaluate <id>. Supported metrics include ACCURACY, SNR, MAE, MSE, PPL, and mAP.
For more details, see the Evaluator page.
Profile
Profiler measures the latency and memory of each checkpoint using the Arm Machine Learning Inference Advisor (MLIA). The measurement is a static estimate, not an on-device run. List the checkpoints in the profile section of run.yaml. You can also run Profiler after processing with np profile <job_id> --method mlia. <job_id> can also be a model or experiment ID. Always pass --method mlia explicitly. See Profiler for more information.
For more details, see the Profiler page.
Visualize
Visualizer opens the model graph in a browser-based viewer to inspect its structure and compare checkpoints side by side. Run post-hoc with np visualize; by default it shows a diff between the input and final output models.
For more details, see the Visualizer page.
Compatibility matrix
A quick reference for what is available in which combination. See the linked feature pages for full detail.
Pipeline steps by model input
| Model input | AQ | GO | GQ |
|---|---|---|---|
| HuggingFace model | ✅ | ✅ | ✅ |
.pt2 file | ❌ not supported | ✅ | ✅ |
AQ requires a Hugging Face model as input. For a .pt2 input, use --steps go,gq instead. See Advanced Quantizer.
Profiling output by backend
| Backend | Devices | Metrics reported |
|---|---|---|
xnnpack | RPi5, M4 MacBook, Graviton4, vivo X300 Pro | MACs and parameters only, no latency or memory |
ethos_u | Alif E8 | Estimated latency and memory, plus MACs and parameters |
Always pass --method mlia explicitly rather than relying on the default. See Profiler for more information. Full MLIA output is available only on ethos_u. This output includes estimated latency, memory, MACs, and parameters. On xnnpack targets, mlia reports only MACs and parameters.
Device and backend combinations
| Device | Runtime | Backend |
|---|---|---|
| Raspberry Pi 5 | executorch | xnnpack |
| M4 MacBook | executorch | xnnpack |
| AWS Graviton4 | executorch | xnnpack |
| vivo X300 Pro | executorch | xnnpack |
| Alif E8 | executorch | ethos_u |
Run np devices ls for the live, authoritative list. See Supported Devices for full specs.
Bringing your own model and data
Alongside preset (Nota-provided) models and datasets, you can register your own models and datasets using the Bring Your Own Model (BYOM) and Bring Your Own Data (BYOD) formats.
| Type | How to get | Metrics available |
|---|---|---|
| Preset | Selected at np projects create --calibration-dataset <name> --evaluation-dataset <name> | SNR + task metrics (mAP, Accuracy, PPL) |
| Custom (BYOM and BYOD) | Registered with np models register or np datasets register | Same task metrics, after you register the model or dataset |
For this release, BYOM registration is limited to large language model (LLM) text-generation models. BYOD accepts any task, provided your data follows one of the supported folder layouts. AQ uses its own internal calibration source from Hugging Face Hub and does not consume the registered dataset.
See Bring Your Own Model (BYOM) and Bring Your Own Data (BYOD) formats and Tutorial 4: Bring Your Own Model (BYOM) and Bring Your Own Data (BYOD).
Model reference names
Use these names in the profile and evaluate sections of the run.yaml file:
| Name | Description |
|---|---|
INPUT_MODEL | Original model before any optimization |
ADVANCED_QUANTIZE | Checkpoint after AQ |
GRAPH_OPTIMIZE | Checkpoint after GO |
GRAPH_QUANTIZE | Checkpoint after GQ |
OUTPUT_MODEL | Final output of the last step |