Tutorial 3: Large language model (LLM)
This section provides an end-to-end walkthrough for optimizing a language model using the AQ + GO + GQ pipeline.
This tutorial uses smollm2_hf_pytorch, a preset LLM provided with Arm AI Model Optimizer. To use a different Hugging Face model, see Supported models. (AQ requires a Hugging Face model as input. For models already in .pt2 format, use the CV model tutorial instead, with GO and GQ only.)
This tutorial assumes you have already run np workspace init (see Quick start or Tutorial 1). Run the following steps from inside that workspace directory.
Step 1. Register your own dataset
Calibration and evaluation data for preset models is licensed for the sample_project Arm AI Model Optimizer generates automatically. For any project you create yourself, bring your own data. For a text-generation model, calibration data uses the plain_text format: non-empty *.txt files at the top level only. Subfolders are not scanned. A Hugging Face arrow-format folder is also recognized.
my_corpus_calib_30/
├─ sample_00001.txt
├─ sample_00002.txt
└─ sample_00003.txt
Graph Quantize calibrates each component (prefill and decode) against a fixed sequence length specified by seq_len, which is baked in at registration time. For smollm2_hf_pytorch, seq_len is 512 tokens.
A document shorter than that gets filtered out entirely rather than padded. If every document in the corpus is too short, calibration fails with No samples were produced ... no document in the dataset reaches the baked seq_len. Make sure each .txt file is long enough on its own to clear the target model's seq_len (several paragraphs, not one sentence).
Register a calibration split and an evaluation split separately:
np datasets register --source ./my_corpus_calib_30 --calibration \
--task TEXT_GENERATION --raw-data plain_text --name smollm2-calib-30
np datasets register --source ./my_corpus_eval_10 --evaluation \
--task TEXT_GENERATION --raw-data plain_text --name smollm2-eval-10
See Bring Your Own Model (BYOM) and Bring Your Own Data (BYOD) formats for the other supported task/format combinations.
Sample output
2026-08-21 13:00.10 [INFO ] [CLI] Registering Dataset
2026-08-21 13:00.10 [INFO ] [CLI] Checking the request with the server and requesting an upload URL...
2026-08-21 13:00.10 [INFO ] [CLI] Archiving the folder into a single tar (this can take a while for a large dataset)...
2026-08-21 13:00.10 [INFO ] [CLI] Computing the SHA-256 of the archive...
2026-08-21 13:00.10 [INFO ] [CLI] Uploading the archive to the server...
2026-08-21 13:00.10 [INFO ] [CLI] Registering and validating on the server. Validation reads every sample once per task, so this can take several minutes - do not interrupt it.
============================================================
Summary
============================================================
2026-08-21 13:00.13 [DONE ] [CLI] Dataset registered successfully.
[ Result Summary ]
Sample smollm2-calib-30
Ref ID b2c3d4e5-6f70-4819-a2b3-c4d5e6f70819
Split CALIBRATION
Tasks Text Generation
Uploaded 170.0 KiB
Status CREATED
[ Next Steps ]
Create a project with this dataset:
$ np projects create --calibration-dataset smollm2-calib-30
The evaluation split registers the same way, with --evaluation instead of --calibration:
2026-08-21 13:01.03 [DONE ] [CLI] Dataset registered successfully.
[ Result Summary ]
Sample smollm2-eval-10
Ref ID 28bddaa5-0dc4-45ab-8627-cba46f9bcf13
Split EVALUATION
Tasks Text Generation
Uploaded 30.0 KiB
Status CREATED
[ Next Steps ]
Create a project with this dataset:
$ np projects create --evaluation-dataset smollm2-eval-10
Step 2. Create a project
Provide the Hugging Face model name. The SDK downloads and converts it to PT2 automatically.
np projects create \
--name smollm2-rpi5 \
--model smollm2_hf_pytorch \
--device raspberry_pi_5_16gb \
--runtime executorch \
--backend xnnpack \
--calibration-dataset smollm2-calib-30 \
--evaluation-dataset smollm2-eval-10 \
--gpu-device cuda:1
Sample output
2026-08-21 13:15.00 [INFO ] [CLI] Creating Project
2026-08-21 13:15.00 [INFO ] [CLI] Parsing arguments...
2026-08-21 13:15.00 [INFO ] [CLI] - Model: smollm2_hf_pytorch
2026-08-21 13:15.00 [INFO ] [CLI] - Device: raspberry_pi_5_16gb
2026-08-21 13:15.00 [INFO ] [CLI] - Runtime: executorch
2026-08-21 13:15.00 [INFO ] [CLI] - Backend: xnnpack
2026-08-21 13:15.00 [INFO ] [CLI] - Calibration Dataset: smollm2-calib-30
2026-08-21 13:15.00 [INFO ] [CLI] - Evaluation Dataset: smollm2-eval-10
============================================================
Summary
============================================================
2026-08-21 13:15.02 [DONE ] [CLI] Project created successfully.
[ Result Summary ]
Project smollm2-rpi5
Location /workspace/workspace/smollm2-rpi5
Model smollm2_hf_pytorch
Device raspberry_pi_5_16gb
Runtime executorch
Backend xnnpack
Calibration smollm2-calib-30
Evaluation smollm2-eval-10
[ Next Steps ]
Go to project folder and create an experiment:
$ cd smollm2-rpi5
$ np experiments create
Step 3. Create an experiment
Generate a run.yaml with AQ + GO + GQ steps:
cd smollm2-rpi5
np experiments create --steps aq,go,gq
Sample output
2026-08-21 13:25.00 [INFO ] [CLI] Creating Experiment
2026-08-21 13:25.02 [DONE ] [CLI] Generated: run.yaml
[ Result Summary ]
Experiment experiment_00
Location /workspace/workspace/smollm2-rpi5/experiment_00
[ Next Steps ]
Go to experiment folder and review configuration:
$ cd experiment_00
$ vi run.yaml
$ np validate
The generated run.yaml:
version: '1.0'
steps:
- advanced_quantize:
- algorithm: AWQ # Available: RTN, AWQ, AUTOROUND, SMOOTHQUANT, GPTQ
scheme: W4-A8_dynamic # Available: W4-A8_dynamic, W8-A8, W8-A8_dynamic
components:
prefill:
algorithm: AWQ # Available: RTN, AWQ, AUTOROUND, SMOOTHQUANT, GPTQ
scheme: W4-A8_dynamic # Available: W4-A8_dynamic, W8-A8, W8-A8_dynamic
decode:
algorithm: AWQ # Available: RTN, AWQ, AUTOROUND, SMOOTHQUANT, GPTQ
scheme: W4-A8_dynamic # Available: W4-A8_dynamic, W8-A8, W8-A8_dynamic
- graph_optimize: null
- graph_quantize:
- scheme: W4-A8_dynamic # Available: W4-A8_dynamic, W8-A8, W8-A8_dynamic
automatic_mixed_precision: null # Available: GREEDY
components:
prefill:
scheme: W4-A8_dynamic # Available: W4-A8_dynamic, W8-A8, W8-A8_dynamic
automatic_mixed_precision: null # Available: GREEDY
decode:
scheme: W4-A8_dynamic # Available: W4-A8_dynamic, W8-A8, W8-A8_dynamic
automatic_mixed_precision: null # Available: GREEDY
profile:
- INPUT_MODEL
- ADVANCED_QUANTIZE
- GRAPH_OPTIMIZE
- OUTPUT_MODEL
evaluate:
- model: INPUT_MODEL
metric: # Available: PPL, HELLASWAG, SNR, MAE, MSE, KLD, KURTOSIS, PCTL_MAX
- SNR
- model: ADVANCED_QUANTIZE
metric: # Available: PPL, HELLASWAG, SNR, MAE, MSE, KLD, KURTOSIS, PCTL_MAX
- SNR
- model: GRAPH_OPTIMIZE
metric: # Available: PPL, HELLASWAG, SNR, MAE, MSE, KLD, KURTOSIS, PCTL_MAX
- SNR
- model: OUTPUT_MODEL
metric: # Available: PPL, HELLASWAG, SNR, MAE, MSE, KLD, KURTOSIS, PCTL_MAX
- SNR
SmolLM2 is a multi-component model whose execution splits into two independent graphs, prefill and decode. np experiments create writes a components: block under both advanced_quantize and graph_quantize, but only the graph_quantize block takes effect. AQ does not support per-component settings yet. AQ quantizes in a single pass using the model-level algorithm and scheme, so editing the values under advanced_quantize.components does not change the result. Leave those values as generated.
The component names and count come from the trace that np models register ran, not from anything you configure. Changing the model-level default (scheme:) still applies to any component without its own override. Single-component models (for example, most computer vision models) never get a components: block at all. See run.yaml: Multi-component models for more.
Step 4. Review and edit run.yaml
Open run.yaml and adjust settings as needed:
vi run.yaml
Common changes for LLMs:
| Field | Default | Options |
|---|---|---|
algorithm | AWQ | AWQ, AUTOROUND, RTN, SMOOTHQUANT, GPTQ |
scheme (AQ) | W4-A8_dynamic | W4-A8_dynamic, W8-A8, W8-A8_dynamic |
scheme (GQ) | W4-A8_dynamic | W4-A8_dynamic, W8-A8, W8-A8_dynamic |
To evaluate with both signal-to-noise ratio (SNR) and perplexity (PPL) simultaneously, modify the evaluate section:
version: '1.0'
steps:
- advanced_quantize:
- algorithm: AWQ # Available: RTN, AWQ, AUTOROUND, SMOOTHQUANT, GPTQ
scheme: W4-A8_dynamic # Available: W4-A8_dynamic, W8-A8, W8-A8_dynamic
profile:
- INPUT_MODEL
- OUTPUT_MODEL
evaluate:
- model: INPUT_MODEL
metric: # Available: PPL, HELLASWAG, SNR, MAE, MSE, KLD, KURTOSIS, PCTL_MAX
- SNR
- PPL
- model: OUTPUT_MODEL
metric: # Available: PPL, HELLASWAG, SNR, MAE, MSE, KLD, KURTOSIS, PCTL_MAX
- SNR
- PPL
See run.yaml Configuration for the full field reference.
For advanced per-module tuning (for example, calibration sample count, weight and activation granularity, or pattern handlers), run np experiments detail to generate run.detail.yaml. This file is auto-generated from your run.yaml and contains the detailed parameters for each step. Edit it before running the pipeline, and np run will merge both files automatically. See run.detail.yaml for the full reference.
Step 5. Validate
Optionally check for configuration errors before running the full pipeline. This step is recommended when using AQ, because it validates algorithm and scheme compatibility.
np validate
Sample output
2026-08-21 13:32.00 [INFO ] [CLI] Validating run configuration
2026-08-21 13:32.00 [DONE ] [CLI] Validation Passed. No issues found.
╭─────────────────────────────── Execution Plan ───────────────────────────────╮
│ Steps: │
│ [1/3] advanced_quantize — algorithm: AWQ, scheme: W4-A8_dynamic │
│ [2/3] graph_optimize │
│ [3/3] graph_quantize — scheme: W4-A8_dynamic │
│ Profile: INPUT_MODEL, ADVANCED_QUANTIZE, GRAPH_OPTIMIZE, OUTPUT_MODEL │
│ Evaluate: │
│ INPUT_MODEL (snr) │
│ ADVANCED_QUANTIZE (snr) │
│ GRAPH_OPTIMIZE (snr) │
│ OUTPUT_MODEL (snr) │
│ Total: 11 jobs │
╰──────────────────────────────────────────────────────────────────────────────╯
Step 6. Run the pipeline
AQ on large models can take significant time, from minutes to an hour depending on model size and GPU. Keep the container session alive.
np run --wait
Sample output
2026-08-21 13:40.00 [INFO ] [CLI] Submitting run (will wait for completion)
2026-08-21 13:40.00 [INFO ] [CLI] Logs appear as each step completes. A quiet screen means the run is still queued or a step is in progress.
2026-08-21 13:40.55 [INFO ] [Workflow] Downloading input model started
2026-08-21 13:41.04 [INFO ] [Workflow] Downloading input model completed (9.1s)
2026-08-21 13:41.04 [INFO ] [Workflow] Preparing calibration started
2026-08-21 13:41.05 [INFO ] [Workflow] Preparing calibration completed (0.7s)
2026-08-21 13:41.05 [INFO ] [Workflow]
════════════ [1/5] Advanced Quantize ════════════
2026-08-21 13:41.05 [INFO ] [Workflow] Advanced Quantize started
2026-08-21 13:43.25 [INFO ] [Workflow] Graph Extract started
2026-08-21 13:44.23 [INFO ] [Workflow] Graph Extract completed (57.5s)
2026-08-21 13:46.04 [INFO ] [Workflow] Uploading artifacts completed (1m 41s)
2026-08-21 13:46.04 [INFO ] [Workflow] Exporting VGF started
2026-08-21 13:52.43 [INFO ] [Workflow] Exporting VGF completed (6m 39s)
2026-08-21 13:53.27 [DONE ] [Workflow] Advanced Quantize completed (12m 22s)
2026-08-21 13:53.27 [INFO ] [Workflow]
════════════ [2/5] Graph Optimize ════════════
2026-08-21 13:53.27 [INFO ] [Workflow] Graph Optimize started
2026-08-21 13:59.21 [INFO ] [Workflow] Uploading artifacts completed (47.7s)
2026-08-21 13:59.21 [INFO ] [Workflow] Exporting VGF started
2026-08-21 14:04.15 [INFO ] [Workflow] Exporting VGF completed (4m 54s)
2026-08-21 14:04.33 [DONE ] [Workflow] Graph Optimize completed (11m 6s)
2026-08-21 14:04.33 [INFO ] [Workflow] Applied 11 patterns (3 loops):
Loop 1:
- Fuse Constant Path (1)
- Remove No Op Cast (1)
- Remove No Op Math (2)
- Remove No Op Slice (1)
- Fuse Consecutive Reshape Transpose (30)
- Fuse QKV (30)
- Remove Graph Asserts Pass (126)
- Replace Matmul With Bmm (1)
- Remove Common Subexpression (87)
- Decompose Scaled Dot Product Attention (30)
Loop 2:
- Replace Matmul With Bmm (60)
- Remove Common Subexpression (29)
Loop 3:
- Combine SDPA (30)
2026-08-21 14:04.33 [INFO ] [Workflow]
════════════ [3/5] Graph Quantize ════════════
2026-08-21 14:04.33 [INFO ] [Workflow] Graph Quantize started
2026-08-21 14:04.33 [INFO ] [Workflow] Preparing calibration for quantization started
2026-08-21 14:04.59 [INFO ] [Workflow] Preparing calibration for quantization completed (25.5s)
2026-08-21 14:08.57 [INFO ] [Workflow] Uploading artifacts completed (1m 20s)
2026-08-21 14:08.57 [INFO ] [Workflow] Exporting VGF started
2026-08-21 14:16.13 [DONE ] [Workflow] Graph Quantize completed (12m 1s)
2026-08-21 14:16.13 [INFO ] [Workflow]
════════════ [4/5] Evaluate ════════════
2026-08-21 14:16.13 [INFO ] [Workflow] Evaluate started
2026-08-21 14:16.13 [INFO ] [Workflow] Preparing evaluation dataset started
2026-08-21 14:16.20 [INFO ] [Workflow] Preparing evaluation dataset completed (7.3s)
2026-08-21 14:20.55 [DONE ] [Workflow] Evaluate completed (4m 42s)
2026-08-21 14:20.55 [INFO ] [Workflow]
════════════ [5/5] Profile ════════════
2026-08-21 14:20.55 [INFO ] [Workflow] Profile started
2026-08-21 14:20.56 [INFO ] [Workflow] INPUT_MODEL:prefill
2026-08-21 14:21.02 [INFO ] [Workflow] INPUT_MODEL:decode
2026-08-21 14:21.08 [INFO ] [Workflow] ADVANCED_QUANTIZE:prefill
2026-08-21 14:21.14 [INFO ] [Workflow] ADVANCED_QUANTIZE:decode
2026-08-21 14:21.20 [INFO ] [Workflow] GRAPH_OPTIMIZE:prefill
2026-08-21 14:21.35 [INFO ] [Workflow] GRAPH_OPTIMIZE:decode
2026-08-21 14:21.41 [INFO ] [Workflow] OUTPUT_MODEL:prefill
2026-08-21 14:21.57 [INFO ] [Workflow] OUTPUT_MODEL:decode
2026-08-21 14:22.05 [DONE ] [Workflow] Profile completed (1m 10s)
2026-08-21 14:22.05 [INFO ] [Workflow] Run completed (41m 34s)
2026-08-21 14:22.08 [DONE ] [CLI] Run completed successfully. (42m 27s)
[ Result Summary ]
Experiment experiment_00
Status success
Measurements
Evaluate base✓ 01_advanced_quantize✓ 02_graph_optimize✓ 03_graph_quantize✓
Profile base- 01_advanced_quantize- 02_graph_optimize- 03_graph_quantize-
Artifacts
VGF base✗ 01_advanced_quantize✗ 02_graph_optimize✗ 03_graph_quantize✗
NPIR base✓ 01_advanced_quantize✓ 02_graph_optimize✓ 03_graph_quantize✓
[ Next Steps ]
View results or export artifacts:
$ np status --experiment experiment_00
$ np report
$ np export
2 things worth calling out in this run, both specific to SmolLM2's multi-component (prefill and decode) executorch/xnnpack export path right now:
VGFshows✗for every checkpoint. VGF export currently fails for this model with a dtype error (an internal tensor comes out asFP16where the exporter expectsINT8,INT16,INT32, orFP32).NPIRstill exports successfully, so the run itself completes and every other measurement (for example, Evaluate, MACs and parameters, SNR) is unaffected. This only blocks the VGF-dependent artifact.Profileshows-instead of✓. With VGF unavailable, per-checkpoint profiling has nothing to measure against, so no measurement is recorded for any checkpoint this run.
If you hit this, check np status --experiment <id> --logs for the underlying error and treat Profile results as unavailable for now on this combination of model and backend.
Step 7. View results
np report
Sample output
Experiment: experiment_00
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Model ┃ Type ┃ Status ┃ Memory(MB) ┃ Latency(ms) ┃ MACs ┃ Params ┃ SNR (vs input) ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ input_model │ - │ - │ -- │ -- │ 54.5G │ 269.0M │ inf │
│ 01_advanced_quantize (2 components) │ advanced_quantize │ success │ -- │ -- │ 54.5G │ 269.0M │ 45.20 │
│ 02_graph_optimize (2 components) │ graph_optimize │ success │ -- │ -- │ 72.7G │ 269.0M │ 18.75 │
│ 03_graph_quantize (2 components) │ graph_quantize │ success │ -- │ -- │ 72.7G │ 342.5M │ 6.42 │
└─────────────────────────────────────┴───────────────────┴─────────┴────────────┴─────────────┴───────┴────────┴────────────────┘
Note: For (N components) models, Latency is the sum across components.
Peak Memory is the largest component's, not the model total.
MACs is the sum for one prefill plus one decode step.
Comparison metrics show the output component's own value, not a composed E2E measurement.
RPi5 is xnnpack, so memory and latency show as -- here too (see Profiler). MACs and parameters are the only per-step numbers available. Each row is (2 components) because SmolLM2 splits into prefill and decode. The previous note explains how the combined values were derived. Throughout that note, component means one of the model's traced graphs, prefill or decode, not a category of memory such as weights or activations. So "the largest component's" peak memory means whichever of prefill or decode peaks higher, not the largest contributor within one graph. Unlike a pure weight-precision change, MACs jump after Graph Optimize (the fused and decomposed ops change the graph's shape as shown in the pattern list in the previous step). Parameters jump after Graph Quantize (quantization adds per-tensor scale/zero-point parameters).
Sort by a metric:
np report --sort latency
np report --sort snr
Step 8. Export the model
np export
Sample output
2026-08-21 14:30.00 [INFO ] [CLI] Exporting experiment artifacts
2026-08-21 14:30.00 [INFO ] [CLI] Fetching S3 artifacts (presigned GET) ...
2026-08-21 14:30.15 [DONE ] [CLI] Export completed.
[ Result Summary ]
Level deploy
Extracted to /netspresso/workspace/smollm2-rpi5/experiment_00/artifacts/deploy
[ Result ]
- smollm2_decode.pte
- smollm2_prefill.pte
[ Next Steps ]
To include config or execution results:
$ np export --level config
$ np export --level result
Multi-component models export one .pte per component (here, smollm2_prefill.pte and smollm2_decode.pte) instead of a single model.pte. Deploy both files together.
Step 9. Visualize the optimized graph
np visualize generates a link to the graph viewer (probe) rather than opening a browser directly. The viewer needs a browser session. Bootstrap that once with np visualize auth:
np visualize auth
Sample output
Session bootstrap ready — use EITHER method (both establish the same session):
1) Open in browser (auto-establishes the session):
https://auth.your-deployment.netspresso.ai/api/v1/visualize/authorize?bootstrap=eyJhbGciOi...
2) Or paste this token into the page's auth field (same value as the end of the link above):
──────────────────── TOKEN — copy the line below ────────────────────
eyJhbGciOi...
─────────────────────────────────────────────────────────────────────
This link/token expires in about 179s. Use it promptly.
Open the link (or paste the token) in your browser once. The session then lasts about an hour. Then generate the graph link:
np visualize
Sample output
[Tool] Using --tool probe (default).
- probe : any model format. Supports --compare diff.
- model-explorer : VGF only. Single-graph view.
Pass --tool to silence this notice.
[Context] Experiment: experiment_00 (Detected)
2026-08-21 14:40.00 [DONE ] [CLI] Visualization link generated
https://app.your-deployment.netspresso.ai?target=<target_ref>&baseline=<model_ref>&workspace=<ws_ref>
[ Next Steps ]
Copy the URL and open it in your browser
See Visualizer for details on reading the diff view.
Trying different algorithms
To compare AQ algorithms (for example, AWQ versus AutoRound), create a second experiment:
cd ..
np experiments create --steps aq,go,gq
cd experiment_01
vi run.yaml # change algorithm to autoround
np run
# Compare from project directory
cd ..
np report
Next steps
- Visualizer: Inspect the optimized graph before deploying with
np visualize - Features: Advanced Quantizer: AWQ, AutoRound, and other algorithms in depth
- Features: Graph Extractor: How Hugging Face models are converted to PT2
- Supported models: Use your own Hugging Face model
- run.yaml configuration: AQ algorithm parameters and full pipeline configuration reference