Evaluator

The Evaluator measures the accuracy of model checkpoints produced during the optimization pipeline. It compares model outputs before and after each optimization step to show how optimization affects accuracy.

Overview

Evaluation runs automatically as part of np run when you define the evaluate section in run.yaml. You specify which model checkpoints to evaluate and which metrics to compute. The Evaluator records results in report.json. To display the results, run np report.

Pipeline position

Supported metrics

Metric	Applicable task	Type	Description	Dataset required
SNR	All	Comparison	Signal-to-noise ratio between 2 model outputs	No
MSE	All	Comparison	Mean squared error between 2 model outputs	No
MAE	All	Comparison	Mean absolute error between 2 model outputs	No
KLD	All	Comparison	Kullback-Leibler divergence (KLD) between 2 model outputs	No
KURTOSIS	All	Comparison	Kurtosis of the output distribution shift	No
PCTL_MAX	All	Comparison	Max-percentile deviation between 2 model outputs	No
TOP1_ACCURACY	Classification	Independent	Top-1 classification accuracy	Yes
TOP5_ACCURACY	Classification	Independent	Top-5 classification accuracy	Yes
ACCURACY	Text classification	Independent	Classification accuracy	Yes
F1	Text classification	Independent	Macro-averaged F1 score	Yes
PAIR_ACCURACY	Vision-language	Independent	Pairwise accuracy on paired image comparison (MMVP)	Yes
QUESTION_ACCURACY	Vision-language	Independent	Per-question accuracy (MMVP)	Yes
RECALL@10	Text embedding	Independent	Recall@10 for retrieval-style embedding tasks	Yes
NDCG@10	Text embedding	Independent	Normalized Discounted Cumulative Gain@10	Yes
I2T_RECALL@1	Image-text matching	Independent	Image→Text Recall@1	Yes
T2I_RECALL@1	Image-text matching	Independent	Text→Image Recall@1	Yes
MAP	Object detection, face detection, audio classification	Independent	Mean Average Precision	Yes
AP50	Object detection, face detection	Independent	Average Precision at IoU 0.50	Yes
AP75	Object detection, face detection	Independent	Average Precision at IoU 0.75	Yes
MIOU_PROMPTED	Mask generation	Independent	Mean IoU for prompted (point/box-conditioned) mask generation	Yes
MASK_AP	Mask generation	Independent	Average Precision over generated masks	Yes
PPL	Text generation	Independent	Perplexity for language models	Yes
HELLASWAG	Text generation	Independent	HellaSwag commonsense-reasoning accuracy	Yes
PSNR	Super resolution	Independent	Peak Signal-to-Noise Ratio	Yes
SSIM	Super resolution	Independent	Structural Similarity Index	Yes
WER	Speech-to-text	Independent	Word Error Rate	Yes
CER	Speech-to-text	Independent	Character Error Rate	Yes

Comparison metrics measure the difference between 2 model checkpoints (for example, original as opposed to quantized). These metrics do not require a labeled dataset.

Independent metrics measure the absolute accuracy of a single model checkpoint. These metrics require an evaluation dataset with ground-truth labels. Most independent metrics also take task-specific parameters in run.detail.yaml.

Face detection metrics are not available yet

MAP, AP50, and AP75 are the confirmed metrics for face detection (mediapipe_face) and match the object detection set. The evaluation dataset and evaluator are not implemented yet. Requests for these metrics currently fail instead of returning a result.

YAML configuration

The evaluate section in run.yaml declares which models to evaluate and which metrics to compute.

evaluate:
- model: INPUT_MODEL
  metric:
  - SNR
- model: GRAPH_OPTIMIZE
  metric:
  - SNR
- model: OUTPUT_MODEL
  metric:
  - SNR
  - TOP1_ACCURACY

Field	Description
model	Model checkpoint to evaluate. Use the reference names in the following table.
metric	List of metrics to compute

Model reference names

Name	Description
INPUT_MODEL	Original model before optimization
ADVANCED_QUANTIZE	Checkpoint after AQ step
GRAPH_OPTIMIZE	Checkpoint after GO step
OUTPUT_MODEL	Final output of the last step (after GQ)

Reference names are case-insensitive. SNR is measured relative to INPUT_MODEL by default. To change the baseline, use run.detail.yaml.

Advanced overrides (run.detail.yaml)

run.detail.yaml mirrors the evaluate list in run.yaml by index (evaluate[0], evaluate[1], ...). Use it to override the default baseline for comparison metrics. You can also set task-specific parameters for independent metrics that delegate to an external evaluation framework, such as lm-evaluation-harness, VLMEvalKit, or Massive Text Embedding Benchmark (MTEB).

metrics is a list of single-key mappings rather than a mapping keyed by metric name. This structure preserves duplicate or repeated metric entries.

Comparison metrics: baseline override

SNR, MSE, MAE, KLD, KURTOSIS, and PCTL_MAX compare against INPUT_MODEL by default. Override the baseline with source_model and target_model:

run.detail.yaml

detail:
  evaluate[1]:                        # Corresponds to evaluate[1] in run.yaml
    metrics:
    - SNR:
        source_model: graph_optimize  # Override baseline (default: input_model)
        target_model: graph_quantize

Text generation: HELLASWAG

Delegated to lm-evaluation-harness. PPL takes no parameters.

run.detail.yaml

detail:
  evaluate[0]:
    metrics:
    - HELLASWAG:
        limit: 100                    # Number of problems to score
        num_fewshot: 0                # Number of few-shot examples (default: 0)
        apply_chat_template: false    # Set true for instruct models

Metric	Key	Type	Default	Range
HELLASWAG	limit	int	100	1-10042
	num_fewshot	int	0	0+ (prompts top out around 56 tokens, so there is no separate cap tied to few-shot count)
	apply_chat_template	bool	false	true for instruct models

Speech-to-text: WER and CER

Delegated to an internal audio adapter and speech-recognition evaluator.

run.detail.yaml

detail:
  evaluate[2]:
    metrics:
    - WER:
        limit: 100                    # Number of utterances to score
        dataset_path: <shard_path>    # Required: data_*.pt shard directory
        normalize_transcript: true    # Lowercase + strip punctuation (default: true)
    - CER:
        limit: 100
        dataset_path: <shard_path>
        normalize_transcript: true

Key	Type	Default	Range
limit	int	100	1 to the shard count
dataset_path	str	(required)	Path to the data_*.pt shard directory
normalize_transcript	bool	true	Lowercases and strips punctuation

These parameters apply identically across every speech-to-text model, including conformer_ctc, executorch_conformer, whisper_base, and qwen3_asr_0_6b. Differences in component structure, such as encoder-decoder, pass-through, or Large Audio-Language Model (LALM), do not change this parameter surface.

Normalization is not Whisper-style

Unlike the Whisper EnglishTextNormalizer, which also expands numbers and abbreviations, normalize_transcript only converts to lowercase and strips punctuation.

Vision-language: PAIR_ACCURACY and QUESTION_ACCURACY

Delegated to VLMEvalKit, evaluated on MMVP.

run.detail.yaml

detail:
  evaluate[3]:
    metrics:
    - QUESTION_ACCURACY:
        limit: 300                    # Even numbers only; 300 = the full MMVP set
        max_new_tokens: 31            # Generation length
        judge: exact_matching
        device: cuda                  # Default is cpu
        backend: auto                 # Only exposed for hf/compressed_tensors outputs
    - PAIR_ACCURACY:
        limit: 300
        max_new_tokens: 31
        judge: exact_matching
        device: cuda
        backend: auto

Key	Type	Default	Range
limit	int	300 (full set)	2-300, even only (an odd value fails an assertion in the scorer)
max_new_tokens	int	31	1-2048 (MMVP answers are a single A/B letter, so 31 is generous)
judge	str	exact_matching	exact_matching
device	str	cpu	cpu, cuda, cuda:N
backend	str	auto	auto, hf, vllm. .pt2 outputs are fixed to auto, and vllm errors; this key is only exposed at all for hf or compressed_tensors outputs

QUESTION_ACCURACY (scored per question) and PAIR_ACCURACY (both halves of a pair correct) share the same parameters and differ only in how the score is aggregated.

Metric name changed after the technical spec was written

The run.yaml and run.detail.yaml schema spec (dated 2026-07-22) still calls this metric ACCURACY. The confirmed metric decision (dated 2026-08-13) renamed it to QUESTION_ACCURACY to distinguish it from other ACCURACY metrics elsewhere in the taxonomy. VLMEvalKit's own output keys are MMVP_Overall_0 (pair accuracy) and MMVP_Average_0 (per-question accuracy). The previous parameter table above is unaffected by the rename. Only the metric name has changed. Use QUESTION_ACCURACY, not ACCURACY, unless this changes again.

Text embedding: NDCG@10 and RECALL@10

Delegated to MTEB, evaluated on Benchmarking Information Retrieval (BEIR) SciFact. @10 is part of the metric name, not a runtime setting. Other values, such as NDCG@1 and RECALL@5, are separate metrics that are not currently exposed.

run.detail.yaml

detail:
  evaluate[4]:
    metrics:
    - NDCG@10:
        batch_size: 32                # hf outputs only; pt2 is fixed at batch_size 1
        max_seq_length: 512           # hf outputs only
        mteb_model_name: null         # MTEB registry name
        trust_remote_code: false
        device: cuda                  # Default is cpu
    - RECALL@10:
        batch_size: 32
        max_seq_length: 512
        mteb_model_name: null
        trust_remote_code: false
        device: cuda

Key	Type	Default	Range
batch_size	int	8	1-256 (hf outputs only; .pt2 graphs are fixed at batch 1, so this is ignored there)
max_seq_length	int	model default (512)	1-512 (hf outputs only)
mteb_model_name	str | null	null	MTEB registry name
trust_remote_code	bool	false	true, false
device	str	cpu	cpu, cuda, cuda:N

Image-text matching: I2T_RECALL@1 and T2I_RECALL@1

Delegated to an internal retrieval evaluator plus MTEB, evaluated on COCO2017 (Karpathy split).

run.detail.yaml

detail:
  evaluate[5]:
    metrics:
    - I2T_RECALL@1:
        batch_size: 32                                 # hf outputs only
        mteb_model_name: openai/clip-vit-base-patch32   # Required
        device: cuda                                    # Default is cpu
    - T2I_RECALL@1:
        batch_size: 32
        mteb_model_name: openai/clip-vit-base-patch32
        device: cuda

Key	Type	Default	Range
batch_size	int	8	1-256 (hf outputs only; .pt2 is fixed at batch 1)
mteb_model_name	str	(required)	MTEB registry name
device	str	cpu	cpu, cuda, cuda:N

I2T_RECALL@1 (image→text) and T2I_RECALL@1 (text→image) are separate metrics that share this parameter table; declare both if both directions are needed.

Built-in metrics: no parameters

The Evaluator computes the following metrics directly, with no delegation to lm-evaluation-harness, VLMEvalKit, or MTEB. You do not need to configure additional parameters in run.detail.yaml. Specify the metric and dataset in run.yaml:

Task	Dataset	Metrics
Image classification	ImageNet-1K	TOP1_ACCURACY, TOP5_ACCURACY
Object detection	COCO2017	MAP, AP50, AP75
Image super resolution	BSD100 (also SET5/SET14/URBAN100)	PSNR, SSIM
Text classification	MNLI/SST2	ACCURACY

Full examples

GO + GQ

This example uses metrics that apply to all tasks. SNR does not require a dataset.

version: '1.0'
steps:
- graph_optimize: null
- graph_quantize:
  - scheme: W8-A8
profile:
- INPUT_MODEL
- GRAPH_OPTIMIZE
- OUTPUT_MODEL
evaluate:
- model: INPUT_MODEL
  metric:
  - SNR
- model: GRAPH_OPTIMIZE
  metric:
  - SNR
- model: OUTPUT_MODEL
  metric:
  - SNR

AQ + GO + GQ

This example uses a large language model (LLM) for text generation. Replace PPL with a task-appropriate metric for other model types.

version: '1.0'
steps:
- advanced_quantize:
  - algorithm: AWQ
    scheme: W8-A8
- graph_optimize: null
- graph_quantize:
  - scheme: W8-A8
profile:
- INPUT_MODEL
- ADVANCED_QUANTIZE
- GRAPH_OPTIMIZE
- OUTPUT_MODEL
evaluate:
- model: INPUT_MODEL
  metric:
  - PPL
- model: ADVANCED_QUANTIZE
  metric:
  - SNR
  - PPL
- model: OUTPUT_MODEL
  metric:
  - SNR
  - PPL

Log output

============================================================
[Phase 5] Evaluation
============================================================
2026-08-21 17:15.00 [INFO ] [Evaluator] Evaluating: input_model → graph_optimize (SNR)
2026-08-21 17:15.02 [INFO ] [Evaluator]   Input: data_000, output_0: 31.42 dB
2026-08-21 17:15.02 [INFO ] [Evaluator]   Input: data_001, output_0: 29.87 dB
2026-08-21 17:15.03 [INFO ] [Evaluator]   Input: data_002, output_0: 30.14 dB
2026-08-21 17:15.03 [INFO ] [Evaluator] Evaluating: graph_optimize → graph_quantize (SNR)
2026-08-21 17:15.05 [INFO ] [Evaluator]   Input: data_000, output_0: 22.15 dB
2026-08-21 17:15.05 [INFO ] [Evaluator]   Input: data_001, output_0: 21.73 dB
2026-08-21 17:15.06 [INFO ] [Evaluator] Evaluation complete. Results saved to report.json

Viewing results

np report

np report shows the evaluation results for the np run pipeline in a top-level table. The table displays the dataset that produced each metric in a separate column. Evaluations that you run later with np evaluate appear in a separate table for each execution. They do not appear in the run table. See CLI Reference: np report for the exact column layout.

Notes

• Omitting the evaluate section skips evaluation entirely.
• Layerwise evaluation (per-layer SNR and KLD, threshold pass or fail) is available as an expert option through np analyze sensitivity, separate from the evaluate section above.