Advanced Quantizer

Advanced Quantizer (AQ) applies quantization algorithms at the PyTorch eager level before the model is converted to intermediate representation (NPIR). It uses weight-aware algorithms that outperform standard post-training quantization in preserving accuracy at aggressive bit widths.

Overview

AQ runs at the beginning of the pipeline and operates directly on the Hugging Face model. When you provide a Hugging Face model, the SDK applies AQ and then runs Graph Extractor to convert the quantized model to PT2 format.

Pipeline position

note

AQ requires a Hugging Face model as input and supports large language model (LLM) optimization. AQ does not support computer vision (CV) models in .pt2 format or TorchVision models. For CV models, use Graph Optimizer and Graph Quantizer directly. See Concepts: Compatibility matrix for compatibility information about each step.

Supported algorithms

Algorithm	Description
AWQ	Activation-aware Weight Quantization: protects salient weights based on activation magnitude
AutoRound	AutoRound with sign gradient descent optimization for fine-grained rounding
RTN	Round-To-Nearest: simple baseline, fast but lower accuracy at INT4
SmoothQuant	Smooths activation outliers by migrating quantization difficulty to weights
GPTQ	Second-order weight quantization using Hessian information

Algorithm names are case-insensitive.

Configuration

Parameters

Parameter	Required	Description	Accepted values
algorithm	Yes	Quantization algorithm to apply	AWQ, AutoRound, RTN, SmoothQuant, GPTQ
scheme	Yes	Weight and activation bit width	W8-A8, W8-A8_dynamic, W4-A8_dynamic

Quantization schemes

Scheme	Weight	Activation	Characteristics
W8-A8	8-bit	8-bit	Balanced compression and accuracy
W8-A8_dynamic	8-bit	8-bit (dynamic)	Moderate compression with dynamic activation
W4-A8_dynamic	4-bit	8-bit (dynamic)	Aggressive compression; recommended for LLMs

YAML configuration

steps:
- advanced_quantize:
  - algorithm: AWQ  # Available: RTN, AWQ, AUTOROUND, SMOOTHQUANT, GPTQ
    scheme: W8-A8  # Available: W8-A8, W8-A8_dynamic, W4-A8_dynamic
- graph_optimize: null
- graph_quantize:
  - scheme: W8-A8
    automatic_mixed_precision: null  # Available: GREEDY

Advanced overrides (run.detail.yaml)

For fine-grained control over calibration behavior, create a run.detail.yaml in the same experiment directory.

run.detail.yaml

detail:
  advanced_quantize[0]:
    weight:
      granularity: GROUP  # Available: CHANNEL, GROUP (use CHANNEL for vision / XNNPACK models)
      group_size: 32      # Available: 32, 64, 128 (applies only when granularity is GROUP)
      calibration_method: MINMAX  # Available: MINMAX, HISTOGRAM, MOVING_AVERAGE_MINMAX, PERCENTILE
    activation:
      granularity: TOKEN  # Available: TENSOR, TOKEN (use TENSOR for vision / XNNPACK models)
      calibration_method: MINMAX  # Available: MINMAX
    parameters:
      calibration:
        num_samples: -1
      clip: true

Parameter	Default	Description
weight.granularity	GROUP	Weight quantization granularity. Use CHANNEL for vision or XNNPACK models, and GROUP for LLMs.
weight.group_size	32	Group size when granularity is GROUP. Available: 32, 64, 128.
weight.calibration_method	MINMAX	Algorithm to determine weight dynamic range
activation.granularity	TOKEN	Activation quantization granularity. Use TENSOR for vision or XNNPACK models, and TOKEN for LLMs.
activation.calibration_method	MINMAX	Algorithm to determine activation dynamic range. Only MINMAX is supported.

Calibration methods

Method	Behavior	When to use
MINMAX	Maps the absolute max float value to the max integer	The default works well for evenly distributed data
PERCENTILE	Clips top and bottom outliers before mapping	Use when activations have occasional spikes

See run.detail.yaml for the full parameter reference, including granularity and Mixed Precision options.

Calibration data

AQ uses its own internal calibration data from Hugging Face Hub; it does not use the dataset configured at np projects create. The np projects create command configures calibration/evaluation data for Graph Quantize only.

Log output

============================================================
[Phase 1] Advanced Quantization
============================================================
2026-08-20 14:05.00 [INFO ] [AdvancedQuantizer] Algorithm: AWQ, Scheme: W8-A8
2026-08-20 14:05.01 [INFO ] [AdvancedQuantizer] Loading HuggingFace model...
2026-08-20 14:05.10 [INFO ] [AdvancedQuantizer] Running calibration (weight: MINMAX, activation: PERCENTILE)
2026-08-20 14:06.30 [INFO ] [AdvancedQuantizer] Quantization complete
2026-08-20 14:06.30 [INFO ] [AdvancedQuantizer] Output saved: 00_advanced_quantize/artifacts/

Choosing an algorithm

• AWQ: Start here for most cases. Best accuracy at INT4, widely tested.
• AutoRound: Try when AWQ does not achieve the accuracy you need.
• RTN: Use for fast iteration. Accuracy is lower but runs quickly.
• GPTQ: High accuracy but computationally intensive. This option is good when accuracy is critical.
• SmoothQuant: Suitable for models with large activation variance.

Notes

• Graph Extractor converts the AQ output to NPIR. The output then passes through Graph Optimizer and Graph Quantizer.