run.detail.yaml
run.detail.yaml contains fine-grained overrides for individual step parameters. Generate it with np experiments detail in the experiment directory. It is optional. If the file is absent, all defaults apply.
All settings use the detail: top-level key.
Global
detail:
batch_size: 1 # Applied to advanced_quantize, graph_quantize, evaluate
gpu_device: cuda:0 # Available: cpu, cuda:0
gpu_device defaults to cuda:0. Set it to cpu if the worker running your job does not have a GPU available. You can also set this option when you create the project with np projects create --gpu-device <value> and override it for each experiment.
advanced_quantize
Each entry uses the index matching the step order in run.yaml (for example, advanced_quantize[0] for the first AQ step).
The following example shows activation-aware weight quantization (AWQ). Running np experiments detail with a different algorithm selected in run.yaml generates the corresponding parameter block automatically.
AWQ
detail:
advanced_quantize[0]:
weight:
granularity: GROUP # Available: CHANNEL, GROUP
group_size: 32 # Available: 32, 64, 128
calibration_method: MINMAX # Available: MINMAX, HISTOGRAM, MOVING_AVERAGE_MINMAX, PERCENTILE
activation:
granularity: TOKEN # Available: TENSOR, TOKEN
calibration_method: MINMAX # Available: MINMAX
parameters:
calibration:
num_samples: -1
clip: true # Available: true, false
graph_optimize
detail:
graph_optimize[0]:
pattern_handlers:
- PatternHandlerAdjustAdd
- PatternHandlerAdjustConstantMathInputRank
- PatternHandlerAdjustNonContiguousConstantMathInput
- PatternHandlerCanonicalizeConvolutionDefault
- PatternHandlerCombineAddRelu
- PatternHandlerCombineConvBN
- PatternHandlerCombineConvBNRelu
- PatternHandlerCombineConvRelu
- PatternHandlerCombineConvTransposeBN
- PatternHandlerCombineConvTransposeBNRelu
- PatternHandlerCombineConvTransposeRelu
- PatternHandlerCombineLinearRelu
- PatternHandlerCombineMulRelu
- PatternHandlerCombineSDPA
- PatternHandlerCommuteBNBeforeJoinOp
- PatternHandlerCommuteBNBeforeLinearOp
- PatternHandlerComposeErfFormulaIntoGelu
- PatternHandlerComposeTanhFormulaIntoGelu
- PatternHandlerDecomposeMaxDimKeepdimFalse
- PatternHandlerDecomposePReluToReluMath
- PatternHandlerDecomposeScaledDotProductAttention
- PatternHandlerFoldConstantBoolMask
- PatternHandlerFuseAffineInputToLinear
- PatternHandlerFuseBNToConv
- PatternHandlerFuseConsecutiveBatchNorm
- PatternHandlerFuseConsecutiveCast
- PatternHandlerFuseConsecutiveConcat
- PatternHandlerFuseConsecutiveIdempotent
- PatternHandlerFuseConsecutiveLogSoftmax
- PatternHandlerFuseConsecutiveReshapeTranspose
- PatternHandlerFuseConsecutiveSlice
- PatternHandlerFuseConsecutiveUnsqueeze
- PatternHandlerFoldArangeToConstant
- PatternHandlerFoldConstantPermute
- PatternHandlerFoldTensorCreationOps
- PatternHandlerFoldInplaceFillOnConstant
- PatternHandlerFuseConstantPath
- PatternHandlerFuseInputScaleToConv
- PatternHandlerFuseMathToConv
- PatternHandlerFuseMathToLinear
- PatternHandlerFusePadToConv
- PatternHandlerFusePadToPool
- PatternHandlerFuseReluPostfixToClamp
- PatternHandlerFuseReluPrefixToClamp
- PatternHandlerFuseMultiBranchReshapeTranspose
- PatternHandlerFuseQKV
- PatternHandlerFuseSplitConvConcat
- PatternHandlerFuseUnsqueezeToReduce
- PatternHandlerNormalizeSpatialMeanKeepdimFalse
- PatternHandlerRemoveGraphAssertsPass
- PatternHandlerRemoveBatchNormTracking
- PatternHandlerRemoveCommonSubexpression
- PatternHandlerRemoveDeadend
- PatternHandlerRemoveDisabledAutocastWrapper
- PatternHandlerRemoveDropout
- PatternHandlerRemoveMonotoneBeforeRanking
- PatternHandlerRemoveNoOpCast
- PatternHandlerRemoveNoOpConcat
- PatternHandlerRemoveNoOpExpand
- PatternHandlerRemoveNoOpFlatten
- PatternHandlerRemoveNoOpMath
- PatternHandlerRemoveNoOpPad
- PatternHandlerRemoveNoOpReshape
- PatternHandlerRemoveNoOpSlice
- PatternHandlerRemoveNoOpSplit
- PatternHandlerRemoveNoOpSqueeze
- PatternHandlerRemoveNoOpTranspose
- PatternHandlerRemovePositive
- PatternHandlerReplaceAddmmWithLinear
- PatternHandlerReplaceBatchNormWithConv
- PatternHandlerReplaceBitwiseRightShiftWithDiv
- PatternHandlerReplaceEinsumWithMatmul
- PatternHandlerReplaceGatherWithConv
- PatternHandlerReplaceGatherWithSliceReshape
- PatternHandlerReplaceInplaceAdd
- PatternHandlerReplaceInplaceHardswish
- PatternHandlerReplaceInplaceIndexPut
- PatternHandlerReplaceInplaceLeakyRelu
- PatternHandlerReplaceInplaceMul
- PatternHandlerReplaceInplaceRelu
- PatternHandlerReplaceInplaceSilu
- PatternHandlerReplaceMatmul2dWithLinear
- PatternHandlerReplaceMatmul3dWithLinear
- PatternHandlerReplaceMatmulWithBmm
- PatternHandlerReplaceMatmulWithLinear
- PatternHandlerReplaceTypeAsWithCast
Comment out individual handlers to exclude them from the optimization pass. This list was captured from a computer vision (CV) project and applies to rc6. Large language model (LLM) and other projects might enable a different subset, depending on the operators in the graph.
graph_quantize
detail:
graph_quantize[0]:
weight:
granularity: CHANNEL # Available: TENSOR, CHANNEL, GROUP
calibration_method: MINMAX # Available: MINMAX, HISTOGRAM, MOVING_AVERAGE_MINMAX, PERCENTILE
activation:
granularity: TENSOR # Available: TENSOR, TOKEN
calibration_method: PERCENTILE # Available: MINMAX, HISTOGRAM, MOVING_AVERAGE_MINMAX, PERCENTILE
percentile: 99.9
parameters:
calibration:
num_samples: -1
profile
detail:
profile[0]:
num_executions: 30
timeout: 60
profile[1]:
num_executions: 30
timeout: 60
profile[2]:
num_executions: 30
timeout: 60
profile[3]:
num_executions: 30
timeout: 60
| Parameter | Default | Description |
|---|---|---|
num_executions | 30 | Inference runs for latency averaging |
timeout | 60 | Maximum measurement time in seconds |
evaluate
Override comparison targets for each evaluate entry. The index corresponds to the order of the evaluate entries in run.yaml.
detail:
evaluate[0]:
metrics:
- SNR:
source_model: INPUT_MODEL # Available: INPUT_MODEL, ADVANCED_QUANTIZE, GRAPH_OPTIMIZE, GRAPH_QUANTIZE, OUTPUT_MODEL
target_model: INPUT_MODEL
evaluate[1]:
metrics:
- SNR:
source_model: INPUT_MODEL
target_model: GRAPH_OPTIMIZE
evaluate[2]:
metrics:
- SNR:
source_model: INPUT_MODEL
target_model: OUTPUT_MODEL
metrics is a list of single-key mappings, with 1 mapping for each metric. It is not a mapping keyed directly by metric name. This structure matches the output from np experiments detail.
run.detail.yaml has no shape-related fields. Generating it does not trigger any shape check. Shape is fully resolved when the model is registered (see run.yaml: Shape handling). For multi-component models, a components: block is generated under both advanced_quantize and graph_quantize. Only graph_quantize acts on this block. AQ still uses its model-level setting for every component, so per-component AQ values here have no effect yet. See run.yaml: Multi-component models.
Mixed precision
2 modes are available and you can use them together. Manual mixed precision has the highest priority, followed by automatic mixed precision (AMP), then the base scheme.
Automatic mixed precision (AMP)
Assign bit-widths to layers automatically based on sensitivity. Enable AMP in run.yaml with automatic_mixed_precision: GREEDY, then configure it in run.detail.yaml.
detail:
# GQ AMP
graph_quantize[0]:
automatic_mixed_precision:
metric: snr # [snr, mse, kld]
level: module_wise
search_algorithm: greedy # [greedy]
candidates:
- scheme: W8-A8
ratio: 0.9
- scheme: W8-A8_dynamic
ratio: 0.1
# Available candidate schemes (XNNPACK): W8-A8, W8-A8_dynamic, W4-A8_dynamic
Manual mixed precision
Override the base scheme for specific layers or operator types.
detail:
# GQ: override by NPIR module ID or operator type
graph_quantize[0]:
mixed_precision:
by_operator:
LINEAR:
weight:
dtype: INT4
by_module:
"42":
weight:
dtype: INT4
"57": null # skip quantization for this module
# AQ: override by name pattern (regex)
advanced_quantize[0]:
mixed_precision:
by_name:
"model.layers.0.*":
weight:
dtype: INT4
"model.lm_head": null