Skip to main content

Graph Optimizer

Graph Optimizer (GO) applies pattern-based transformations to the computation graph at the intermediate representation (NPIR) level. It restructures, simplifies, and rearranges operators to improve hardware execution efficiency without changing the mathematical output of the model.

Overview​

Graph optimization runs after model conversion to NPIR. If you use AQ, graph optimization runs after AQ. The optimizer automatically selects a set of PatternHandlers based on the target device. It verifies each PatternHandler after application. Each PatternHandler is hardware-aware.

Pipeline position​

No parameters are required in run.yaml. The optimizer automatically selects and applies the appropriate patterns for the target hardware.

Core features​

Hardware-aware PatternHandler​

  • Selects device-specific pattern handlers based on the target set at project creation
  • Applies all eligible patterns and verifies the output after each transformation
  • Supports user-defined custom optimizations (advanced use)

Supported optimization patterns​

The optimizer includes more than 120 pattern handlers across the following categories. Representative examples are listed below.

Fuse​

Combines multiple operators into a single mathematically equivalent operator to improve speed.

  • Conv + BN → Conv
  • Slice + Slice + ... + Slice → Slice
  • Linear + Math → Linear

Decompose​

If the graph contains an operator that the neural processing unit (NPU) does not support, Graph Optimizer decomposes it into multiple supported operators. The NPU can then execute the resulting operators.

  • LayerNorm → mean, variance, normalize, scale, shift
  • SiLU → sigmoid + multiply
  • Softmax → exp, sum, div (with numerical stability)
  • ScaledDotProductAttention → matmul, scale, mask, softmax, matmul
  • GLU → split + sigmoid + multiply
  • GroupNorm → reshape + primitive normalization ops

Replace​

Replaces unsupported or problematic operators with mathematically equivalent operators supported by the NPU.

  • Einsum → Matmul
  • Slice (under certain conditions) → Conv
  • BatchNorm → Conv
  • In-place operators → Out-of-place operators

Remove​

Removes unnecessary operators that do not affect the execution results of the graph.

  • Dead-end operators → remove
  • No-op operators → remove

Combine​

Applies a special configuration called a "fused module" to the graph. This configuration does not modify the graph. It marks information used for quantization in the later stage.

In GO, "combine" refers to the NPIR "fused module." This terminology distinguishes it from "fuse," which is already used in general graph optimization research.

  • Conv2d + BatchNorm + ReLU → mark as a combined op
  • ConvTranspose + BatchNorm → mark as a combined op
  • Linear + ReLU and Linear + Hardtanh → mark as a combined op
  • Add + ReLU and Mul + ReLU → mark as a combined op
  • LayerNorm formula (mean, variance, normalize, scale, shift) → mark as a combined op

In addition to these general-purpose patterns, GO automatically selects hardware-specific pattern handlers based on the target device configured when you create the project.

YAML configuration​

Graph Optimize requires no parameters:

steps:
- graph_optimize: null
- graph_quantize:
  - scheme: W8-A8
    automatic_mixed_precision: null  # Available: GREEDY

Log output​

============================================================
[1/4] Graph Optimize
============================================================
2026-08-21 16:05.00 [INFO ] [Workflow] Graph Optimize started
2026-08-21 16:08.22 [DONE ] [Workflow] Graph Optimize completed (3m 21s)
  Applied 6 patterns (3 loops):
  Loop 1:
   - Remove Batch Norm Tracking (53)
   - Replace Inplace Add (16)
   - Replace Inplace Relu (49)
   - Fuse BN To Conv (53)
  Loop 2:
   (no patterns applied)
  Loop 3:
   - Combine Add Relu (16)
   - Combine Conv Relu (33)

Performance benefits​

  • Reduced operator count: Folding and removal passes eliminate redundant computations
  • Lower memory usage: Fused operators reduce intermediate activation storage
  • Better hardware utilization: Device-specific patterns exploit hardware-native capabilities
  • Faster quantization: A simpler, cleaner graph leads to more stable calibration in GQ