Part 5: SME2 Quarter-tile and Structured Sparsity Outer Product Instructions

This blog post is part 5 of a series about SME2. Read the earlier parts of the series:  

• Part 1: Arm Scalable Matrix Extension (SME) Introduction
• Part 2: Arm Scalable Matrix Extension (SME) Instructions
• Part 3: Matrix-matrix multiplication. Neon, SVE, and SME compared 
• Part4: Arm SME2 Introduction 

Matrix multiplication kernels rarely process only one ideal matrix shape. Real machine learning systems combine many workload types. These include large and small GEMMs, attention blocks, convolution lowering fragments, mixture-of-experts dispatch, quantized operators, and sparse weight matrices. Two bottlenecks commonly occur:

• Small matrix inefficiency: The standard SME outer-product instruction maps one pair of vectors into 1 full ZA tile update. This approach works well for large tiles however, small independent problems can underuse the tile. The new quarter-tile instructions, introduced with the Arm architecture feature FEAT_SME_MOP4, enable 1 instruction to update 4 independent quarter-tiles. The input halves can be treated as 4 separate matrix fragments.
• Sparse matrix inefficiency: In neural network inference, many model weights can be pruned to 0. If software fetches and multiplies these 0 values, it wastes memory bandwidth and multiply-accumulate bandwidth. The new structured sparsity instructions, introduced with the Arm architecture feature FEAT_SME_TMOP, enable the matrix datapath to consume compressed sparse data and metadata tags directly.

Quarter-tile outer product instructions

The quarter-tile outer product instructions generate 4 independent quarter-tile outer products from submatrices held in the half-vectors of the first and second source vectors. The 4 quarter-tile results are then added to or subtracted from the destination ZA tile.

The instruction splits the row source and the column source into 2 halves, then performs 4 quarter-tile outer products. The main instruction groups include floating-point, BFloat16, FP8, and integer variants. 

The quarter-tile instructions are available in several vector configurations:

Variant	Syntax Example	Description
Single × Single	FMOP4A ZAda.S, Zn.S, Zm.S	One Zn, one Zm — each split into halves for 4 quadrants
Multi × Single	FMOP4A ZAda.S, {Zn1.S-Zn2.S}, Zm.S	Two Zn vectors provide rows, one Zm split into halves
Single × Multi	FMOP4A ZAda.S, Zn.S, {Zm1.S-Zm2.S}	One Zn split into halves, two Zm vectors provide columns
Multi × Multi	FMOP4A ZAda.S, {Zn1.S-Zn2.S}, {Zm1.S-Zm2.S}	Two Zn for rows, two Zm for columns — maximum flexibility

A kernel can produce the 4 quarter-tiles in several ways. It can split 1 vector or combine multiple row and column vectors.

The following table shows the complete instruction family:

Mnemonic	Data Types	Description
FMOP4A / FMOP4S	FP16→FP16, FP32→FP32, FP64→FP64	Floating-point quarter-tile (non-widening)
FMOP4A / FMOP4S	FP16→FP32	Half-precision widening (2-way)
FMOP4A	FP8→FP16, FP8→FP32	FP8 widening (2-way, 4-way)
BFMOP4A / BFMOP4S	BF16→BF16, BF16→FP32	BFloat16 (non-widening and widening)
SMOP4A / SMOP4S	INT16→INT32 (2-way), INT8→INT32 (4-way)	Signed integer quarter-tile
UMOP4A / UMOP4S	INT16→INT32 (2-way), INT8→INT32 (4-way)	Unsigned integer quarter-tile
SUMOP4A / USMOP4A	INT8→INT32 (4-way)	Mixed signed/unsigned integer

Quarter-Tile Instruction Examples

The following sections use FMOP4A non-widening instructions as examples. Other quarter-tile instructions use the same mapping between source-vector halves and their corresponding quadrants.

These instructions generate 4 independent quarter-tile floating-point outer products. They use submatrices in the half-vectors of the first and second source vectors and accumulate the results in the corresponding elements of a ZA tile. Each quarter-tile outer product is generated by multiplying the SVL_S÷2 × 1 submatrix of single-precision values from the first source vectors by a 1 × SVL_S÷2 submatrix from the second source vectors.

Example 1: FMOP4A — Non-Widening FP32, Single Vectors

FMOP4A <ZAda>.S, <Zn>.S, <Zm>.S

The following diagram illustrates the operation using a 128-bit SVL for simplicity. 4 independent output quadrants are generated:

• Upper-left:   Outer product accumulating the lower half of Zn with the lower half of Zm
• Upper-right: Outer product accumulating the lower half of Zn with the upper half of Zm
• Lower-left:   Outer product accumulating the upper half of Zn with the lower half of Zm
• Lower-right: Outer product accumulating the upper half of Zn with the upper half of Zm

Example 2: FMOP4A — Non-Widening FP32, single and multiple vectors

FMOP4A <ZAda>.S, Zn.S, {<Zm1>.S-<Zm2>.S}

The following diagram illustrates the operation using a 128-bit SVL for simplicity. 4 independent output quadrants are generated:

• Upper-left:   Outer product accumulating the lower half of Zn with the lower half of Zm1
• Upper-right: Outer product accumulating the lower half of Zn with the upper half of Zm1
• Lower-left:   Outer product accumulating the upper half of Zn with the lower half of Zm2
• Lower-right: Outer product accumulating the upper half of Zn with the upper half of Zm2

Example 3: FMOP4A — Non-Widening FP32, multiple and single vectors

FMOP4A  <ZAda>.S, { <Zn1>.S-<Zn2>.S }, <Zm>.S

The following diagram illustrates the operation using a 128-bit SVL for simplicity. 4 independent output quadrants are generated:

• Upper-left:   Outer product accumulating the lower half of Zn1 with the lower half of Zm
• Upper-right: Outer product accumulating the lower half of Zn2 with the upper half of Zm
• Lower-left:   Outer product accumulating the upper half of Zn1 with the lower half of Zm
• Lower-right: Outer product accumulating the upper half of Zn2 with the upper half of Zm

Example 4: FMOP4A — Non-Widening FP32, multiple and multiple vectors

FMOP4A <ZAda>.S, {<Zn1>.S-<Zn2>.S}, {<Zm1>.S-<Zm2>.S}

The following diagram illustrates the operation using a 128-bit SVL for simplicity. 4 independent output quadrants are generated:

• Upper-left:   Outer product accumulating the lower half of Zn1 with the lower half of Zm1
• Upper-right: Outer product accumulating the lower half of Zn2 with the upper half of Zm1
• Lower-left:   Outer product accumulating the upper half of Zn1 with the lower half of Zm2
• Lower-right: Outer product accumulating the upper half of Zn2 with the upper half of Zm2

Structured sparsity outer product instructions

In machine learning, sparsity refers to 0-valued elements in weight tensors. Neural network pruning sets some weights to 0 to reduce model size and compute. Structured sparsity constrains the 0 pattern to a regular, hardware-friendly structure. A fine-grained N:M structured sparsity pattern keeps N nonzero elements in each group of M elements.

2:4 Structured Sparsity

In 2:4 structured sparsity, exactly 2 out of every 4 consecutive elements are nonzero. This pattern is widely adopted because it provides minimal accuracy loss, consistent 2× data compression, and is easy for hardware to process.

The dense matrix is compressed to half its size, with 4-bit metadata tags recording which positions are nonzero.

1:2 Structured Sparsity

In 1:2 structured sparsity, exactly 1 out of every 2 consecutive elements is nonzero.

A 2-bit metadata field for each pair selects which element is nonzero.

Introduction to Structured Sparsity Outer Product Instructions

SME2 structured sparsity outer product instructions accumulate products or sums of products. They combine all elements of the second source vector with selected elements of the first source vectors. The control vector Zk[index] selects the active elements from the first-source vectors.

The architecture distinguishes 2 classes:

• Widening structured sparsity outer products: These widen and multiply a dense matrix by a 2:4 sparse matrix and accumulate to the destination tile
• Non-widening structured sparsity outer products: These multiply a dense matrix by a 1:2 sparse matrix and accumulate to the destination tile.

The operands of those instructions include:

• ZAda: accumulator tile
• {Zn1-Zn2}: dense activations from which elements are selected
• Zm: Compressed weights
• Zk[index]: metadata controls specifying which activations are selected

For example,

FTMOPA  <ZAda>.S, { <Zn1>.S-<Zn2>.S }, <Zm>.S, <Zk>[<index>]
                    ↑                      ↑        ↑
                    Activations            Compressed Metadata
                    (2 dense vectors)      weights    tags

Structured Sparsity Outer Product Instruction examples

The important instruction families include:

Mnemonic	Sparsity	Data Types	Description
FTMOPA	1:2	FP32→FP32, FP16→FP16	Non-widening floating-point sparse OP
FTMOPA	2:4 (2-way)	FP16→FP32	Widening FP16 sparse outer product
FTMOPA	2:4 (2-way)	FP8→FP16	Widening FP8 to FP16 sparse OP
FTMOPA	2:4 (4-way)	FP8→FP32	Widening FP8 to FP32 sparse OP
BFTMOPA	1:2 / 2:4	BF16→BF16 / BF16→FP32	BFloat16 sparse outer product
STMOPA	2:4 (2-way)	INT16→INT32	Signed integer sparse, 2-way widening
STMOPA	2:4 (4-way)	INT8→INT32	Signed integer sparse, 4-way widening
UTMOPA	2:4	UINT16→INT32, UINT8→INT32	Unsigned integer sparse
SUTMOPA / USTMOPA	2:4 (4-way)	Mixed INT8→INT32	Mixed signed/unsigned sparse

Some examples are provided in the following sections.

Example 1: FTMOPA ( Non-Widening FP32)

FTMOPA  <ZAda>.S, { <Zn1>.S-<Zn2>.S }, <Zm>.S, <Zk>[<index>]

This instruction performs a 1:2 sparse outer product. Zn1 and Zn2 hold dense activation rows. Zm holds compressed weights that contain only nonzero values. Zk[index] provides 2-bit control fields that select which activation element multiplies each weight.

This instruction generates a floating-point outer product from the first and second source vectors. It multiplies the selected 1-in-2 elements from the dense submatrices in the 2 first source vectors by the corresponding elements from the compressed sparse submatrix in the second source vector. This instruction accumulates the results in the destination ZA tile.

The outer product is generated by multiplying the selected 1-in-2 single-precision value from each overlapping 32-bit containers of the 2 SVLS×1 submatrices in the first source vectors. Each selected value is multiplied by the single-precision value from the corresponding 32-bit container of the compressed 1×SVLS submatrix in the second source vector.

The 2-bit controls in the indexed segment of the control vector register select the 1-in-2 floating-point value from the first source vectors. If the control bit for an element in the first source vectors is 0, that element is discarded and does not contribute to the result. If both bits of the 2-bit control are 1, only the element corresponding to the least significant bit is selected.

The following diagram illustrates this operation using a 128-bit SVL for simplicity.

The diagram shows only 4 of the 16 elements in the ZAda.S tile. The remaining elements are computed in the same way. A 2-bit control field selects activations from Zn1 or Zn2 for 1 column of ZAda.S. With a 128-bit SVL, the bits in Zk are arranged into, 8-bit groups. The index in the instruction specifies which control group is used. This example uses index=1.

 

Example 2: FTMOPA (widening, 2-way, FP16 to FP32)

FTMOPA  <ZAda>.S, { <Zn1>.H-<Zn2>.H }, <Zm>.H, <Zk>[<index>]

It performs a 2:4 sparse sum of outer products. Zn1 and Zn2 hold dense activation rows. Zm holds compressed weights that contain only nonzero values. Zk[index] provides 4-bit control fields that select which activation element multiplies each weight.

This instruction generates a half-precision sum of outer products. It multiplies the selected 2-in-4 elements from the dense submatrices in the 2 first source vectors by the corresponding elements of the compressed sparse submatrix in the second source vector. The results accumulate in the corresponding elements of a ZA tile with 32-bit elements.

The 4-bit controls in the indexed segment of the control vector register select the 2-in-4 half-precision values from overlapping 32-bit containers of the first source vectors. If the control bit for an element in the first source vectors is 0, that element is discarded and does not contribute to the sum of products. If more than 2 bits of the 4-bit control are 1, only the elements corresponding to the least 2 significant bits are selected.

The following diagram illustrates the operation using a 128-bit SVL for simplicity.

The diagram shows only 4 of the 16 elements in the ZAda.S tile. The remaining elements are computed in the same way. A 4-bit control field selects activations from Zn1 or Zn2 for 1 column of ZAda.S. With a 128-bit SVL, the bits in Zk are arranged into16-bit groups. The index in the instruction specifies which control group is used. This example uses index=0 .

Example 3: UTMOPA (widening, 4-way, UINT8 to INT32)

UTMOPA  <ZAda>.S, { <Zn1>.B-<Zn2>.B }, <Zm>.B, <Zk>[<index>]

It performs a 2:4 sparse sum of outer products. Zn1 and Zn2 hold dense activation rows. Zm holds compressed weights that contain only nonzero values. Zk[index] provides 8-bit control fields that select which activation element multiplies each weight.

This instruction generates unsigned integer sum of outer products. It multiplies the selected 2-in-4 elements from the dense submatrices in the 2 first source vectors by the corresponding elements of the compressed sparse submatrix in the second source vector. The results accumulate in the corresponding elements of a ZA tile with 32-bit elements.

The sum of outer products uses 2 sets of selected 2-in-4 8-bit unsigned values from each overlapping 32-bit container of the two SVL_S×4 submatrices in the first source vectors. These values are multiplied by the four 8-bit unsigned values from the corresponding 32-bit container of the 4×SVL_S submatrix in the second source vector. The 4 selected elements from the overlapping 32-bit containers of the first source vectors correspond to 2 sets of 2-in-4 elements. These elements come from the rows of 2 SVL_S×4 sub-matrices. Each 32-bit container of the second source vector holds 4 column elements from a compressed 4×SVLS submatrix.

Pairs of 4-bit controls in the indexed segment of the control vector register select the 2 sets of 2-in-4 8-bit unsigned values from overlapping 32-bit containers of the first source vectors. If the control bit for an element in the first source vectors is 0, that element is discarded and does not contribute to the sum of products. If more than 2 bits of the 4-bit control for each 32-bit container are 1, only the elements corresponding to the 2 least significant bits are selected.

The following diagram illustrates the operation using a 128-bit SVL for simplicity.

The diagram shows only 1 of the 16 elements in the ZAda.S tile. The remaining elements are computed in the same way. An 8-bit control field selects activations from Zn1 or Zn2 for 1 column of ZAda.S. With a 128-bit SVL, the bits in Zk are arranged into 32-bit groups. The 'index' in the instruction specifies which control group is used. This example uses index=0.

Conclusion

Quarter-tile outer product instructions improve SME2 efficiency for small or blocked matrix problems. They enable 1 instruction to update 4 independent quarter-tiles in ZA.

Structured sparsity outer product instructions improve sparse ML inference by consuming compressed sparse values and metadata directly. This reduces memory traffic and avoids multiply-accumulates that would not change the result.

Together, FEAT_SME_MOP4 and FEAT_SME_TMOP make the SME2 programming model better suited to small, dense, quantized, and sparse matrix workloads in modern AI software.