Background

The industry is moving from high-precision formats (FP32, FP16, BF16) to lower-precision data types for Machine Learning (ML) inference and training. In cloud and data center environments, lower-precision data types deliver higher throughput by packing more compute per cycle from the same platform. On edge and mobile devices, smaller formats reduce model size, memory footprint and bandwidth, which directly affects power consumption and latency.

Arm, Intel, and NVIDIA published an OCP 8-bit Floating Point Specification (OFP8). It is an interchange format that allows software systems to share the models easily. FP8 is not a future concept, it is being deployed today. It is already supported by NVIDIA’s Hopper H100 GPU, with software deployment appearing in major ML frameworks.

Arm added the FP8 support to the Arm v9 ISA and is exploring the best way to integrate the FP8 across all the ML platforms.

This blog post explains the FP8 data formats, the new Floating-point Mode Register (FPMR), and the instruction set that makes FP8 practical on Arm CPUs.

FP8 Data format

Unlike IEEE FP64, FP32, and FP16, FP8 is not a single format, it is a family. The whitepaper FP8 Formats for Deep Learning proposes an FP8 format consisting of two encodings: E4M3 and E5M2.

Why a family of formats?

With only 8 bits available, FP8 makes a trade-off between precision and dynamic range. A single encoding cannot fit all tensors equally well. Activations and weights usually have a smaller and more stable value distribution after training or calibration. They can therefore benefit from more mantissa bits, which help represent nearby values more precisely. Gradients can change much more during forward and backward passes. They can therefore benefit from more exponent bits and a larger dynamic range, which helps avoid overflow or underflow.

The two encodings enable software to choose the right format for each tensor role. Use E4M3 when precision is more important and use E5M2 when dynamic range is more important. This is the key idea from FP8, and Arm reflects the same idea in the FPMR register. With FPMR, each source operand can independently select which FP8 format it uses.

Supported data types

	E4M3	E5M2
Exponent bits	4	5
Mantissa bits	3	2
Bias	7	15
Max normal value	448	57344
Special values	NaN only	Inf and NaN

E4M3 uses 1 sign bit, 4 exponent bits, and 3 mantissa bits, with a bias of 7. E4M3 uses most bit patterns for normal finite numbers instead of saving many of them for special values. This helps E4M3 extend dynamic range, which is more useful for deep learning workloads.

E5M2 uses 1 sign bit, 5 exponent bits, and 2 mantissa bits, with a bias of 15. E5M2 follows IEEE 754 conventions for the exponent field and special values, including infinities and NaNs. Because of this, it can be viewed as IEEE FP16 with fewer mantissa bits, like how bfloat16 and TF32 can be viewed as IEEE FP32 with fewer bits. This makes conversion between E5M2 and IEEE FP16 straightforward, while still providing a wider dynamic range than E4M3

FPMR: The Floating-Point Mode Register

When hardware executes an FP8 instruction, it must know the format of each source operand, the format of the destination result, and the scale factors to apply.

Because FP8 modes can change per tensor, software may need to update the format and scaling settings frequently inside performance-critical code. For this reason, Arm introduced a dedicated control register - FPMR. FPMR is separate from FPCR. Hardware can recognize and rename FPMR writes more efficiently, making per-tensor FP8 mode changes closer to a normal register write rather than a pipeline-serializing control update.

FPMR contains the following fields:

Field	Bits	Description
F8S1	[2:0]	Format of 1st FP8 source. 0b000 = E5M2, 0b001 = E4M3.
F8S2	[5:3]	Format of 2nd FP8 source. Same encoding as F8S1.
F8D	[8:6]	Destination result Format for instructions that convert other floating-point formats to FP8. Same encoding as F8S1
OSM	[14]	Overflow Saturation for FP8 multiplication instructions.
OSC	[15]	Overflow Saturation for FP8 convert instructions.
LSCALE	[22:16]	Unsigned Log2 down-scaling factor. Result × 2^(−LSCALE).
NSCALE	[31:24]	Unsigned Log2 up-scaling factor for widening FP8 sources.
LSCALE2	[37:32]	Second down-scaling value for dual-source convert instructions.

FP8 instruction set on Arm

The 2023 and 2024 architecture extensions add FP8 instructions across Advanced SIMD, SVE and SME. The following table summarizes the architectural features and the instructions they provide.

Architectural features	Supported category	Instructions	Description
FEAT_FP8	Advanced SIMD/SVE/SME	FCVTN, FCVT, F1CVT, F2CVT, BF1CVT, BF2CVT, FSCALE	FP8 convert instructions; a floating-point scale instruction.
FEAT_FP8FMA	Advanced SIMD/SVE	FMLALB, FMLALT, FMLALLBB, FMLALLBT, FMLALLTB, FMLALLTT	FP8 widening multiply-accumulate.
FEAT_FP8DOT4	Advanced SIMD/SVE	FDOT (.S)	FP8 → FP32 4-way dot product.
FEAT_FP8DOT2	Advanced SIMD/SVE	FDOT (.H)	FP8 → FP16 2-way dot product.
FEAT_SSVE_FP8FMA	SME	FMLALB, FMLALT, FMLALLBB, FMLALLTT	FP8 widening multiply-accumulate in Streaming SVE mode.
FEAT_SSVE_FP8DOT4	SME	FDOT (.S)	FP8 → FP32 4-way dot product and accumulate in Streaming SVE mode.
FEAT_SSVE_FP8DOT2	SME	FDOT (.H)	FP8 → FP16 2-way dot product and accumulate in Streaming SVE mode.
FEAT_SME_F8F32	SME	FMOPA, FDOT, FVDOTB, FVDOTT, FMLALL	FP8 outer product and accumulate into FP32 ZA tile ; FP8 dot product and accumulate into FP32 ZA array ; FP8 widening multiply-accumulate into FP32 ZA array;
FEAT_SME_F8F16	SME	FMOPA, FDOT, FVDOT, FMLAL	FP8 outer product and accumulate into FP16 ZA tile ; FP8 dot product and accumulate into FP16 ZA array ; FP8 widening multiply-accumulate into FP16 ZA array.
FEAT_F8F32MM	Advanced SIMD/SVE	FMMLA	FP8 matrix multiply-accumulate to FP32
FEAT_F8F16MM	Advanced SIMD/SVE	FMMLA	FP8 matrix multiply-accumulate to FP16

All FP8 compute instructions read the source format from FPMR (F8S1/F8S2) and apply scaling via LSCALE/NSCALE. These widening instructions always use FP8 data as the input format. Accumulation uses a wider type, such as FP16 or FP32. This keeps the bandwidth advantage of FP8 inputs while preserving numerical stability during accumulation.

Further reading

For more detailed information about FP8, see the Arm Architecture Reference Manual for A-profile architecture and the FP8 Formats for Deep Learning paper.

In Part 2, explore FP8 instructions, from conversion and scaling to dot products, matrix multiplication, and outer products in Advanced SIMD, SVE, and SME

Part 2