Diagnosing VK_ERROR_DEVICE_LOST on Arm GPUs with VK_EXT_device_fault

VK_ERROR_DEVICE_LOST indicates that the logical device is no longer available to the application. Vulkan objects previously created for the device can no longer be used.

However, the error itself usually provides very little information about the cause.

The problem might result from an invalid GPU memory access, a shader that did not complete or another issue detected while the GPU executes submitted work.

Vulkan validation layers are an important tool for debugging an application. They can detect incorrect API usage, invalid object states, synchronization problems, and other common issues. However, validation layers primarily identify problems from the CPU and Vulkan API perspective. They cannot identify every problem that occurs while a shader or workload executes on the GPU.

Arm Mali drivers support the VK_EXT_device_fault and VK_EXT_device_address_binding_report extensions to provide more diagnostic information. Together, these extensions provide more useful diagnostic information about a device-loss error.

What can cause device loss?

A device-lost error can have several causes, and the exact behavior can vary between GPU implementations. Common examples during application and driver development include:

• Out-of-bounds memory accesses in shaders
• Invalid GPU addresses
• Buffer overflows
• Incorrect resource lifetime or memory management
• Shaders that fail to terminate
• GPU timeout detection
• Out-of-memory conditions during GPU execution

Some device-loss errors can occur even when the application uses the Vulkan API correctly.

For example, a workload with a very high vertex load can exhaust an internal GPU memory resource. This issue is particularly relevant when moving workloads from desktop GPUs to mobile tile-based architectures.

For more information about this type of issue, see Vulkan Best Practices: Memory limits with Vulkan on Mali GPUs.

Other failures occur during shader execution. A shader can calculate an incorrect pointer, access an element beyond the end of a buffer, or enter a loop that does not terminate. Validation layers might not detect these issues, because the submitted Vulkan commands and bound resources can still appear valid to the API.

Getting more information with VK_EXT_device_fault

VK_EXT_device_fault allows an application to query the driver for additional information after a VK_ERROR_DEVICE_LOST result occurs.

Depending on the GPU and driver implementation, the returned information can include:

• A description of the fault
• The type of memory access involved
• One or more reported GPU virtual addresses
• Vendor-specific diagnostic information
• Additional vendor-provided binary data

This information does not necessarily identify the exact line of shader source code that caused the failure. However, this information provides details about GPU execution that are not normally available to the application.
Consider an out-of-bounds shader access. Without device-fault reporting, the application might only observe that a queue submission returned VK_ERROR_DEVICE_LOST.

With VK_EXT_device_fault, the driver can also report the GPU address involved in the fault. The address provides a starting point for identifying the accessed resource and help determine how the shader calculated the invalid location.

However, an address alone does not always provide enough information.

A value such as 0x5FF9E85200 has little meaning unless you can relate it to the application’s buffers, images, or memory allocations. The second extension provides this information

Tracking GPU addresses with VK_EXT_device_address_binding_report

VK_EXT_device_address_binding_report reports when GPU virtual address ranges are bound or unbound.For each event, an application can receive the following information:

• The base GPU address
• The size of the address range
• Whether the range is being bound or unbound

Applications can record this information with resource names, allocation labels, or engine-specific identifiers.

If a later device-fault report contains a GPU address, you can compare it with the previously reported address ranges. This comparison can help determine whether the fault occurred within a known allocation or near a particular resource.

For example, suppose the application records that a storage buffer occupies a specific GPU address range. A later device fault reports an address several megabytes beyond the start of that buffer.

You can then focus the investigation on:

• Shaders that access that storage buffer
• Index and offset calculations
• Buffer device address usage
• Descriptor contents
• Resource lifetime
• The command submission that used the buffer

This information does not automatically solve the problem however, it can significantly narrow the investigation.

Using the extensions together

The two extensions provide different types of debugging information.

VK_EXT_device_fault answers questions such as;

• What type of fault did the GPU report?
• Was a GPU virtual address associated with the failure?

VK_EXT_device_address_binding_report provides the application context needed to answer:

Which resource or allocation is related to this address?

Together, the extensions provide a practical debugging workflow:

1. Record GPU address binding and unbinding events while the application runs.

2. Associate the events with meaningful resource names or allocation labels.

3. Detect when a Vulkan operation returns VK_ERROR_DEVICE_LOST.

4. Query the available device-fault information.

5. Compare any reported fault address with the recorded address ranges

6. Investigate the shaders and submissions that accessed the matching resource.

This workflow is especially useful for complex applications that create and destroy many resources during a frame.

Demonstrating the workflow in Vulkan Samples

We added a device_fault extension sample to the Khronos Vulkan Samples repository.

The sample demonstrates how to enable both extensions, collect GPU address binding information, query device-fault data, and correlate a reported fault address with an application resource.

To make the behavior reproducible, the sample deliberately triggers an invalid memory access from a compute shader. The shader accesses a storage buffer through a buffer device address and attempts to write far beyond the valid allocation. On the tested implementation, the invalid access produces an MMU page fault and results in VK_ERROR_DEVICE_LOST.

The application then queries the available fault information. You can compare the address reported by the driver with the base address recorded for the storage buffer.

In the sample test, the difference between the buffer base address and the reported fault address matched the shader’s invalid offset. This result linked the device-lost error to the invalid storage buffer access.
The sample README explains how to:

• Enable both Vulkan extensions
• Configure the debug-utils messenger
• Receive address binding callbacks
• Query VkDeviceFaultInfoEXT
• Trigger the invalid shader access
• Calculate and verify the reported fault address

Understanding the limitations

Treat device-fault information as diagnostic evidence rather than a complete GPU crash debugger.

Not every invalid memory access generates a device-lost error. How the GPU handles an access can depend on the GPU architecture, memory mapping, or driver implementation.

Similarly, implementations can report fault addresses at different granularities. A reported address might identify the exact accessed location, a nearby address, or a larger memory region associated with the fault.
The amount of vendor-specific information can also vary. Do not assume that every implementation provides the same fields or level of detail.

The extensions do not currently provide a standard way to map a reported GPU address or instruction offset to:

• A SPIR-V instruction
• A line of shader source-code
• A specific draw or dispatch
• A specific engine object

You can therefore benefit from retaining additional debugging context. Useful practices include:

• Assigning debug names to Vulkan objects
• Labeling command-buffer regions
• Recording resources used by each submission
• Tracking GPU address ranges
• Recording resource lifetime information
• Capturing validation layer output
• Reducing failures to a smaller reproducible workload

The extensions complement these existing techniques rather than replacing them.

Turning an opaque error into an actionable clue

A device-lost error can initially provide little information about the cause. The application knows that the GPU stopped processing its work, but it might not identify the resource, shader, or submission that caused the failure.
VK_EXT_device_fault provides information from the GPU fault. VK_EXT_device_address_binding_report helps connect that information to resources managed by the application.

Together, the extensions can help you move from a general debugging question:

Why did the GPU stop processing the workload?

to a more specific question:

Which workload accessed this GPU address, and how was that address calculated?

This additional information can make a device-loss investigation easier to manage.

The new Vulkan Samples implementation provides a practical reference for integrating these extensions into an application or engine. For the full source code, technical walkthrough, and reproducible fault example, see the device_fault sample in the Khronos Vulkan Samples repository.

Khronos Vulkan Samples repository