Undefined behavior sanitizers in Arm Toolchain for Embedded

Modern C/C++ compilers include comprehensive compile-time checks that catch many potential code issues. These checks help software developers identify code that could result in unwanted behavior. However, compile-time checking cannot catch errors that only occur with specific runtime values. These defects can cause crashes, undefined behavior, or data corruption. They may also cause security vulnerabilities that attackers can exploit for remote code execution or privilege escalation.

These defects are often intermittent, difficult to catch in testing, and hard to diagnose. As C/C++ compilers cannot catch these errors, the responsibility lies with the developer.
Fortunately, the Arm Toolchain for Embedded (ATfE) family uses functionality from the LLVM project. This means it has access to some useful LLVM functionality such as code and memory sanitizers. This blog post focuses on lightweight sanitizers that can run on embedded platforms and are compatible with LLVM’s minimal runtime mode.

Please note that the code snippets provided in this post are intentionally simple and include obvious defects. The code is presented as a minimal way to get started with sanitizers and look at what they are doing. For clarity and space, the assembler code simplifies the compiler output and shows only the critical instructions.

Getting started with sanitizers

Enabling sanitizers in ATfE is easy, and needs just the addition of a compiler flag:

• -fsanitize= to select the desired sanitizers

The Arm Toolchain for Embedded Professional user guide describes the available sanitizers. For the 21.1.1 releases, the information is here. The -fsanitize=undefined and -fsanitize=bounds as “superset” options to enable most of the sanitizers. In this blog post I focus on two of the most widely used sanitizers.

You also need a way to handle check violations. There are two ways you can do this:

1. For trap mode use the compiler option -fsanitize-trap=all. Every check violation emits a hardware trap instruction and stops execution on the first hit. The size impact is small, and your project does not need additional code to handle check failures and diagnostics. No runtime is needed. Trap mode is ideal where space is limited and there is no diagnostic message capability.  You can use it in production code.
2. This blog post uses the minimal runtime, It prints basic diagnostic messages instead of halting execution at the first violation. Calls to the minimal runtime can be added using the -fsanitize-minimal-runtime compiler option. The LLVM project documentation provides reference handlers, or you could implement your own handlers like this:

  void __ubsan_handle_shift_out_of_bounds_minimal() {                   
    puts("Undefined behavior detected: shift-out-of-bounds\n"); 
  }

To get started quickly I used the Armv8-Ax1_CPP_ATfE example project available in Development Studio and added code with obvious defects.

Shift overflow (shift out of bounds)

This simple function with a single left shift is enough to show our first sanitizer working in a small enough volume of code to be easily analyzed:

void TestShift()
{
    int32_t val = 0x40000000;
    val = val << 1;
}

Catching of shift overflows is enabled using -fsanitize=shift. The generated code is immediately larger. In this case, the function grows from 8 instructions to 34. Our first key learning point is that turning on sanitizers may have a significant impact on both size and speed. For this reason, sanitizers are often more suitable for test runs than production code. I have not reproduced all 34 instructions here, but the critical instructions could be condensed to:

EL3:0x000000008000268C : MOV      w8,#0x40000000
EL3:0x0000000080002690 : LSR      w8,w8,#30
EL3:0x0000000080002694 : SUBS     w8,w8,#0
EL3:0x0000000080002698 : CSET     w8,EQ
EL3:0x000000008000269C : TBNZ     w8,#0,TestShift+112 ; 0x800026A4
EL3:0x00000000800026A0 : BL       __ubsan_handle_shift_out_of_bounds_minimal

The sanitizer right shifts the initial value 30 places rather than left shifting one place. If the result is not zero, then the left shift would cause a signed overflow and the code branches to the handler function.

If you run the code with the same starting project that I did, you will find that it catches a signed overflow and branch to the handler function, but not from the test function. The overflow is caught earlier in the run, from a bit of code that sets up the GIC:

void SetPrivateIntSecurityBlock(uint32_t gicr, GICIGROUPRBits_t group)
{
    GICv3_redistributor_SGI *const gicrSGI = getgicrSGI(gicr);
    const uint32_t nbits = (sizeof group * 8) - 1;
    uint32_t groupmod;

    /*
     * get each bit of group config duplicated over all 32 bits
     */
    groupmod = (uint32_t)(((int32_t)group << (nbits - 1)) >> 31);
    group = (uint32_t)(((int32_t)group << nbits) >> 31);

The example project was written years ago to demonstrate typical boot code, so any changes require careful consideration. Signed values are used so that the top bit is duplicated into all positions by the right shift. However, using uint32_t causes a shift overflow to be caught during the left shift. Performing the left shift as unsigned and the right shift as signed avoids flagging an overflow and preserves the outcome of the shifting. However, this change requires careful consideration in a real project. This is our second key learning point: turning on sanitizers may well catch things in existing code that seems to have worked fine for years. It may catch latent bugs or it may cause you to think about unclear or fragile code. Either way it is a win, and you will end up with better code. However, there is a time and effort cost.

Another sanitizer: out of bounds array access

This function contains an obvious bug that corrupts the stack. The effect of this corruption will depend on what else is on the stack, leaving a chance that a latent defect will not be caught in testing:

void TestShift()
{
    int v[3] __attribute__((aligned(8))) = {1, 2, 3};
    for (int i = 0; i < 8; i++)
        v[i] = 0xFFFFFFFF;
}

If you’re curious about the alignment attribute, it’s there because the FVP I used enforced 8-byte alignment for the 8-byte RO read for the data to populate the first two members of the array. Changing the code changed the location of the data and meant it wasn’t always aligned, so I got intermittent run-time alignment fails. Forcing alignment of the data fixed the problem.

Adding the -fsanitize=bounds option enables array bounds checking and increases the function size from 25 instructions to 51. Stack usage also increases from 32 bytes to 64. For space and clarity, the instructions could be re-written and commented like this:

EL3:0x0000000080002570 : LDURSW   x8,[x29,#-0x14]                       ; Get the loop count, stored at X29 - 0x14
EL3:0x0000000080002574 : SUBS     x8,x8,#3                              ; Subtract the size of the array
EL3:0x0000000080002578 : B.CC     {pc}+8 ; 0x80002580                   ; Branch if Carry Clear
EL3:0x000000008000257C : BL       __ubsan_handle_out_of_bounds_minimal

It is an effective solution, and caches the problem. My first attempt at writing buggy code for this sanitizer was simpler and replaced the loop with v[5] = 0xFFFFFFFF;. The compiler flagged this as a warning at build time because it could determine the array size and index at compile time.

However, this bounds check is effective only for stack-based arrays with a size that can be statically determined. Changing the function to store the array in the heap rather than on the stack means -fsanitize=bounds can no longer catch the problem. 

In this case, the LLVM Address Sanitizer, Asan would be helpful. However, ASan is not compatible with the minimal runtime because of its complexity. The -fsanitize-minimal-runtime enables lightweight runtime instrumentation intended for resource-constrained targets, fast compile times and bare-metal environments. AddressSanitizer relies on a full shadow memory layout and complex runtime initialization. It requires the following:

• A large redzone and shadow memory (usually 1/8 of virtual memory)
• Heap interception
• Runtime tracking of allocations and metadata

This complexity cannot be reduced to a "minimal runtime" without losing its core functionality.

Impact of using sanitizers

As we have seen above, sanitizers add code. This increases image size and will slows execution. How large is the impact?

I built the Fireworks example project with -fsanitize=undefined which turns on most of the undefined behavior sanitizers except array bounds checking, and -fsanitize=bounds, which enables array bounds checking. The sanitizers caught a few places in the code where a shift of a signed integer was causing a signed overflow. In the Fireworks project, casting to unsigned before shifting or replacing the shift with a multiplication or division resolved the problem without affecting the outcome. In a real project you would need to consider this kind of change carefully.

I compared the code size (.text region) and execution time (for 1,000 iterations) at both -O0 and -O2, against a non-protected version (no checks enabled). For execution time I used the FVP cycle count. It is not perfect, but it is good enough for a comparison like this. The results were:

	-O0			-O2
	Without sanitizers	With sanitizers	Increase	Without sanitizers	With sanitizers	Increase
Code size	440,180	471,588	7%	436,724	448,748	3%
Cycle count	581,143,378	2,841,628,277	389%	164,769,025	564,932,522	243%

The size increase does not look so bad, showing slightly less than a 3% increase with common optimization enabled. However, the performance impact is much greater, with sanitizers causing a significant slowdown.

Conclusion

C/C++ compilers cannot pick up everything that might produce undefined behavior using static analysis. This behavior can lead to data corruption, crashes, or exploitable security vulnerabilities at run-time. These kinds of defects can be intermittent, difficult to catch through testing, and time-consuming to catch and analyze.

However, LLVM provides a wide range of sanitizers which that can prevent, catch, or analyze undefined behavior at runtime. We have looked at two sanitizers in this blog post and showed how they can catch issues such as unsigned signed shift overflows, array overflows, and stack corruption. In each case, we have looked briefly at what the compiler is actually doing and how code generation is being changed to provide protection.

We have also looked at some of the limitations of using the sanitizers. They cannot catch everything we might like, can add overhead in performance, code size, or stack and memory usage, and can expose existing behavior that requires analysis and fixes.

Sanitizers are not a complete solution that will instantly make code safe and secure. However, they provide valuable assistance to the developer in finding, analyzing, and preventing issues that could cause serious run-time defects.

In experimenting with sanitizers, I have found two stack-related challenges: assessing the increase in stack usage caused by sanitizers and protecting against stack corruption that sanitizers cannot catch. I will explore both these areas in a future blog post.