Running Trusted Firmware-A on gem5

Security is increasingly becoming more relevant, as the vulnerability surface grows along with the number of connected devices. Arm has prioritized security for many years, introducing TrustZone® as a security hardware architecture in 2003. One key TrustZone feature involves the system-level hardware isolation into two worlds: Secure world and Non-secure World (Normal). Processing elements (PEs) may be configured in one of two states through the Secure Configuration Register (SCR). 

Traditional applications, general-purpose Operating Systems (OS) and Virtual Machine Monitors (VMM) run in a Rich Execution Environment (REE), which is hosted in the Non-secure World. By contrast, a Trusted Execution Environment (TEE) may be hosted in the Secure world. This provides services such as Secure Boot, credential authentication, Digital Rights Management (DRM) or encryption, which is managed by one or more Trusted OSs. It is important to remark that REE software failures and misbehaviors are isolated from impacting TEE.

A key component of TrustZone is the Secure Monitor (SM). This provides context switching between REE and TEE, while additionally handling Secure Monitor Calls (SMC), which constitute communication and service provisioning between worlds. In Armv8, a new architectural Exception level EL3 is designed for the SM to operate in. Arm provides a reference implementation of Secure world software known as Trusted Firmware (TF), including an SM implementation. This post briefly introduces TF, the gem5 simulator, motivations for TF support in gem5 and the future it enables.

Trusted Firmware-A

Trusted Firmware-A (TF-A) is Arm’s reference implementation of Secure world software for A-profile architectures. It enables Secure Boot flow models, and provides implementations for the Secure Monitor executing at Exception level 3 (EL3) as well as for several Arm low-level software interface standards, including:

• Secure World PE and other Arm IP initialization
• System Control and Management Interface (SCMI) driver for accessing System Control Processors (SCP)
• Power State Coordination Interface (PSCI) library support for power management
• SMC handling
  • Secure Payload Dispatchers (SPD) for several Trusted OSs, such as OP-TEE and Android Trusty TEE.
• Firmware updates

TF-A supports Secure Boot flow models. These are based on a hardware root-of-trust. The boot process is divided in multiple consecutive boot stages, in which each stage verifies the signature of the next one; the initial stage is usually hosted in ROM. In particular, the following stages are defined:

• BL1: Trusted ROM
• BL2: Trusted Boot Firmware
• BL31: EL3 Runtime Software
• BL32: Secure EL1-Payload
• BL33: Non-trusted Firmware

BL33 may be a traditional boot loader, including UEFI implementations such as Das U-boot or TianoCore EDK II, which are not part of TF-A. These stages may be packed in a single binary archive known as Firmware Image Package (FIP), which is later loaded in the platform.

Figure 1: Trusted Firmware boot stages

gem5 

gem5 is an event-driven system simulator designed for computer architecture research. It is widely used in academia and industry, as well as within Arm; we have teams actively working to maintain and improve the upstream distribution of the simulator.

A wide spectrum of modelling solutions are available based on accuracy to simulation speed trade-offs: Loosely Timed (LT) solutions, such as  QEMU or Arm Fast Models, provide high simulation speed environments to serve use cases such as software development, validation and integration. Cycle Accurate (CA) solutions, such as RTL simulations or Arm Cycle Models, cover fine-level IP tuning, benchmarking and selection. gem5 stands in the middle of the spectrum as an Approximately Timed (AT) solution, covering performance and power exploration of early system-level solutions.

Figure 2: Modelling solutions spectrum

gem5 can run existing real-world software, such as upstream Linux kernels, in what is known as Full System Mode. It also supports faster simulations through Syscall Emulation Mode, where the simulator itself provides kernel services, in a similar way to QEMU user-space emulation. Model parameters may be fine-tuned at simulation design time to cover different use cases based on specific micro-architectures or hardware generations.

TF-A Support in gem5

Motivations

Some of the motivations behind adding TF-A support to gem5 are:

• Performance / power exploration and early prototyping of:
  • Secure Boot flow models
  • EL3 runtime implementations
  • Secure OSs
  • Existing and new SMC calls
  • Power management
• UEFI-based boot flow compatibility in Arm platforms
  • Step towards Windows on Arm support

Starting Point

TF-A includes support for multiple Platforms. Each Platform is characterized by the set of hardware components it hosts (such as PEs, peripherals, memory devices), as well as by the location of these in its memory map.

Arm provides a set of reference Fixed Virtual Platforms (FVP) developed on Fast Models. BasePlatform is the basis for FVPs and is offered free for customers to experiment and develop their Arm-targeted software. A minimal version of BasePlatform known as Foundation is provided, also free, as the simplest solution for an Arm-enabled platform. TF-A has official support for both BasePlatform and Foundation.

gem5 also provides a set of Arm reference platforms (the VExpress_GEM5_Base family). These are loosely based on a Versatile Express RS1 platform with a slightly modified memory map to accommodate for gem5-specific peripherals. There are two variants: VExpress_GEM5_V1 and VExpress_GEM5_V2, the latter provides a GICv3 model. Unfortunately, there is no TF-A support for VExpress_GEM5_Base.

Approach

Initially, three main approaches were considered:

1. Adding VExpress_GEM5_V2 support in TF-A by porting Foundation support
2. Modifying VExpress_GEM5_V2 to be Foundation compatible
3. Adding support for Foundation platform in gem5

The first approach involves establishing VExpress_GEM5_V2 as a reference platform to justify the contribution to TF-A. This means firmware developers would have to decide between Foundation and VExpress_GEM5_V2 as their target platform, while no compatibility would be provided.

The second approach involves restructuring existing models in VExpress_GEM5_V2, such as the GICv3, as well as adding missing ones, such as extra UARTs and SP805, to accommodate the Foundation memory map. Although feasible, this breaks backwards compatibility for existing VExpress_GEM5_V2 users.

On top of avoiding the previous drawbacks, the third approach offers a path for unifying gem5 and Fast Models reference platforms, which yield additional benefits stated in the conclusions.

Highlights

From the decision above, the following contributions to gem5 have arisen:

• New VExpress_GEM5_Foundation platform model with support for unmodified TF-A implementation for Foundation.
  • New FVPBasePwrCtrl power controller model to enable bring-up of VExpress_GEM5_Foundation, including support for multiprocessor configurations through PSCI.
  • New SP805 model for both Secure and Non-secure world watchdogs.
• New ArmTrustedFirmware workload model to help load the BL1 and FIP into physical memory at simulation-design time.

To test these contributions, we have successfully brought up a system where TF-A loads upstream Das U-boot as BL33, and the latter loads upstream Linux kernel as Non-trusted EL1-Payload. Within this simulation, we have run the official TF-A tests to stress EL3 services. The fundamental Framework Validation tests are successful, along with the main PSCI services, including PE suspend and resume, PE hot plug and affinity info. We have also brought up OP-TEE OS as Secure EL1-Payload, however this has not been tested further.

The following are extra gem5 contributions resulting from the project work:

• GenericTimer model has been reworked and now includes support for memory-mapped timers, multiple timer frames and Event Stream generation.
• GICv3 WakeRequest support to enable power management through FVPBasePwrCtrl
• Improvements to semihosting as it is needed by Das U-Boot to load the Linux kernel

Usage Guide

This section introduces a step-by-step guide on bringing up a TF-A-based simulation in gem5. Please, follow the suggested order, as some build stages take artifacts from previous ones.

Make sure the build requirements for each software artifact is satisfied before following this guide. Tested on Ubuntu 20.04.

Obtain and Build Software

We create a directory to store build outputs:

mkdir -p resources/binaries resources/disks resources/semihosting

gem5 (simulator)

git clone https://gem5.googlesource.com/public/gem5 -b v20.0.0.1
cd gem5/
scons -j $(nproc) build/ARM/gem5.opt
cd -
make -C gem5/util/term

Linux (Non-trusted EL1-Payload)

git clone https://github.com/torvalds/linux.git -b v5.4
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -C linux defconfig
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -C linux -j $(nproc)
mv linux/arch/arm64/boot/Image resources/semihosting/

Das U-Boot (BL33)

git clone https://github.com/u-boot/u-boot.git -b v2020.07-rc3
make CROSS_COMPILE=aarch64-linux-gnu- -C u-boot vexpress_aemv8a_semi_defconfig

We use the default FVP configuration and the following parameter values:

make CROSS_COMPILE=aarch64-linux-gnu- -C u-boot menuconfig

• Delay in seconds before automatically booting -> 0
  • Skip the waiting period for auto-booting. Note: you can leave this as 1 and press any key during the waiting period to get into U-Boot shell to do a manual boot. Access to the shell has been tested, but not manual booting.
• Boot arguments
  • Remove "debug user_debug=31 loglevel=9"
  • Append "root=/dev/vda rw norandmaps"
• Environment
  • Environment is not stored -> Enabled
  • Environment in flash memory -> Disabled
    • No Flash memory support.
• Networking Support -> Disabled
  • No support for the Ethernet controller in the Foundation platform, so we disable network support for avoiding crash on access

make CROSS_COMPILE=aarch64-linux-gnu- -C u-boot -j $(nproc)
mv u-boot/u-boot.bin resources/binaries/

Trusted Firmware-A (BL1, BL2, BL31)

git clone https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git -b v2.3
make CROSS_COMPILE=aarch64-linux-gnu- \
     PLAT=fvp \
     BL33=../resources/binaries/u-boot.bin \
     -C trusted-firmware-a all fip
mv trusted-firmware-a/build/fvp/release/bl1.bin resources/binaries/
mv trusted-firmware-a/build/fvp/release/fip.bin resources/binaries/

Note: you may build a debug version of TF-A by passing DEBUG=1 and LOG_LEVEL=50 to the build command. You can then check gem5's terminal output to see step by step the boot process of TF-A.

Buildroot (Minimul root filesystem)

git clone https://github.com/buildroot/buildroot.git -b 2020.05-rc3
make ARCH=aarch64 BR2_JLEVEL=$(nproc) CROSS_COMPILE=aarch64-linux-gnu- \
     -C buildroot/ arm_foundationv8_defconfig

We use the default Foundation configuration and the following parameter values:

make ARCH=aarch64 BR2_JLEVEL=$(nproc) CROSS_COMPILE=aarch64-linux-gnu- \
     -C buildroot/ menuconfig

• Target Options
  • Target Architecture Variant -> cortex-A57
• Toolchain
  • Kernel Headers -> Manually specified Linux version
  • linux version -> 5.4
  • Custom kernel headers series -> 5.4.x or later
• Kernel -> Disabled
  • We build the Linux Kernel externally.
• Target packages
  • BusyBox configuration file to use?
    • package/busybox/busybox-minimal.config
  • Networking applications
    • ifupdown scripts -> Disabled
    • We do not support networking.

make ARCH=aarch64 BR2_JLEVEL=$(nproc) CROSS_COMPILE=aarch64-linux-gnu- \
     -C buildroot/
mv buildroot/output/images/rootfs.ext2 resources/disks/

Device Tree Blob (DTB)

We use gem5 auto-DTB generation:

export M5_PATH=resources/ && \
gem5/build/ARM/gem5.opt gem5/configs/example/arm/baremetal.py \
    --workload ArmTrustedFirmware \
    --disk-image resources/disks/rootfs.ext2 \
    --num-cores 4 \
    --machine-type VExpress_GEM5_Foundation \
    --dtb-gen \
    --semi-enable --semi-path resources/semihosting/
mv m5out/system.dtb resources/semihosting/devtree.dtb

Note: configurations from 1 to 4 cores are supported.

Run and Connect to the Simulation

Before continuing, check that your resources directory looks as follows:

resources
    binaries
        bl1.bin
        fip.bin
        u-boot.bin
    disks
        rootfs.ext2
    semihosting
        Image
        devtree.dtb

Now you may run the simulation:

export M5_PATH=resources/ && \
gem5/build/ARM/gem5.opt gem5/configs/example/arm/baremetal.py \
    --workload ArmTrustedFirmware \
    --disk-image resources/disks/rootfs.ext2 \
    --num-cores 4 \
    --machine-type VExpress_GEM5_Foundation \
    --semi-enable --semi-path resources/semihosting/

And connect to it:

gem5/util/term/m5term 3456

You should see the different bootloader stages and finally the Linux kernel boot process. Once finished, you’ll be presented with the shell login provided by the root filesystem; enter “root” as username and you’ll gain access.

Conclusion and Next Steps

By providing a Foundation model in gem5, we not only inherit TF-A support, but also any other Fast Models Foundation software support, existing and future. This enables Arm customers to easily integrate Fast Models and gem5 in their development process, enabling reasoning about system-level performance and power of their solutions.

Initial TF-A support has been contributed, reviewed and was published along with the first major gem5 release of 2020, v20.0.0.0. The project has comprised development of the necessary features for TF-A support in gem5, however extensive testing of these features is still required for a full verification.

TF-A provides its own Secure EL1-Payload test, which would help stress dispatchers and SMC handling. EL1-Payloads other than OP-TEE OS, such as Android Trusty TEE should also be tested. Official TF-A tests should be exhausted as to verify other features, including firmware updates or SCMI. We look forward to the gem5 community improving its support as interest grows. Should you have any questions, reach out to myself or Giacomo Travaglini.

Contact Adrian Herrera       Contact Giacomo Travaglini