Virtualization Host Extensions

The following diagram shows a simplified version of the software stack and Exception level that we looked at in the section on virtualizing exceptions:

Standalone hypervisor with Armv8-A Exception levels 

You can see how a standalone hypervisor maps to the Arm Exception levels.  The hypervisor is running at EL2 and the virtual machines (VMs) at EL0/1. This situation is more problematic for hosted hypervisors, as shown in the following diagram:

Hosted hypervisor pre VHE 

Traditionally, kernels run at EL1, but the virtualization controls are in EL2. This means that most of the Host OS is at EL1, with some stub code running in EL2 to access the virtualization controls. This arrangement can be inefficient, because it may involve additional context switching.

The Virtualization Host Extensions (VHE) were introduced in Armv8.1-A.  VHE aims to allow hosted hypervisors to run efficiently on Arm platforms, by having the kernel of the Host OS run at EL2 instead of EL1.  VHE also aims to allow the same kernel binary to run at EL2 as a Host OS, or at EL1 as a Guest OS. The kernel binary can be mostly unmodified between the two cases, and only a few lines of code are needed to handle the distinction during the early boot phase.

Note: The DynamIQ processors (Cortex-A55, Cortex-A75 and Cortex-A76) support VHE.

Running the Host OS at EL2

VHE is controlled by two bits in HCR_EL2.  These bits can be summarized as:

  • E2H: Controls whether VHE is enabled.
  • TGE: When VHE is enabled, controls whether EL0 is Guest or Host.

The following table summarizes the typical settings:

Executing in:
Guest kernel (EL1) 1 0
Guest application (EL0) 1 0
Host kernel (EL2) 1 1*
Host application (EL0) 1 1

* On an exception that exits from a VM into the hypervisor, TGE would initially be 0. Software would have to set the bit before running the main part of the host kernel.

You can see these typical settings in the following diagram:

E2H and TGE combinations 
Virtual address space

The following diagram shows what the virtual address spaces of EL0/EL1 looked like before VHE was introduced:

The EL01 and EL2 virtual address spaces pre VHE 

As discussed in Memory Management, EL0/1 has two regions.  By convention, the upper region is referred to as kernel space, and the lower region is referred to as user space.  However, EL2 only has a single region at the bottom of the address range.  This difference is because, traditionally, a hypervisor would not host applications. This means that the hypervisor does not need a split between kernel space and user space.

Note: The allocation of kernel space to the upper region, and user space to the lower region, is simply a convention.  It is not mandated by the Arm architecture.

The EL0/1 virtual address space also supports Address Space Identifiers (ASID), but EL2 does not. This is because the hypervisor would not usually host applications.

To allow our Host OS to execute efficiently in EL2, we need to add the second region and ASID support. Setting HCR_EL2.E2H addresses these issues, as you can see in the following diagram:

 EL0/2 virtual address space when E2H==1

While in EL0, HCR_EL2.TGE controls which virtual address space is used: either the EL1 space or the EL2 space.  Which space is used depends on whether the application is running under the Host OS (TGE==1) or the Guest OS (TGE==0).

Re-directing register accesses

We saw in the section on Virtualizing generic timers that enabling VHE changes the layout of the EL2 virtual address space. However, we still have a problem with the configuration of the MMU.  This is because our kernel will try to access _EL1 registers, such as TTBR0_EL1, rather than _EL2 registers such as TTBR0_EL2.

To run the same binary at EL2, we need to redirect the accesses from the EL1 registers to the EL2 equivalents. Setting E2H will do this, so that accesses to _EL1 system registers are redirected to their EL2 equivalents.  This redirection illustrated in the following diagram:

The effect of E2H on system registers access at EL2 

However, this redirection leaves us with a new problem. A hypervisor still needs access to the real _EL1 registers, so that it can implement task switching.  To resolve this, a new set of register aliases are introduced with an _EL12 or _EL02 suffix.  When used at EL2, with E2H==1, these give access to the EL1 register for context switching. You can see this in the following diagram:

Access EL1 registers from EL2 when E2H==1 

Usually, the HCR_EL2.IMO/FMO/AMO bits control whether physical exceptions are routed to EL1 or EL2.  When executing in EL0 with TGE ==1, all physical exceptions are routed to EL2, unless they are routed to EL3 by SCR_EL3. This is the case regardless of the actual values of the HCR_EL2 routing bits. This is because the application is executing as a child of the Host OS, and not a Guest OS. Therefore, any exceptions should be routed to the Host OS that is running in EL2.

Previous Next