Nested virtualization

In theory, a hypervisor can be run within a virtual machine (VM). This concept is called nested virtualization:

Nested virtualization 

We refer to the first hypervisor as the Host Hypervisor, and the hypervisor within the VM as the Guest Hypervisor.

Before the release of Armv8.3-A, it was possible to run a Guest Hypervisor in a VM by running the Guest Hypervisor in EL0. However, this required a significant amount of software emulation, and was both complicated to implement and resulted in poor performance. With the features added in Armv8.3-A, it is possible to run the Guest Hypervisor in EL0. With the features added in Armv8.4-A, this process is even more efficient, although it still involves extra intelligence in the Host Hypervisor.

Guest Hypervisor access to virtualization controls

We do not want to give a Guest Hypervisor direct access to the virtualization controls. This is because giving direct access could potentially allow the VM to breach its sandbox, or to discover information about the host platform. This potential problem is similar to the issues demonstrated in the previous examples on trapping and emulating.

Guest Hypervisors run at EL1.  New controls in HCR_EL2 allow the Host Hypervisor to trap the attempts of the Guest Hypervisor to access the virtualization controls:

  • HCR_EL2.NV Enables hardware support for nested virtualization
  • HCR_EL2.NV1 Enables an extra set of traps
  • HCR_EL2.NV2 Enables re-direction to memory
  • VNCR_EL2 When NV2==1, points to a structure in memory

Armv8.3-A added the NV and NV1 controls.  Accesses to _EL2 registers from EL1 are usually undefined, and accesses would cause an exception to EL1.  The NV and NV1 bits cause EL1 accesses to _EL2 registers to trap to EL2 instead. This allows a Guest Hypervisor to run at EL1, with the Host Hypervisor at EL2 emulating some of its operations.  NV also traps ERETs from EL1.

The following diagram shows a Guest Hypervisor setting up and entering a VM:

Guest hypervisor setting up and entering a VM 
  1. Accesses to _EL2 register by the Guest Hypervisor are trapped to EL2. The Host Hypervisor records the configuration that the Guest Hypervisor is setting up.
  2. The Guest Hypervisor attempts to enter its Guest VM (the Guest VM of the Guest), and the ERET is trapped to EL2
  3. The Host Hypervisor retrieves the configuration for the Guest of the Guest and loads this configuration into the appropriate registers. Then the host hypervisor clears the NV bit and enters the Guest of the Guest.

The problem with this approach is that each individual access to an EL2 register by the Guest Hypervisor must be trapped.  Many registers are accessed when task switching between two vCPUs or VMs, and result in many traps. Each trap has the overhead of an exception entry and return.

A better solution is to capture the configuration of the EL2 registers, and only trap to the Host Hypervisor on the ERET.  This solution is possible with Armv8.4-A.  When NV2 is set, EL1 accesses to _EL2 registers are redirected to a structure in memory.  The Guest Hypervisor can read and write the registers as many times as it needs to, without a single trap.  The ERET still traps to EL2, at which time the Host Hypervisor can re-retrieve the configuration from memory.

The following diagram illustrates this concept:

ERET still traps to EL2 
  1. Accesses to _EL2 registers from the Guest Hypervisor in EL1 are redirected to a structure in memory. The location of the structure is specified by Host Hypervisor using VNCR_EL2
  2. The Guest Hypervisor attempts to enter its Guest VM, which is the Guest VM of the Guest, and the ERET is trapped to EL2.
  3. The Host Hypervisor retrieves the configuration for the Guest of the Guest, and loads them into the appropriate registers. The host hypervisor then clears the NV bit and enters the Guest of the Guest.

The advantage to this approach is that there are fewer traps, and therefore fewer entries into the Host Hypervisor.

Previous Next