In theory, a hypervisor can be run within a virtual machine (VM). This concept is called nested virtualization:
We refer to the first hypervisor as the Host Hypervisor, and the hypervisor within the VM as the Guest Hypervisor.
Before the release of Armv8.3-A, it was possible to run a Guest Hypervisor in a VM by running the Guest Hypervisor in EL0. However, this required a significant amount of software emulation, and was both complicated to implement and resulted in poor performance. With the features added in Armv8.3-A, it is possible to run the Guest Hypervisor in EL0. With the features added in Armv8.4-A, this process is even more efficient, although it still involves extra intelligence in the Host Hypervisor.
Guest Hypervisor access to virtualization controls
We do not want to give a Guest Hypervisor direct access to the virtualization controls. This is because giving direct access could potentially allow the VM to breach its sandbox, or to discover information about the host platform. This potential problem is similar to the issues demonstrated in the previous examples on trapping and emulating.
Guest Hypervisors run at EL1. New controls in
HCR_EL2 allow the Host Hypervisor to trap the attempts of the Guest Hypervisor to access the virtualization controls:
HCR_EL2.NVEnables hardware support for nested virtualization
HCR_EL2.NV1Enables an extra set of traps
HCR_EL2.NV2Enables re-direction to memory
VNCR_EL2When NV2==1, points to a structure in memory
Armv8.3-A added the NV and NV1 controls. Accesses to
_EL2 registers from EL1 are usually undefined, and accesses would cause an exception to EL1. The
NV1 bits cause EL1 accesses to
_EL2 registers to trap to EL2 instead. This allows a Guest Hypervisor to run at EL1, with the Host Hypervisor at EL2 emulating some of its operations.
NV also traps
ERETs from EL1.
The following diagram shows a Guest Hypervisor setting up and entering a VM:
- Accesses to
_EL2register by the Guest Hypervisor are trapped to EL2. The Host Hypervisor records the configuration that the Guest Hypervisor is setting up.
- The Guest Hypervisor attempts to enter its Guest VM (the Guest VM of the Guest), and the
ERETis trapped to EL2
- The Host Hypervisor retrieves the configuration for the Guest of the Guest and loads this configuration into the appropriate registers. Then the host hypervisor clears the
NVbit and enters the Guest of the Guest.
The problem with this approach is that each individual access to an EL2 register by the Guest Hypervisor must be trapped. Many registers are accessed when task switching between two vCPUs or VMs, and result in many traps. Each trap has the overhead of an exception entry and return.
A better solution is to capture the configuration of the EL2 registers, and only trap to the Host Hypervisor on the
ERET. This solution is possible with Armv8.4-A. When
NV2 is set, EL1 accesses to
_EL2 registers are redirected to a structure in memory. The Guest Hypervisor can read and write the registers as many times as it needs to, without a single trap. The
ERET still traps to EL2, at which time the Host Hypervisor can re-retrieve the configuration from memory.
The following diagram illustrates this concept:
- Accesses to
_EL2registers from the Guest Hypervisor in EL1 are redirected to a structure in memory. The location of the structure is specified by Host Hypervisor using
- The Guest Hypervisor attempts to enter its Guest VM, which is the Guest VM of the Guest, and the
ERETis trapped to EL2.
- The Host Hypervisor retrieves the configuration for the Guest of the Guest, and loads them into the appropriate registers. The host hypervisor then clears the NV bit and enters the Guest of the Guest.
The advantage to this approach is that there are fewer traps, and therefore fewer entries into the Host Hypervisor.