Changes to the Hardware Design
Two hardware design changes are needed to enable SMP Linux.
- First, the CPU must be updated to the dual-core Arm Cortex-A15 processor. This requires updating the CPU model to the Cortex-A15x2 CPU. To build models using Arm IP exchange, the web-based portal that builds models directly from RTL code, use the configuration page to select the Cortex-A15 model parameters. Ensure that “Number of CPU Cores” is set to 2 as shown below:
- The second change is the addition of an extra memory which is used to communicate the starting address for the secondary core. This memory will take the place of the Versatile Express System Registers. For SMP Linux there is a register (offset 0x30) which is used to pass the jump address to the secondary CPU. For the minimal system, the only behavior needed is to provide a simple memory at the base address of the System Registers, this is system address 0x1c010000. Only offsets 0x30 and 0x34 are used and the values must be initialized to 0 because the secondary code waits for a non-zero value. When the secondary CPU sees a non-zero value it will jump to the address contained in at 0x1c010030. If this address is not 0 at startup the system will not boot properly. SoC Designer simple memory models initialize to 0 so no special handling is needed.
Once this is configured, and the model is created, it can be instantiated on the SoC Designer canvas in place of the single-core Cortex-A15 model and connected to the interrupt from the PL011 UART and to the CCI-400 in the same way.
The Linux image needs to be recompiled with SMP support. This is done using the normal kernel configuration process.
$ make ARCH=arm menuconfig
- The option to enable SMP is under the Kernel Features menu. Make sure Symmetric Multi-Processing is selected as shown below.
- After enabling SMP rebuild the kernel as before, adjusting the CROSS_COMPILE to match the prefix of your ARM cross compiler:
- The remainder of the steps to prepare the final software image are the same. For the dual-core example, name the final product a15x2-linux.axf and load this file into the simulator.
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- -j 4
This will create a new zImage which is ready for the dual-core A15 design.
Starting the Secondary CPU
When running SMP Linux it can be challenging to get the secondary CPU started. After reset, both CPUs will start running the code in boot.S which is located at 0x80000000.
- The first step is to determine if the code is running CPU0 or CPU1. This is done by reading the CPU ID register located in co-processor 15 (CP15). This register is also referred to as the Multiprocessor Affinity Register, MPIDR. It provides information about which core of an MPCore processor the code is running on, and which cluster of a multi-cluster system the code is running on. In this example we have a single cluster and two cores so the code simply identifies CPU ID 0 as the primary core and CPU ID 1 as the secondary.
- The primary core finishes the boot loader and immediately starts running Linux, while the secondary core waits in the boot loader for a jump address to be provided at address 0x1c010030. The image below shows the code for the secondary CPU:
- The primary CPU which is running Linux is responsible to release the secondary CPU by writing the jump address and sending an interrupt. In the Linux 3.13.1 kernel, this is found in arch/arm/mach-vexpress/platsmp.c at line 225. Putting a breakpoint on this line of code and single stepping will reveal the details. The underlying code will write the jump address and take care of all the details to start the secondary CPU. The well commented last line in the screenshot below gives the details:
- Below is a screenshot of the memory contents for the System Registers address range. The primary CPU has written 0xffffffff into address 0x1c010034 and then the 32-bit jump address into 0x1c010030. As this is a memory view from the perspective of the CPU, the virtual address can be entered into the memory viewer window, which is 0xf8010000.
- Messages in the boot log should indicate that 2 CPUs are running.
- Boot SMP Linux and create a Swap & Play checkpoint for a final check. Restore the checkpoint into a cycle-accurate simulation and issue the command: $ cat /proc/cpuinfo
- The terminal output will confirm that the state of the simulation was successfully transferred to the cycle accurate simulation and both cores are reported to be running.
With the addition of an extra memory model in the location of the Versatile Express System Registers and by modifying the Linux kernel configuration parameter to enable SMP you can begin working with a dual-core Arm Cortex-A15 minimal hardware design. This design supports all of the Swap & Play and benchmarking features as a single-core design and can be extended to add more hardware detail for specific design tasks as needed.