Debugging a Linux Symmetric Multi-Processing (SMP) Kernel using DS-5

Start by creating a Debug Configuration to configure connections to the Fixed Virtual Platform (FVP) model. When you create the Debug Configuration, you set up the kernel to load on the FVP.

1. Launch DS-5.
2. From the main menu, select Run > Debug configurations to open the Debug Configurations dialog.
3. In the Debug Configurations dialog:
   1. Select DS-5 Debugger.
   2. Click the New launch configuration button.

      
      
      This creates a DS-5 debug configuration and displays the various tabs required to specify settings for loading your application on the target.
      
      
      

   3. Give a name to the debug configuration. For example, Linux SMP kernel debug.
   4. In the Connection tab:
      1. Under Select target, browse and select Arm FVP (Installed with DS-5) > VE_Cortex-A9x4 > Linux Kernel Debug > Debug Cortex-A9x4 SMP.
         
         
         
      2. Under Connections > Linux Kernel Debug, in the Model parameters field, enter the following as a single line:

         Note

         • The code below is listed in separate lines for reading clarity. It must be in a single line with single spaces between the commands when copied to the Model parameters field.
         • Replace <DS-5 Install folder> with the path to your DS-5 installation, and <DS-5 Workspace> with the path to your workspace.

         --data "<DS-5 Install folder>\arm\linux_distribution\kernel_ve@0x80008000"

         --data "<DS-5 Install folder>\arm\linux_distribution\rtsm_ve-cortex_a9x4.dtb@0x80f00000"

         -C motherboard.mmc.p_mmc_file="<DS-5 Install folder>\arm\linux_distribution\rootfs.image"

         -C motherboard.vfs2.mount="<DS-5 Workspace>"

         -C motherboard.terminal_3.start_telnet=false

         -C motherboard.terminal_3.mode=raw

         -C motherboard.pl011_uart3.untimed_fifos=true

         -C motherboard.terminal_3.start_port=5003

         -C motherboard.smsc_91c111.enabled=1

         -C motherboard.hostbridge.userNetworking=1

         -C motherboard.hostbridge.userNetPorts="8080=8080"

   5. In the Files tab:
      1. Under Target Configuration > Application on host to download field, click File System.
      2. Browse to your DS-5 installation folder, and select <DS-5 Install folder>\arm\linux_distribution\bootwrapper_ve.axf to load the bootwrapper.

         What is a boot wrapper?

         The boot wrapper (bootwrapper_ve.axf) is a small program that must execute before starting the kernel.

         The boot wrapper is responsible for setting up the execution environment for the kernel, including:

         • Detecting the processor type.
         • Initializing the vector table.
         • Setting up the RAM.
         • Initializing a serial port.
         • Handling the device tree and kernel command line.
         • And finally, starting the kernel.
      3. Deselect the Load symbols option. The kernel debug information is loaded later.
   6. In the Debugger tab, under Run Control, select Debug from entry point.
   7. Click Debug to start debugging.
4. If the Confirm Perspective Switch dialog is displayed, click Yes to switch to the DS-5 Debug Perspective.

DS-5 connects to the model and displays the connection status in the Debug Control view and opens a window.

You have now created a debug configuration, loaded the files on the FVP, and started the debug connection. The next step is to start debugging the kernel.

Set a temporary hardware breakpoint on the entry point into the kernel

1. In the Commands view, enter thbreak 0x80008000 to set a temporary hardware breakpoint on the entry point into the kernel.

   0x80008000 is the base address in RAM where this Linux kernel is loaded, and is also the entry point for the kernel. It is the same address as specified in the --data "<DS-5 Install folder>\arm\linux_distribution\kernel_ve@0x80008000" parameter to the FVP model. This address is also hard-coded within the boot wrapper as the address to jump to when the boot wrapper has completed its setup tasks.

   You can view the breakpoint in the Breakpoints view.

   

2. In the Debug Control view, click Continue to start the target. The code is executed and stops at the breakpoint awaiting the next step.

Disable OS awareness before loading the debug symbols

Since the MMU is currently off, DS-5 Debugger's OS support must be disabled before the debug symbols in the vmlinux file are loaded. This is to avoid the debugger from trying to read the kernel structures before they are set up (and possibly resulting in data aborts).

Enter set os enabled off in the Command view and click Submit or press Enter. This disables OS awareness.

Calculate the offset

The debug symbols in the vmlinux file have virtual addresses. So, when the vmlinux file is loaded, the debugger assumes that the operating system is up and running with the MMU enabled. But this also can be used to debug pre-MMU at source-level by applying an offset when loading the symbol file.

To calculate the offset, calculate the difference between the physical (P) and virtual (V) addresses of the code. For example, if the kernel is linked at virtual address 0xC0008000 and is loaded at physical address 0x80008000, the offset is -0x40000000, which is 0x80008000 - 0xC0008000 (P-V).

For the DS-5 Linux Distribution example used in this tutorial, the kernel is linked at virtual address 0x80008000 and loaded at physical address 0x80008000, so the offset is 0x0.

Load the vmlinux file

Now, load the vmlinux file with the kernel debug symbols required for this tutorial. The file is available in the Linux distribution example that you downloaded and imported into your workspace at the start of this tutorial.

To load the file using the user interface:

1. In the Debug Control view, select the drop-down menu and select Load.
   

2. In the Load File dialog:
   1. Load Type - Select Add Symbols File
   2. Workspace - Click and select the vmlinux file. You can find this under <DS-5 Workspace>\distribution\kernel\linux-linaro-3.4-rc3-2012.04-0-patched\built\VE_V7\vmlinux
   3. Load Offset - Enter 0x0.
      
      

   4. Click OK.

      The head.S file opens in the Editor view.

      You might need to single step once to open head.S in the editor. Also, if you get any Source Not Found alerts, you have to set the substitution path to the sources you set up as part of Prerequisites. The necessary substitution path to the head.S source file should be <extraction_directory>/arch/arm/kernel.
      
      The Disassembly view shows the symbol stext, at the entry point for the kernel.

   Loading the vmlinux file using the command line

   You can also load the vmlinux file using the command-line. Use the add-symbol-file command to add the symbol file.

   For this tutorial, enter: add-symbol-file "<Your Project>\distribution\kernel\linux-linaro-3.4-rc3-2012.04-0-patched\built\VE_V7\vmlinux" 0x0 in the Commands view.

   Replace <Your Project> with the name of the project you created as part of Prerequisites.

You can now, at source level, set breakpoints and watchpoints, view registers, view memory, single step, and other usual debug operations at this pre-MMU stage.

Now that you have debugged the pre-MMU stage, we can move on to the post-MMU debugging stage.

1. In the Commands view, enter thbreak __turn_mmu_on to set a temporary hardware breakpoint to see when the MMU is turned on.
2. Click Continue (or F8 on your keyboard).
3. When the breakpoint at __turn_mmu_on is reached, note the value of SP (Stack Pointer) under the Core register group in the Registers view. It contains the virtual address of __mmap_switched and is the place that the code jumps to after the MMU is enabled. The value in this case is 0x804C005C.
   
   

   You can also view the status of the MMU. In the Registers view, expand CP15 > SCTLR > M. You can see that it is disabled at this point.
   

4. At the Command prompt, enter thbreak *$SP to place a temporary hardware breakpoint on the virtual address of __mmap_switched.
5. Click Continue to run the code. When the breakpoint at __mmap_switched is hit, the MMU is on. You can view the updated status of the MMU in the Registers view.
   
6. Enter file at the command prompt to discard currently loaded symbols.

   This is done because the old set of symbols no longer apply, now that the MMU is on.

7. Now, reload the symbol file without the offset applied, so that post-MMU debugging will be done using unadjusted virtual addresses.

   At the Command prompt, enter: add-symbol-file "<DS-5 Workspace>\distribution\kernel\linux-linaro-3.4-rc3-2012.04-0-patched\built\VE_V7\vmlinux" in the Commands view.

   Replace <DS-5 Workspace> with the location of your workspace.

   The source code for __mmap_switched in head-common.S appears in the Editor.

You can now set breakpoints, view registers, view memory, single step, and other usual debug operations at this post-MMU stage, all at source level.

1. After all the architecture-specific setup is done, the main C code entry into the kernel is in the start_kernel() function. This is available in the source at: <DS-5 Workspace>\distribution\kernel\linux-linaro-3.4-rc3-2012.04-0-patched\source\init\main.c.

   Enter thbreak start_kernel at the Command prompt to set a temporary hardware breakpoint on it.

2. Click Continue to run the code to reach the breakpoint.

   This displays the main.c source in the Editor.
   
   Again, if you get any Source Not Found alerts, you have to set the substitution path to the sources you set up as part of Prerequisites. The necessary substitution path to the main.c source file should be <extraction_directory>/init/.

3. Enter set os enabled on to enable OS support in the DS-5 Debugger.
   • When OS support is enabled, the debugger also displays the Linux kernel support details. For example, you should see something similar to: Enabled Linux kernel support for version "Linux 3.4.0-rc3-ve-v7-ds5 #1 SMP Mon Oct 31 00:17:24 GMT 2016 arm".
   • You can click in the Debug Control view to switch display to show threads. Similarly, you can click  to show cores.
     In this example, you can view four cores:
     

     
     
     Similarly, when viewing threads, you can view Active Threads and All Threads:
     
     

   • When OS awareness is enabled and kernel symbols are loaded from the vmlinux file, DS-5 Debugger tries to access some locations in the kernel. For example, it tries to read init_nsproxy.uts_ns->name to get the kernel name and version. It also tries to set breakpoints automatically on sys_init_module() and sys_delete_module() to trap when kernel modules are inserted (insmod) and removed (rmmod).
     
     The debugger handles internal breakpoints automatically, you cannot enable or disable them, or control them in any way. When internal breakpoints are hit, the debugger automatically performs the corresponding action, and then restarts program execution by itself.
     
     You can see these breakpoints appearing in the Breakpoints view.
     
     
4. To confirm the kernel symbols match the kernel image, enter info os-version in Commands view.

   Your operating system version should match the Linux kernel version. For example, Operating system version: Linux 3.4.0-rc3-ve-v7-ds5 #1 SMP Mon Oct 31 00:17:24 GMT 2016 arm.

   This is similar to entering output init_nsproxy.uts_ns->name, which should display an output similar to: {sysname = "Linux", nodename = "(none)", release = "3.4.0-rc3-ve-v7-ds5", version = "#1 SMP Mon Oct 31 00:17:24 GMT 2016", machine = "arm", domainname = "(none)"}.

   This might take a few moments to display because DS-5 Debugger has to process lots of debug symbols.

So far, CPU_0 performed all the tasks. The next steps explore using secondary cores for debugging.

While CPU_0 is performing its tasks, the secondary cores are held in a standby state. Conceptually, you can think of the secondary cores being in a holding pen and being released one at a time, under the control of the primary core.

The secondary processors are released from their holding pen when the platform_smp_prepare_cpus() function is executed.

1. To observe when the function is executed, set a breakpoint on platform_smp_prepare_cpus() function.

   At the Command prompt, enter thbreak platform_smp_prepare_cpus to set a temporary hardware breakpoint.

2. Click Continue to run the code to reach the breakpoint.

   The last call in this function writes the address of secondary startup into the system-wide flags register. This write releases the secondary CPUs, which then branch to the address (&versatile_secondary_startup = 0x804C7480C).

3. In the Debug Control view, notice that Core_0 is in C code with its MMU on, and has started some threads, but Core_1 is still in a holding pen (WFE/LDR/CMP/BNE) assembler loop with its MMU off.

   Set a breakpoint on the versatile_secondary_startup() function. Enter thbreak versatile_secondary_startup.

4. Click Continue to run to the breakpoint.
5. Expand Core_1 in the Debug Control view. You can see that it has entered versatile_secondary_startup().

There are two levels of holding pens during an SMP kernel start-up. The first one holds all secondary cores until Core_0 reaches a known good state. Once Core_0 is in a known good state, the secondary cores are all released together. They then enter into a second-level holding pen. They are then released, one at a time by Core_0.

All cores have their MMUs enabled at this point, so you can explore this second-level holding pen using symbols with virtual addresses by setting breakpoints.

• Enter thbreak boot_secondary at the Command prompt to set a breakpoint on boot_secondary function. This is executed by Core_0.
• Enter thbreak platform_secondary_init at the Command prompt to set a breakpoint on the platform_secondary_init function. This is executed by Core_1.

Run to each breakpoint. The platsmp.c file opens up in the Editor. As you hit each breakpoint, click on Core_0 and switch to the Stack window, then Core_1 and switch to the Stack window, to see functions under the respective cores.

It would be very useful to display early printk outputs in DS-5 Debugger's command window during kernel bring-up. Ideally, before the console is enabled, so no serial output is present.

For example, set a breakpoint on console_init() in main.c. Click Continue to run to the breakpoint. No log messages are output to the console because it is not enabled yet. But you can view the entire log for activities so far.

• Enter info os-log in the Command prompt to view the log.
  

• If you want to view the log output line by line as the code is processed, enter set os log-capture on at the Command prompt.

You can single step through a single core or thread/process. Select either the core or the thread/process in the Debug Control view, then press F5 on your keyboard.

When you single step through a process, it might be migrated to another core. If a breakpoint is set on a process, the debugger can track the migration of process-specific breakpoints to the other core.

You can use the Expressions view to create and work with expressions in DS-5. For example, to view the kernel's thread_info structure, in the Expressions view, enter:

(struct thread_info*)($sp_svc & ~0x1FFF).

When the view refreshes, expand the tree structure to explore its contents.