Debugging a Linux Symmetric Multi-Processing (SMP) Kernel using DS-5

In this tutorial, you will learn how to debug a Linux SMP kernel using DS-5.

Debugging a Linux Symmetric Multi-Processing (SMP) Kernel using DS-5

You can develop, port, and debug the Linux kernel for a platform using DS-5 Debugger. You can set breakpoints, view registers, view memory, single step at source level, and perform all other standard debugging functions. You can debug the Linux kernel before the MMU is enabled (with a physical memory map), and after the MMU is enabled (with a virtual memory map). DS-5 allows you to do all this and more, not just for single-core platforms, but for SMP platforms too.

In DS-5, you configure a debugging session to a target using the Debug Configuration dialog. The Linux kernel debug options available in the dialog are primarily designed for post-MMU debug. But with some extra controls, you can use it for pre-MMU debug also. This makes it possible to debug the Linux kernel at source-level, all the way from its entry point, through the pre-MMU stages. You can then continue debugging seamlessly through the MMU enable stage to the post-MMU debug stage with full kernel awareness. This is achieved without disconnecting, reconfiguring, and reconnecting to your target.

This tutorial takes you through:

This tutorial also briefly looks at logging features, single-stepping through individual cores or threads, and using the command-line to view information.


  • DS-5 installation and licensing

    You need to have Arm DS-5 installed and the license to use it. If you do not have a license, see the Getting Started tutorial to see how to obtain a license.

  • Enable telnet on your Windows host

    The FVP model uses a telnet client for its simulated serial ports. Telnet is disabled by default in newer versions of Windows. You need to enable it. Check the documentation for your Windows version for details about enabling the telnet client.

  • DS-5 Linux Distribution Example

    Download and import the DS-5 Linux Distribution Example into DS-5. This archive file contains the Linux kernel built with debug information, the vmlinux symbol file, file system, and the full source code.

    Note: If you are working on a Windows machine, you will encounter issues with file naming incompatibilities between Linux and Windows. These can be ignored for the purpose of this tutorial.

    1. Go to the DS-5 Downloads page and download the file from the DS-5 Extra Downloads section.
    2. In DS-5, select File > Import to display the Import dialog.
    3. Under General, select Existing Projects into Workspace and click Next.
    4. Select the Select archive file option, and Browse and select the file you downloaded from the Arm download center. The distribution project is available in the projects area.

    5. Click Finish to import the files into the project folder. You can view the imported files in the Project Explorer.

      Project Explorer

  • Set up the sources

    The DS-5 Linux Distribution Example archive file also contains the required sources. You can find it under <DS-5 Workspace>\distribution\kernel\linux-linaro-3.4-rc3-2012.04-0-patched\source.tar.bz2. Unzip this file into a location on your workstation.

Creating a debug configuration

Start by creating a Debug Configuration to configure connections to the Fixed Virtual Platform (FVP) model. When you create the Debug Configuration, you set up the kernel to load on the FVP.

  1. Launch DS-5.
  2. From the main menu, select Run > Debug configurations to open the Debug Configurations dialog.
  3. In the Debug Configurations dialog:
    1. Select DS-5 Debugger.
    2. Click the New launch configuration button.

      This creates a DS-5 debug configuration and displays the various tabs required to specify settings for loading your application on the target.

    3. Give a name to the debug configuration. For example, Linux SMP kernel debug.
    4. In the Connection tab:
      1. Under Select target, browse and select Arm FVP (Installed with DS-5) > VE_Cortex-A9x4 > Linux Kernel Debug > Debug Cortex-A9x4 SMP.

      2. Under Connections > Linux Kernel Debug, in the Model parameters field, enter the following as a single line:


        • The code below is listed in separate lines for reading clarity. It must be in a single line with single spaces between the commands when copied to the Model parameters field.
        • Replace <DS-5 Install folder> with the path to your DS-5 installation, and <DS-5 Workspace> with the path to your workspace.

        --data "<DS-5 Install folder>\arm\linux_distribution\kernel_ve@0x80008000"

        --data "<DS-5 Install folder>\arm\linux_distribution\rtsm_ve-cortex_a9x4.dtb@0x80f00000"

        -C motherboard.mmc.p_mmc_file="<DS-5 Install folder>\arm\linux_distribution\rootfs.image"

        -C motherboard.vfs2.mount="<DS-5 Workspace>"

        -C motherboard.terminal_3.start_telnet=false

        -C motherboard.terminal_3.mode=raw

        -C motherboard.pl011_uart3.untimed_fifos=true

        -C motherboard.terminal_3.start_port=5003

        -C motherboard.smsc_91c111.enabled=1

        -C motherboard.hostbridge.userNetworking=1

        -C motherboard.hostbridge.userNetPorts="8080=8080"

    5. In the Files tab:
      1. Under Target Configuration > Application on host to download field, click File System.
      2. Browse to your DS-5 installation folder, and select <DS-5 Install folder>\arm\linux_distribution\bootwrapper_ve.axf to load the bootwrapper.

        What is a boot wrapper?

        The boot wrapper (bootwrapper_ve.axf) is a small program that must execute before starting the kernel.

        The boot wrapper is responsible for setting up the execution environment for the kernel, including:

        • Detecting the processor type.
        • Initializing the vector table.
        • Setting up the RAM.
        • Initializing a serial port.
        • Handling the device tree and kernel command line.
        • And finally, starting the kernel.
      3. Deselect the Load symbols option. The kernel debug information is loaded later.
    6. In the Debugger tab, under Run Control, select Debug from entry point.
    7. Click Debug to start debugging.
  4. If the Confirm Perspective Switch dialog is displayed, click Yes to switch to the DS-5 Debug Perspective.

DS-5 connects to the model and displays the connection status in the Debug Control view and opens a window.

You have now created a debug configuration, loaded the files on the FVP, and started the debug connection. The next step is to start debugging the kernel.

Debugging the kernel - Pre-MMU stage

Set a temporary hardware breakpoint on the entry point into the kernel

  1. In the Commands view, enter thbreak 0x80008000 to set a temporary hardware breakpoint on the entry point into the kernel.

    0x80008000 is the base address in RAM where this Linux kernel is loaded, and is also the entry point for the kernel. It is the same address as specified in the --data "<DS-5 Install folder>\arm\linux_distribution\kernel_ve@0x80008000" parameter to the FVP model. This address is also hard-coded within the boot wrapper as the address to jump to when the boot wrapper has completed its setup tasks.

    You can view the breakpoint in the Breakpoints view.

  2. In the Debug Control view, click Continue Continue Buttonto start the target. The code is executed and stops at the breakpoint awaiting the next step.

At this stage, you might want to view additional information about system registers. By default, the Registers view displays information about each register set.

  1. You can use the Registers view to check if the Core and CP15 system registers are set, as recommended by the Booting Arm Linux reference page (For Armv8 targets use the AArch64 Linux reference page).

    For example, using the Registers view you can:

    • Check the Core is in SVC (supervisor) mode with both IRQ and FIQ interrupts disabled.

      IRQ/FIQ disabled

    • Check R0 is 0.

      R0 is Zero

    • Check R1 contains 0xFFFFFFFF to indicate this is a "device tree" platform.

      R1 is 0xFFFFFFFF

    • Check R2 contains a pointer to the device tree. Right-mouse click R2 and select Show Memory Pointed To By R2.
    • Expand CP15 > System > SCTLR > M and check that the MMU is off.

      SCTLR MMU disabled

    • Expand CP15 > System > SCTLR > C and check that the data cache is off.

      SCTLR Data Cache off

    • Expand CP15 > System > SCTLR > I and check the status of the instruction cache. Depending on the status, this could be either on or off.

      SCTLR instruction cache

  2. You can set up the Registers view to filter the displayed additional information. For example, if you only want to view information present in the CP15 group of registers, you have to create a new Register set:
    • In the Registers view, use the drop down arrow to expand the viewed register sets and select Create.
    • In the All registers section, select CP15 and click Add >. Your selected registers appear under Chosen registers. Give the register set a name such as Filtered CP15 Register Set. You can create multiple register groups if needed. To finish creating your new register sets, click OK.
    • The Registers view displays the specific register group you selected.
    • To switch between various register groups, click the All Registers drop down and select the group you want.
    • To edit or remove existing register sets, in the Registers view, under the Register Set drop down, select Manage to open the Manage Register Set dialog.
      • To edit an existing register set, select a register set, and click Edit. Edit the register set in the Create or Modify Register Set dialog.
      • To remove a register set, select the register set you want to delete and click Remove.
      • Click OK to confirm your changes.

Disable OS awareness before loading the debug symbols

Since the MMU is currently off, DS-5 Debugger's OS support must be disabled before the debug symbols in the vmlinux file are loaded. This is to avoid the debugger from trying to read the kernel structures before they are set up (and possibly resulting in data aborts).

Enter set os enabled off in the Command view and click Submit or press Enter. This disables OS awareness.

Calculate the offset

The debug symbols in the vmlinux file have virtual addresses. So, when the vmlinux file is loaded, the debugger assumes that the operating system is up and running with the MMU enabled. But this also can be used to debug pre-MMU at source-level by applying an offset when loading the symbol file.

To calculate the offset, calculate the difference between the physical (P) and virtual (V) addresses of the code. For example, if the kernel is linked at virtual address 0xC0008000 and is loaded at physical address 0x80008000, the offset is -0x40000000, which is 0x80008000 - 0xC0008000 (P-V).

For the DS-5 Linux Distribution example used in this tutorial, the kernel is linked at virtual address 0x80008000 and loaded at physical address 0x80008000, so the offset is 0x0.

Load the vmlinux file

Now, load the vmlinux file with the kernel debug symbols required for this tutorial. The file is available in the Linux distribution example that you downloaded and imported into your workspace at the start of this tutorial.

Loading the vmlinux file enables you to perform source-level debugging of the Linux kernel.

To load the file using the user interface:

  1. In the Debug Control view, select the drop-down menu and select Load.

  2. In the Load File dialog:
    1. Load Type - Select Add Symbols File
    2. Workspace - Click and select the vmlinux file. You can find this under <DS-5 Workspace>\distribution\kernel\linux-linaro-3.4-rc3-2012.04-0-patched\built\VE_V7\vmlinux
    3. Load Offset - Enter 0x0.

    4. Click OK.

      The head.S file opens in the Editor view.

      You might need to single step once to open head.S in the editor. Also, if you get any Source Not Found alerts, you have to set the substitution path to the sources you set up as part of Prerequisites. The necessary substitution path to the head.S source file should be <extraction_directory>/arch/arm/kernel.

      The Disassembly view shows the symbol stext, at the entry point for the kernel.

    Loading the vmlinux file using the command line

    You can also load the vmlinux file using the command-line. Use the add-symbol-file command to add the symbol file.

    For this tutorial, enter: add-symbol-file "<Your Project>\distribution\kernel\linux-linaro-3.4-rc3-2012.04-0-patched\built\VE_V7\vmlinux" 0x0 in the Commands view.

    Replace <Your Project> with the name of the project you created as part of Prerequisites.

You can now, at source level, set breakpoints and watchpoints, view registers, view memory, single step, and other usual debug operations at this pre-MMU stage.

Debugging the kernel - Post-MMU stage

Now that you have debugged the pre-MMU stage, we can move on to the post-MMU debugging stage.

  1. In the Commands view, enter thbreak __turn_mmu_on to set a temporary hardware breakpoint to see when the MMU is turned on.
  2. Click Continue Continue Button (or F8 on your keyboard).
  3. When the breakpoint at __turn_mmu_on is reached, note the value of SP (Stack Pointer) under the Core register group in the Registers view. It contains the virtual address of __mmap_switched and is the place that the code jumps to after the MMU is enabled. The value in this case is 0x804C005C.

    You can also view the status of the MMU. In the Registers view, expand CP15 > SCTLR > M. You can see that it is disabled at this point.

  4. At the Command prompt, enter thbreak *$SP to place a temporary hardware breakpoint on the virtual address of __mmap_switched.
  5. Click Continue Continue Button to run the code. When the breakpoint at __mmap_switched is hit, the MMU is on. You can view the updated status of the MMU in the Registers view.
  6. Enter file at the command prompt to discard currently loaded symbols.

    This is done because the old set of symbols no longer apply, now that the MMU is on.

  7. Now, reload the symbol file without the offset applied, so that post-MMU debugging will be done using unadjusted virtual addresses.

    At the Command prompt, enter: add-symbol-file "<DS-5 Workspace>\distribution\kernel\linux-linaro-3.4-rc3-2012.04-0-patched\built\VE_V7\vmlinux" in the Commands view.

    Replace <DS-5 Workspace> with the location of your workspace.

    The source code for __mmap_switched in head-common.S appears in the Editor.

You can now set breakpoints, view registers, view memory, single step, and other usual debug operations at this post-MMU stage, all at source level.

Viewing the kernel's structure

  1. After all the architecture-specific setup is done, the main C code entry into the kernel is in the start_kernel() function. This is available in the source at: <DS-5 Workspace>\distribution\kernel\linux-linaro-3.4-rc3-2012.04-0-patched\source\init\main.c.

    Enter thbreak start_kernel at the Command prompt to set a temporary hardware breakpoint on it.

  2. Click Continue Continue Button to run the code to reach the breakpoint.

    This displays the main.c source in the Editor.

    Again, if you get any Source Not Found alerts, you have to set the substitution path to the sources you set up as part of Prerequisites. The necessary substitution path to the main.c source file should be <extraction_directory>/init/.

  3. Enter set os enabled on to enable OS support in the DS-5 Debugger.
    • When OS support is enabled, the debugger also displays the Linux kernel support details. For example, you should see something similar to: Enabled Linux kernel support for version "Linux 3.4.0-rc3-ve-v7-ds5 #1 SMP Mon Oct 31 00:17:24 GMT 2016 arm".
    • You can click in the Debug Control view to switch display to show threads. Similarly, you can click  to show cores.
      In this example, you can view four cores:

      Similarly, when viewing threads, you can view Active Threads and All Threads:

    • When OS awareness is enabled and kernel symbols are loaded from the vmlinux file, DS-5 Debugger tries to access some locations in the kernel. For example, it tries to read init_nsproxy.uts_ns->name to get the kernel name and version. It also tries to set breakpoints automatically on sys_init_module() and sys_delete_module() to trap when kernel modules are inserted (insmod) and removed (rmmod).

      The debugger handles internal breakpoints automatically, you cannot enable or disable them, or control them in any way. When internal breakpoints are hit, the debugger automatically performs the corresponding action, and then restarts program execution by itself.

      You can see these breakpoints appearing in the Breakpoints view.

  4. To confirm the kernel symbols match the kernel image, enter info os-version in Commands view.

    Your operating system version should match the Linux kernel version. For example, Operating system version: Linux 3.4.0-rc3-ve-v7-ds5 #1 SMP Mon Oct 31 00:17:24 GMT 2016 arm.

    This is similar to entering output init_nsproxy.uts_ns->name, which should display an output similar to: {sysname = "Linux", nodename = "(none)", release = "3.4.0-rc3-ve-v7-ds5", version = "#1 SMP Mon Oct 31 00:17:24 GMT 2016", machine = "arm", domainname = "(none)"}.

    This might take a few moments to display because DS-5 Debugger has to process lots of debug symbols.

Multi-core features

So far, CPU_0 performed all the tasks. The next steps explore using secondary cores for debugging.

While CPU_0 is performing its tasks, the secondary cores are held in a standby state. Conceptually, you can think of the secondary cores being in a holding pen and being released one at a time, under the control of the primary core.

The secondary processors are released from their holding pen when the platform_smp_prepare_cpus() function is executed.

  1. To observe when the function is executed, set a breakpoint on platform_smp_prepare_cpus() function.

    At the Command prompt, enter thbreak platform_smp_prepare_cpus to set a temporary hardware breakpoint.

  2. Click Continue Continue Button to run the code to reach the breakpoint.

    The last call in this function writes the address of secondary startup into the system-wide flags register. This write releases the secondary CPUs, which then branch to the address (&versatile_secondary_startup = 0x804C7480C).

  3. In the Debug Control view, notice that Core_0 is in C code with its MMU on, and has started some threads, but Core_1 is still in a holding pen (WFE/LDR/CMP/BNE) assembler loop with its MMU off.

    Set a breakpoint on the versatile_secondary_startup() function. Enter thbreak versatile_secondary_startup.

  4. Click Continue Continue Button to run to the breakpoint.
  5. Expand Core_1 in the Debug Control view. You can see that it has entered versatile_secondary_startup().

There are two levels of holding pens during an SMP kernel start-up. The first one holds all secondary cores until Core_0 reaches a known good state. Once Core_0 is in a known good state, the secondary cores are all released together. They then enter into a second-level holding pen. They are then released, one at a time by Core_0.

All cores have their MMUs enabled at this point, so you can explore this second-level holding pen using symbols with virtual addresses by setting breakpoints.

  • Enter thbreak boot_secondary at the Command prompt to set a breakpoint on boot_secondary function. This is executed by Core_0.
  • Enter thbreak platform_secondary_init at the Command prompt to set a breakpoint on the platform_secondary_init function. This is executed by Core_1.

Run to each breakpoint. The platsmp.c file opens up in the Editor. As you hit each breakpoint, click on Core_0 and switch to the Stack window, then Core_1 and switch to the Stack window, to see functions under the respective cores.

Logging Features

It would be very useful to display early printk outputs in DS-5 Debugger's command window during kernel bring-up. Ideally, before the console is enabled, so no serial output is present.

For example, set a breakpoint on console_init() in main.c. Click Continue Continue Button to run to the breakpoint. No log messages are output to the console because it is not enabled yet. But you can view the entire log for activities so far.

  • Enter info os-log in the Command prompt to view the log.

  • If you want to view the log output line by line as the code is processed, enter set os log-capture on at the Command prompt.

Single-stepping individual cores or threads

You can single step through a single core or thread/process. Select either the core or the thread/process in the Debug Control view, then press F5 on your keyboard.

When you single step through a process, it might be migrated to another core. If a breakpoint is set on a process, the debugger can track the migration of process-specific breakpoints to the other core.

Using expressions to view data

You can use the Expressions view to create and work with expressions in DS-5. For example, to view the kernel's thread_info structure, in the Expressions view, enter:

(struct thread_info*)($sp_svc & ~0x1FFF).

When the view refreshes, expand the tree structure to explore its contents.