Debugging a Linux Symmetric Multi-Processing (SMP) Kernel using DS-5
Debugging a Linux Symmetric Multi-Processing (SMP) Kernel using DS-5
You can develop, port, and debug the Linux kernel for a platform using DS-5 Debugger. You can set breakpoints, view registers, view memory, single step at source level, and perform all other standard debugging functions. You can debug the Linux kernel before the MMU is enabled (with a physical memory map), and after the MMU is enabled (with a virtual memory map). DS-5 allows you to do all this and more, not just for single-core platforms, but for SMP platforms too.
In DS-5, you configure a debugging session to a target using the Debug Configuration dialog. The Linux kernel debug options available in the dialog are primarily designed for post-MMU debug. But with some extra controls, you can use it for pre-MMU debug also. This makes it possible to debug the Linux kernel at source-level, all the way from its entry point, through the pre-MMU stages. You can then continue debugging seamlessly through the MMU enable stage to the post-MMU debug stage with full kernel awareness. This is achieved without disconnecting, reconfiguring, and reconnecting to your target.
This tutorial takes you through:
- Creating a debug configuration, and loading the Linaro Linux 3.4 SMP kernel on a Cortex-A9x4 Fixed Virtual Platform (FVP) model supplied with DS-5.
- Debugging the kernel at the pre-MMU stage.
- Debugging the kernel at the post-MMU stage.
- Viewing the kernel's structure.
- Multi-core features.
This tutorial also briefly looks at logging features, single-stepping through individual cores or threads, and using the command-line to view information.
Prerequisites
- DS-5 installation and licensing
You need to have Arm DS-5 installed and the license to use it. If you do not have a license, see the Getting Started tutorial to see how to obtain a license.
- Enable telnet on your Windows host
The FVP model uses a telnet client for its simulated serial ports. Telnet is disabled by default in newer versions of Windows. You need to enable it. Check the documentation for your Windows version for details about enabling the telnet client.
- DS-5 Linux Distribution Example
Download and import the DS-5 Linux Distribution Example into DS-5. This archive file contains the Linux kernel built with debug information, the
vmlinux
symbol file, file system, and the full source code.Note: If you are working on a Windows machine, you will encounter issues with file naming incompatibilities between Linux and Windows. These can be ignored for the purpose of this tutorial.
- Go to the DS-5 Downloads page and download the Linux_distribution_example.zip file from the DS-5 Extra Downloads section.
- In DS-5, select File > Import to display the Import dialog.
- Under General, select Existing Projects into Workspace and click Next.
- Select the Select archive file option, and
Browse and select the file you downloaded from the
Arm download center. The distribution project is
available in the projects area.
- Click Finish to import the files into the project
folder. You can view the imported files in the Project
Explorer.
- Set up the sources
The DS-5 Linux Distribution Example archive file also contains the required sources. You can find it under
<DS-5 Workspace>\distribution\kernel\linux-linaro-3.4-rc3-2012.04-0-patched\source.tar.bz2
. Unzip this file into a location on your workstation.
Creating a debug configuration
Start by creating a Debug Configuration to configure connections to the Fixed Virtual Platform (FVP) model. When you create the Debug Configuration, you set up the kernel to load on the FVP.
- Launch DS-5.
- From the main menu, select Run > Debug configurations to open the Debug Configurations dialog.
- In the Debug Configurations dialog:
- Select DS-5 Debugger.
- Click the New launch configuration button.
This creates a DS-5 debug configuration and displays the various tabs required to specify settings for loading your application on the target.
- Give a name to the debug configuration. For example, Linux SMP kernel debug.
- In the Connection tab:
- Under Select target, browse and select
Arm FVP (Installed with DS-5) > VE_Cortex-A9x4 >
Linux Kernel Debug > Debug Cortex-A9x4 SMP.
- Under Connections > Linux Kernel Debug, in the
Model parameters field, enter the following
as a single line:
Note
- The code below is listed in separate lines for reading clarity. It must be in a single line with single spaces between the commands when copied to the Model parameters field.
- Replace
<DS-5 Install folder>
with the path to your DS-5 installation, and<DS-5 Workspace>
with the path to your workspace.
--data "<DS-5 Install folder>\arm\linux_distribution\kernel_ve@0x80008000"
--data "<DS-5 Install folder>\arm\linux_distribution\rtsm_ve-cortex_a9x4.dtb@0x80f00000"
-C motherboard.mmc.p_mmc_file="<DS-5 Install folder>\arm\linux_distribution\rootfs.image"
-C motherboard.vfs2.mount="<DS-5 Workspace>"
-C motherboard.terminal_3.start_telnet=false
-C motherboard.terminal_3.mode=raw
-C motherboard.pl011_uart3.untimed_fifos=true
-C motherboard.terminal_3.start_port=5003
-C motherboard.smsc_91c111.enabled=1
-C motherboard.hostbridge.userNetworking=1
-C motherboard.hostbridge.userNetPorts="8080=8080"
- Under Select target, browse and select
Arm FVP (Installed with DS-5) > VE_Cortex-A9x4 >
Linux Kernel Debug > Debug Cortex-A9x4 SMP.
- In the Files tab:
- Under Target Configuration > Application on host to download field, click File System.
- Browse to your DS-5 installation folder, and select
<DS-5 Install folder>\arm\linux_distribution\bootwrapper_ve.axf
to load the bootwrapper.What is a boot wrapper?
The boot wrapper (
bootwrapper_ve.axf
) is a small program that must execute before starting the kernel.The boot wrapper is responsible for setting up the execution environment for the kernel, including:
- Detecting the processor type.
- Initializing the vector table.
- Setting up the RAM.
- Initializing a serial port.
- Handling the device tree and kernel command line.
- And finally, starting the kernel.
- Deselect the Load symbols option. The kernel debug information is loaded later.
- In the Debugger tab, under Run Control, select Debug from entry point.
- Click Debug to start debugging.
- If the Confirm Perspective Switch dialog is displayed, click Yes to switch to the DS-5 Debug Perspective.
DS-5 connects to the model and displays the connection status in the Debug Control view and opens a window.
You have now created a debug configuration, loaded the files on the FVP, and started the debug connection. The next step is to start debugging the kernel.
Debugging the kernel - Pre-MMU stage
Set a temporary hardware breakpoint on the entry point into the kernel
- In the Commands view, enter
thbreak 0x80008000
to set a temporary hardware breakpoint on the entry point into the kernel.0x80008000
is the base address in RAM where this Linux kernel is loaded, and is also the entry point for the kernel. It is the same address as specified in the--data "<DS-5 Install folder>\arm\linux_distribution\kernel_ve@0x80008000"
parameter to the FVP model. This address is also hard-coded within the boot wrapper as the address to jump to when the boot wrapper has completed its setup tasks.You can view the breakpoint in the Breakpoints view.
- In the Debug Control view, click Continue
to start the target. The code is executed and stops at the breakpoint awaiting the next step.
At this stage, you might want to view additional information about system registers. By default, the Registers view displays information about each register set.
-
You can use the Registers view to check if the Core and CP15 system registers are set, as recommended by the Booting Arm Linux reference page (For Armv8 targets use the AArch64 Linux reference page).
For example, using the Registers view you can:
- Check the
Core
is inSVC
(supervisor) mode with bothIRQ
andFIQ
interrupts disabled. - Check
R0
is0
. - Check
R1
contains0xFFFFFFFF
to indicate this is a "device tree" platform. - Check
R2
contains a pointer to the device tree. Right-mouse clickR2
and select Show Memory Pointed To By R2.
- Expand
CP15
>System
>SCTLR
>M
and check that the MMU is off. - Expand
CP15
>System
>SCTLR
>C
and check that the data cache is off. - Expand
CP15
>System
>SCTLR
>I
and check the status of the instruction cache. Depending on the status, this could be either on or off.
- Check the
- You can set up the
Registers view to filter the displayed additional information. For
example, if you only want to view information present in the
CP15
group of registers, you have to create a new Register set:- In the Registers view, use the drop down arrow to expand the viewed register sets and select Create.
- In the All registers section, select
CP15
and click Add >. Your selected registers appear under Chosen registers. Give the register set a name such as Filtered CP15 Register Set. You can create multiple register groups if needed. To finish creating your new register sets, click OK.
- The Registers view displays the specific register group you selected.
- To switch between various register groups, click the All Registers drop down and select the group you want.
- To edit or remove existing register sets, in the Registers view, under the Register Set drop down, select Manage to open the Manage Register Set dialog.
- To edit an existing register set, select a register set, and click Edit. Edit the register set in the Create or Modify Register Set dialog.
- To remove a register set, select the register set you want to delete and click Remove.
- Click OK to confirm your changes.
- In the Registers view, use the drop down arrow to expand the viewed register sets and select Create.
Disable OS awareness before loading the debug symbols
Since the MMU is currently off, DS-5 Debugger's OS support must be disabled before
the debug symbols in the vmlinux
file are loaded. This is to avoid the
debugger from trying to read the kernel structures before they are set up (and
possibly resulting in data aborts).
Enter set os enabled off
in the Command view and click
Submit or press Enter. This disables OS
awareness.
Calculate the offset
The debug symbols in the vmlinux
file have virtual addresses. So, when
the vmlinux
file is loaded, the debugger assumes that the operating
system is up and running with the MMU enabled. But this also can be used to debug
pre-MMU at source-level by applying an offset when loading the symbol file.
To calculate the offset, calculate the difference between the physical (P) and
virtual (V) addresses of the code. For example, if the kernel is linked at virtual
address 0xC0008000
and is loaded at physical address
0x80008000
, the offset is -0x40000000
, which is
0x80008000
- 0xC0008000
(P-V).
For the DS-5 Linux Distribution example used in this tutorial, the kernel is linked
at virtual address 0x80008000
and loaded at physical address
0x80008000
, so the offset is 0x0
.
Load the vmlinux file
Now, load the vmlinux
file with the kernel debug symbols required for
this tutorial. The file is available in the Linux distribution example that you
downloaded and imported into your workspace at the start of
this tutorial.
vmlinux
file enables you to perform
source-level debugging of the Linux kernel. To load the file using the user interface:
- In the Debug Control view, select the drop-down menu and select
Load.
- In the Load File dialog:
- Load Type - Select Add Symbols File
- Workspace - Click and select the vmlinux file. You can
find this under
<DS-5 Workspace>\distribution\kernel\linux-linaro-3.4-rc3-2012.04-0-patched\built\VE_V7\vmlinux
- Load Offset - Enter
0x0
.
- Click OK.
The
head.S
file opens in the Editor view.You might need to single step once to open
head.S
in the editor. Also, if you get any Source Not Found alerts, you have to set the substitution path to the sources you set up as part of Prerequisites. The necessary substitution path to thehead.S
source file should be <extraction_directory>/arch/arm/kernel.
The Disassembly view shows the symbolstext
, at the entry point for the kernel.
Loading the vmlinux file using the command line
You can also load the vmlinux file using the command-line. Use the
add-symbol-file
command to add the symbol file.For this tutorial, enter:
add-symbol-file "<Your Project>\distribution\kernel\linux-linaro-3.4-rc3-2012.04-0-patched\built\VE_V7\vmlinux" 0x0
in the Commands view.Replace
<Your Project>
with the name of the project you created as part of Prerequisites.
You can now, at source level, set breakpoints and watchpoints, view registers, view memory, single step, and other usual debug operations at this pre-MMU stage.
Debugging the kernel - Post-MMU stage
Now that you have debugged the pre-MMU stage, we can move on to the post-MMU debugging stage.
- In the Commands view, enter
thbreak __turn_mmu_on
to set a temporary hardware breakpoint to see when the MMU is turned on. - Click Continue
(or F8 on your keyboard).
- When the breakpoint at
__turn_mmu_on
is reached, note the value ofSP
(Stack Pointer) under theCore
register group in the Registers view. It contains the virtual address of__mmap_switched
and is the place that the code jumps to after the MMU is enabled. The value in this case is0x804C005C
.
You can also view the status of the MMU. In the Registers view, expand CP15 > SCTLR > M. You can see that it is disabled at this point.
- At the Command prompt, enter
thbreak *$SP
to place a temporary hardware breakpoint on the virtual address of__mmap_switched
. - Click Continue
to run the code. When the breakpoint at
__mmap_switched
is hit, the MMU is on. You can view the updated status of the MMU in the Registers view.
- Enter
file
at the command prompt to discard currently loaded symbols.This is done because the old set of symbols no longer apply, now that the MMU is on.
- Now, reload the symbol file without the offset applied, so that post-MMU debugging will be done using unadjusted virtual addresses.
At the Command prompt, enter:
add-symbol-file "<DS-5 Workspace>\distribution\kernel\linux-linaro-3.4-rc3-2012.04-0-patched\built\VE_V7\vmlinux"
in the Commands view.Replace
<DS-5 Workspace>
with the location of your workspace.The source code for
__mmap_switched
inhead-common.S
appears in the Editor.
You can now set breakpoints, view registers, view memory, single step, and other usual debug operations at this post-MMU stage, all at source level.
Viewing the kernel's structure
- After all the architecture-specific setup is done, the main C code entry into
the kernel is in the
start_kernel()
function. This is available in the source at:<DS-5 Workspace>\distribution\kernel\linux-linaro-3.4-rc3-2012.04-0-patched\source\init\main.c
.Enter
thbreak start_kernel
at the Command prompt to set a temporary hardware breakpoint on it. - Click Continue
to run the code to reach the breakpoint.
This displays the
main.c
source in the Editor.
Again, if you get any Source Not Found alerts, you have to set the substitution path to the sources you set up as part of Prerequisites. The necessary substitution path to themain.c
source file should be <extraction_directory>/init/. - Enter
set os enabled on
to enable OS support in the DS-5 Debugger.- When OS support is enabled, the debugger also displays the Linux kernel
support details. For example, you should see something similar to:
Enabled Linux kernel support for version "Linux 3.4.0-rc3-ve-v7-ds5 #1 SMP Mon Oct 31 00:17:24 GMT 2016 arm"
. - You can click
in the Debug Control view to switch display to show threads. Similarly, you can click
to show cores.
In this example, you can view four cores:
Similarly, when viewing threads, you can view Active Threads and All Threads:
- When OS awareness is enabled and kernel symbols are loaded from the
vmlinux
file, DS-5 Debugger tries to access some locations in the kernel. For example, it tries to readinit_nsproxy.uts_ns->name
to get the kernel name and version. It also tries to set breakpoints automatically onsys_init_module()
andsys_delete_module()
to trap when kernel modules are inserted (insmod
) and removed (rmmod
).
The debugger handles internal breakpoints automatically, you cannot enable or disable them, or control them in any way. When internal breakpoints are hit, the debugger automatically performs the corresponding action, and then restarts program execution by itself.
You can see these breakpoints appearing in the Breakpoints view.
- When OS support is enabled, the debugger also displays the Linux kernel
support details. For example, you should see something similar to:
- To confirm the kernel symbols match the kernel image, enter
info os-version
in Commands view.Your operating system version should match the Linux kernel version. For example,
Operating system version: Linux 3.4.0-rc3-ve-v7-ds5 #1 SMP Mon Oct 31 00:17:24 GMT 2016 arm
.This is similar to entering
output init_nsproxy.uts_ns->name
, which should display an output similar to:{sysname = "Linux", nodename = "(none)", release = "3.4.0-rc3-ve-v7-ds5", version = "#1 SMP Mon Oct 31 00:17:24 GMT 2016", machine = "arm", domainname = "(none)"}
.This might take a few moments to display because DS-5 Debugger has to process lots of debug symbols.
Multi-core features
So far, CPU_0
performed all the tasks. The next steps explore using secondary cores for debugging.
While CPU_0
is performing its tasks, the secondary cores are held in a
standby state. Conceptually, you can think of the secondary cores being in a holding
pen and being released one at a time, under the control of the primary core.
The secondary processors are released from their holding pen when the platform_smp_prepare_cpus()
function is executed.
- To observe when the function is executed, set a breakpoint on
platform_smp_prepare_cpus()
function.At the Command prompt, enter
thbreak platform_smp_prepare_cpus
to set a temporary hardware breakpoint. - Click Continue
to run the code to reach the breakpoint.
The last call in this function writes the address of secondary startup into the system-wide flags register. This write releases the secondary CPUs, which then branch to the address
(&versatile_secondary_startup = 0x804C7480C)
. - In the Debug Control view, notice that
Core_0
is in C code with its MMU on, and has started some threads, butCore_1
is still in a holding pen (WFE/LDR/CMP/BNE
) assembler loop with its MMU off.Set a breakpoint on the
versatile_secondary_startup()
function. Enterthbreak versatile_secondary_startup
. - Click Continue
to run to the breakpoint.
- Expand
Core_1
in the Debug Control view. You can see that it has enteredversatile_secondary_startup()
.
There are two levels of holding pens during an SMP kernel start-up. The first one
holds all secondary cores until Core_0
reaches a known good state. Once
Core_0
is in a known good state, the secondary cores are all
released together. They then enter into a second-level holding pen. They are then
released, one at a time by Core_0
.
All cores have their MMUs enabled at this point, so you can explore this second-level holding pen using symbols with virtual addresses by setting breakpoints.
- Enter
thbreak boot_secondary
at the Command prompt to set a breakpoint onboot_secondary
function. This is executed byCore_0
. - Enter
thbreak platform_secondary_init
at the Command prompt to set a breakpoint on theplatform_secondary_init
function. This is executed byCore_1
.
Run to each breakpoint. The platsmp.c
file opens up in the
Editor. As you hit each breakpoint, click on Core_0
and switch to the Stack window, then Core_1
and switch to the Stack window, to see functions under the respective cores.
Logging Features
It would be very useful to display early printk
outputs in DS-5
Debugger's command window during kernel bring-up. Ideally, before the console is
enabled, so no serial output is present.
For example, set a breakpoint on console_init()
in main.c
.
Click Continue to run to the breakpoint. No log messages are output to the console because it is not enabled yet. But you can
view the entire log for activities so far.
- Enter
info os-log
in the Command prompt to view the log.
- If you want to view the log output line by line as the code is processed, enter
set os log-capture
on at the Command prompt.
Single-stepping individual cores or threads
You can single step through a single core or thread/process. Select either the core or the thread/process in the Debug Control view, then press F5 on your keyboard.
When you single step through a process, it might be migrated to another core. If a breakpoint is set on a process, the debugger can track the migration of process-specific breakpoints to the other core.
Using expressions to view data
You can use the Expressions view to create and work with expressions in DS-5. For example, to view the kernel's thread_info
structure, in the Expressions view, enter:
(struct thread_info*)($sp_svc & ~0x1FFF)
.
When the view refreshes, expand the tree structure to explore its contents.