In this guide, we will explore Linux application and system performance analysis and learn how to find where a system is spending time. Annotating applications and finding performance bottlenecks helps focus software optimization efforts to improve system performance.

The Streamline Performance Analyzer provides system performance metrics, software tracing, and statistical profiling to help engineers get the most performance from hardware and find important bottlenecks in software.

The Raspberry Pi 3 and the Raspberry Pi 4 are low-cost boards with Cortex-A processors. This means that a Raspberry Pi is a useful tool for learning Linux profiling with Streamline. Because Raspberry Pi boards are designed for education, they do not require complex procedures to enable profiling features.

Before you begin

To work through this guide, you will need access to Streamline, which we will explain in Starting Streamline. You will also need a Raspberry Pi 3 or Raspberry Pi 4 board. An HDMI monitor and USB keyboard and mouse is the easiest way to interact with the Raspberry Pi. Finally, an SD card is needed to hold the Raspbian operating system.

A basic understanding of working with terminals in Linux will also be helpful.

Starting Streamline

In this section of the guide, we will install and start Streamline on a Windows or Linux computer.

Streamline is included with Arm Development Studio. A 30-day free trial is available, which you can use to work through this guide. If you already have Arm Development Studio installed, check the downloads area to make sure that you have the latest version.

Follow these steps to start Arm Development Studio. You can refer to the Getting Started Guide for more information about the requirements and install process.

There is only one step required to start Streamline for Linux:

  1. Start Streamline for Linux using the Streamline command
    $ Streamline &

There are two steps required to start Streamline for Windows:

  1. Find Streamline for Windows on the start menu. If everything is installed correctly and the license is working, you will see the following screenshot:

  2. Install ssh and scp for Windows if you want to, which enable connection to the Raspberry Pi. Possible options are PuTTY, Windows SSH, and Git Bash SSH.
Preparing the target

There are two important things to consider when you use Streamline on a new target system:

  • Configuring the Linux kernel (if necessary)
  • Installing the Gator daemon application

The Linux kernel configuration involves ensuring that the profiling features are enabled in the kernel configuration. If the kernel has perf_event support, Arm Performance Monitoring Unit (PMU) counters are visible to Streamline. The required kernel support is enabled in the Raspberry Pi kernel, so no special configuration is required.

In the past, a Linux kernel driver facilitated data collection for Streamline, but this has been deprecated. Instead, standard Linux interfaces including perf and trace are used by Streamline to capture data.

Streamline relies on the Gator daemon application to collect profiling information from a target system. The separation between the data collection and data display makes the daemon approach ideal for embedded systems which have no user interface.

The Gator daemon can be copied from Streamline or compiled from source. Both methods are covered in Installing Gator daemon. For best results, you must run the Gator daemon as root on the target system. This is easy to do with a Raspberry Pi.

Related information includes a link to more information about target setup for Streamline. 

Configuring the Raspberry Pi

In this section of the guide, we will configure the Raspberry Pi to run with Streamline.

Follow these steps:

  1. Navigate to the latest version of Raspbian from Raspberry Pi. Follow the instructions to create an SD card, using whatever path is easiest.
  2. Use the dd command on Linux, or the Raspberry Pi Imager on Linux, macOS or Ubuntu, to copy the .img file to the SD card. The Raspberry Pi Imager is easy to use. We recommend that you use an SD card that is 16GB or larger.
  3. Boot the system for the first time. The easiest way to boot the system is to connect an HDMI monitor and keyboard. The default username is pi and the default password is raspberry
  4. Connect the wireless or wired network for Internet connectivity. A wired connection is available by plugging an Ethernet cable into your router. Usually no additional steps are needed. 

Connect to a wireless network

Let’s explore how to connect the Raspberry Pi to a wireless network.

Follow these steps to connect to a wireless network:

  1. Start the console-based Raspberry Pi configuration tool:

    $ sudo raspi-config
  2. Use option 2 to select Network Options, then option 2 again to select Wi-Fi. Enter the SSID and key, if needed, as you can see in this screenshot:

  3. Enable ssh. Use option 5 Interfacing Options from the top menu, then 2 SSH, as you can see in this screenshot:

    If the Raspbian desktop is running, enable the ssh server using the configuration Preferences -> Raspberry Pi. Click the Interfaces tab.

  4. Change the password to something other than the default value. The use of sudo is automatically enabled for user pi. Enter passwd on the command line and click Enter. You will be prompted to enter the current password and then asked for a new password. The new password will be requested a second time to confirm.

In Installing the Gator daemon, we will set up the gator daemon to enable the connection to Streamline. 

Installing the Gator daemon

In this section of the guide, we will install the Gator daemon. There are two ways to get the Gator daemon software:

  • A pre-compiled executable that is provided with Arm Development Studio. This can be copied to the Raspberry Pi and run.
  • An open-source project on github. The source code can be downloaded, compiled on the Raspberry Pi, and run.

Let’s look at each in turn.

Installing the gatord executable

To copy gatord from the Arm DS installation, follow these steps:

  1. Use the Linux scp command and substitute the IP address of your Raspberry Pi.

For example:

$ scp $ARMDS_HOME/sw/streamline/bin/arm/gatord  pi@ 
Building gatord from source code

To download the source code from github and compile it, follow these steps:

  1. Download the software, using the following commands:
    $ git clone https://github.com/ARM-software/gator.git
    $ cd daemon 
    $ make
  2. Start the Gator daemon with root privileges for system profiling. Starting it without root limits the visibility to only those processes that are owned by the user account. The gator command is:
    $ sudo gatord  &

Connecting Streamline

To connect Streamline from a Windows or Linux machine, follow these steps:

  1. Start the Streamline GUI using the menu or by running Streamline from the command line, as described in Installing Streamline.
  2. Click the eyeball on the Target tab.
  3. Click Select if the Raspberry Pi shows up immediately. Click Setup Target if the Raspberry Pi does not show up automatically.
  4. Enter the IP address of the Raspberry Pi and the username pi.

    If you enter the IP address of the Pi but Streamline still cannot find it, ensure that Streamline and the Raspberry Pi are on the same network. Only the last number of the IP addresses of the host and target machine should be different. Below is the target setup dialog:

  5. Click Install.
  6. Click No in the dialog box that is displayed after you click Install. This dialog box is confusing, because it asks you to install the gatord on the target system. You do not need to do this, because the gatord is already running.
  7. Select the target, as you can see in the following screenshot:

Capturing some profiling data

In this section of the guide, we will start a capture session, run a program, and end the capture session. Obtaining a capture confirms that data collection is working.

Follow these steps:

  1. Select the name and location of the capture file using the red circle. Click OK. You will be promoted for a location and filename to save the capture as shown in this screenshot:

  2. Click the red stop sign icon ends the capture.

Without any source code or software images, the Call Paths, Functions, and Code tabs do not provide much information. Instead, these tabs show process names and process ID values with some blank screens and unknown code messages. This will improve when Streamline has the software images and source. In Profiling an example application, we will download an example application to profile, and learn how to insert markers in the code for Streamline to visualize. 

Profiling an example application

In this section of the guide, we use the LMbench applications to demonstrate profiling with Streamline. These applications are easy to download and build with -g. With these applications, Streamline can map the source code of the applications. In Profiling an example application, we will download an example application to profile, and learn how to insert markers in the code for Streamline to visualize. 

Follow these steps:

  1. Download an example Linux application:
    $ wget
    $ tar xvfz lmbench-3.0-a9.tgz
    $ cd lmbench-3.0-a9/src
    $ make debug
    This generates executables in the bin/armv7l-linux-gnu directory.
  2. Run the application, using these commands:
    $ cd ..
    $ bin/armv7l-linux-gnu/bw_mem  512M rd
  3. Enable the Streamline capture and run the test.

  4. Stop the test to see the results.

Source level profiling

To see the application source code in Streamline, the files must be on the host machine where Streamline is running. To do this, follow these steps:

  1. Copy the lmbench-3.0-a9 directory from the Raspberry Pi to the host machine using scp. Again, substitute the IP address of the Pi in the following command:
    $ scp -r pi@ .

    The Code tab in Streamline will not be able to find the source code. This is because the path on the target is different from the path on the host.

  2. Use Click here to locate source and navigate to the file that is being referenced, in this case bw_mem.c. The following screenshot shows the missing source code and how to fix it:

    When the source code is found, the window looks like what you can see in the following screenshot:

Inserting markers

You can insert markers into the source code of any application, to make it easier to track progress on the Streamline timeline. You can find everything that you need in the annotate/ directory of the gator software.

To insert a marker, follow these steps:

  1. Include the following header file at the top of the source file, to add a marker to the LMbench source file bw_mem.c:
    #include "streamline_annotate.h"
  2. Add ANNOTATE_SETUP; during the setup, somewhere in the main() function before the test is run.
  3. Put the markers where they are needed, to track the test on the Streamline timeline, as seen in the following code:
    rd(iter_t iterations, void *cookie)
                state_t *state = (state_t *) cookie;
                register TYPE *lastone = state->lastone;
                register int sum = 0;
                ANNOTATE_MARKER_STR("start of rd");
                while (iterations-- > 0) {
                    register TYPE *p = state->buf;
                    while (p <= lastone) {
                        sum +=
        #define DOIT(i) p[i]+
                        DOIT(0) DOIT(4) DOIT(8) DOIT(12) DOIT(16) DOIT(20) DOIT(24)
                        DOIT(28) DOIT(32) DOIT(36) DOIT(40) DOIT(44) DOIT(48) DOIT(52)
                        DOIT(56) DOIT(60) DOIT(64) DOIT(68) DOIT(72) DOIT(76)
                        DOIT(80) DOIT(84) DOIT(88) DOIT(92) DOIT(96) DOIT(100)
                        DOIT(104) DOIT(108) DOIT(112) DOIT(116) DOIT(120)
                        p +=  128;
  4. Edit the Makefile to: 
    • Add the directory containing streamline_annotate.h to the compiler include path. 
    • Add streamline_annotate.c to the list of source files to compile.

Next steps

In this guide, we described the steps to set up and run Streamline for application and full system profiling on an Arm target system. The Raspberry Pi is one of the best systems to learn with because it is easy to gain root access, change configuration, and build software from the start. There are also numerous resources to learn about the Raspberry Pi and the Raspbian operating system. Learning this flow on the Raspberry Pi makes it easier when it is time to apply Streamline on any other target system.