The PAPI instructions-per-cycle metric measures the mean number of instructions executed per CPU cycle. This is often used as a proxy for computational intensity – values below 1.0 suggest the CPU is spending a lot of time stalled. Modern superscalar architectures can issue many instructions per cycle, but for HPC applications, 1.0 is generally considered acceptable.
MAP will let us track this metric over time and correlate it to our existing CPU, MPI and I/O metrics as well as against our source code. We're going to use the PAPI library to add this as a custom metric, but you can use the same approach to add any kind of metric you like to MAP. Let's go!
Note: There's a set of high-quality reference docs in your forge/map/metrics/doc/ directory, both in PDF and interactive HTML format. You don't need them to follow along with this guide but they'll be an invaluable help when you want to implement your own metrics!
Life's easier when you start with a working template. In the forge/map/metrics/examples/ directory there's a “sample” directory containing a sample metric that measures something boring like interrupts per second. We're going to use it as a framework for our new PAPI IPC metric.
$ cd forge/map/metrics/examples/
$ cp -r sample papi
$ cd papi
Makefile sample.c sample.xml
That's all you need for a new metric. Let's rename ours:
$ mv sample.c my_papi.c
$ mv sample.xml my_papi.xml
We now need to update the Makefile with the new filenames:
$ vim Makefile
This Makefile builds a shared library that MAP will automatically load into your program while it is running. Functions from this library will be called to take measurements.
Here we need to rename sample to my_papi, and we also want to add a -lpapi flag as well to ensure we can use the PAPI functions in our code:
That was easy! Now let's write the code!
Writing the my_papi.c file
We copied and renamed the existing sample.c file but it's still full of references to its original goal, which was reading from /proc/interrupts:
We can strip this back down to the minimum pretty quickly:
Now we are ready. We have 3 hooks available:
allinea_plugin_initialize – this is called once at the start of the program to initialize any counters or variables we might need.
allinea_plugin_cleanup – this is called at the end of the program in case we need to do any cleaning up of files or memory (both rare).
sample_interrupts – this will be called by MAP once per sampling interval (initially 50Hz but MAP will reduce this dynamically as the application runtime increases and will automatically resample your data for you).
To get the PAPI instructions-per-cycle you just need to use one function from the PAPI API:
This is almost as easy as it gets. We don't need to do any special initialization or cleanup, we can just call PAPI_ipc each sampling interval. The manpage doesn't say whether PAPI_ipc is signal safe. This is an issue because we'll be calling it from a signal handler (that's how statistical profilers like MAP work). We'll just try it and see!
Using this in our custom metric functions is straightforward. First we rename sample_interrupts to something more descriptive, like sample_ipc and implement it like this:
There are 5 things I want to call out here:
- I changed the type of the out_value parameter from uint64_t* to double*. MAP will let you use either of these. Instructions per cycle will be a float, so a double* is the type I want.
- We set the in_out_sample_time to allinea_get_current_time(). It's usually fine to do this as the first thing this function does.
- We call PAPI_ipc, which stores a bunch of information we don't need and one piece that we do, namely the IPC since the last call.
- We check the return value of the function and use the allinea_set_metric_error_messagef function to report any errors back up to the user. We then return -1. Error handling across a distributed cluster is as simple as that!
- Finally we set *out_value to the value we have recorded (ipc) and return 0 for success.
It's important that these functions be fast because the slower they are the more overhead we are adding to the program.
Telling MAP about our metric in my_papi.xml
This file tells MAP all the extra information about our metric – what to call it in the GUI, which group of metrics to put it in, which types to expect and so on. The version we copied from sample.xml looks like this:
Let's start making some changes. There are two main XML elements here – metric and metricGroup. For metric:
The id can change from com.allinea.metrics.sample.interrupts to something else. I've chosen com.allinea.metrics.papi.ipc.
We used the double type, not uint64_t, so we need to change that.
The com.allinea.metrics.sample_src reference should be changed too. I used com.allinea.metrics.papi_src.
The functionName is not sample_interrupts – I renamed it to sample_ipc.
The divideBySampleTime property should be false – this is used if you are measuring e.g. bytes and want MAP to automatically display that as a rate over time (bytes/s). IPC/s would make no sense!
The <display> controls names and groups in the GUI. The colour here is specified in HTML notation.
The metricGroup element describes the group this metric will belong to in the GUI. I decided to add it to a PAPI group of metrics – perhaps we'll add more later! This is straightforward, but make sure the metric ref and source id match the ones used in your metric section.
The whole file with some comments removed looks like this: