How to build MPICH with Arm Compiler for HPC by using the CH4 device layer implementation and OpenUCX.

MPICH is a high performance, and widely portable, implementation of the Message Passing Interface (MPI) standard.

OpenUCX is a collaboration between industry, laboratories, and academia to create an open-source production grade communication framework for data centric and high-performance applications.

More information can be found on the MPICH and OpenUCX web pages.

For the purposes of this build, the following components are used:

Component Form
Arm Compiler for HPC
OpenUCX 1.4
Operating system Ubuntu 16.04
Cavium ThunderX2

Recipes for other versions of the application are available in the GitLab Packages Wiki.

Before you begin


Arm recommends that you rebuild your MPI implementation (Open MPI, MVAPICH, MPICH) after each installation of a new version of Arm Compiler for HPC. This ensures that the Fortran interface incorporates any changes to the armflang module format, and that the correct run-time libraries are used.

Installing OpenUCX

To build and install OpenUCX, follow these steps:

  1. Ensure that your paths are set up appropriately for Arm Compiler for HPC, for example:

    export PATH=/proj/ArmCompiler/bin:$PATH
    export LD_LIBRARY_PATH=/proj/ArmCompiler/lib:$LD_LIBRARY_PATH
  2. Clone the OpenUCX repository into a new directory, ucx in the following example:

    git clone https://github.com/openucx/ucx.git ucx
    1. Optionally, checkout the release version 1.4 tag in the newly created ucx directory.

      git checkout v1.4.0 -b v1.4.0
  3. Change into the ucx directory and run the OpenUCX autogen.sh script:

    cd ucx
  4. Create a new directory called build, change into it, and run the OpenUCX configure script, choosing armclang as the C compiler, and armclang++ as C++ compiler:

    mkdir build
    cd build
    CC=armclang CXX=armclang++ ../configure --prefix=/<path_to_openucx_installation>

    To strip all debugging and profiling code for the highest performance configuration, use ../contrib/configure-release

  5. Build and install OpenUCX, using:

    make install

Building MPICH with OpenUCX

  1. Download and unpack the MPICH package:

    mkdir MPICH
    cd MPICH
    wget http://www.mpich.org/static/downloads/3.3/mpich-3.3.tar.gz
    tar -zxvf mpich-3.3.tar.gz
    cd mpich-3.3
  2. Choose a destination to install MPICH into, for example /opt/arm/mpich-33_arm-hpc-compiler-19.2_Generic-AArch64_Ubuntu-16.04_aarch64-linux::

    export MPICH_INSTALL_DIR=/opt/arm/mpich-33_arm-hpc-compiler-19.2_Generic-AArch64_Ubuntu-16.04_aarch64-linux
  3. Create a build directory, change into it, and run the MPICH configure script from within:

    mkdir build
    cd build
    ../configure CC=armclang CXX=armclang++ F77=armflang FC=armflang -prefix=$MPICH_INSTALL_DIR --enable-fast=all,O3 ‑‑with-device=ch4:ucx --with-ucx=/<path_to_openucx_installation>

    Note: To build a production environment with -O3 optimization for all language bindings, append --enable-fast=all,O3 to the configure line. The default is -O2.

  4. The configure script does not detect armflang. To correct for this, the libtool script generated by the configure script, requires some post-configuration patching:

    sed -i -e 's#wl=""#wl="-Wl,"#g' libtool
    sed -i -e 's#pic_flag=""#pic_flag=" -fPIC -DPIC"#g' libtool
  5. Build and install MPICH:

    make -j
    make install
  6. Add the installation to your path:


    Alternatively, setup an appropriate system environment module.

Note: If the system memory limits are set too low, you might encounter runtime memory allocation errors of the following form when using OpenUCX:

UCX ERROR ibv_create_cq failed: Cannot allocate memory

You can address errors like this by increasing the locked memory limit on the command line using ulimit -l unlimited. If necessary, raise the hard memlock limit. You might need to consult with your system administrator to raise the hard memlock limit because it applies to the whole system, and requires administrator privileges.