Building MPICH with OpenUCX

  1. Download and unpack the MPICH package:

    mkdir MPICH
    cd MPICH
    wget http://www.mpich.org/static/downloads/3.3/mpich-3.3.tar.gz
    tar -zxvf mpich-3.3.tar.gz
    cd mpich-3.3
  2. Choose a destination to install MPICH into, for example /opt/arm/mpich-33_arm-hpc-compiler-19.2_Generic-AArch64_Ubuntu-16.04_aarch64-linux::

    export MPICH_INSTALL_DIR=/opt/arm/mpich-33_arm-hpc-compiler-19.2_Generic-AArch64_Ubuntu-16.04_aarch64-linux
  3. Create a build directory, change into it, and run the MPICH configure script from within:

    mkdir build
    cd build
    ../configure CC=armclang CXX=armclang++ F77=armflang FC=armflang -prefix=$MPICH_INSTALL_DIR --enable-fast=all,O3 ‑‑with-device=ch4:ucx --with-ucx=/<path_to_openucx_installation>

    Note: To build a production environment with -O3 optimization for all language bindings, append --enable-fast=all,O3 to the configure line. The default is -O2.

  4. The configure script does not detect armflang. To correct for this, the libtool script generated by the configure script, requires some post-configuration patching:

    sed -i -e 's#wl=""#wl="-Wl,"#g' libtool
    sed -i -e 's#pic_flag=""#pic_flag=" -fPIC -DPIC"#g' libtool
  5. Build and install MPICH:

    make -j
    make install
  6. Add the installation to your path:

    export PATH=$MPICH_INSTALL_DIR/bin:$PATH

    Alternatively, setup an appropriate system environment module.

Note: If the system memory limits are set too low, you might encounter runtime memory allocation errors of the following form when using OpenUCX:

UCX ERROR ibv_create_cq failed: Cannot allocate memory

You can address errors like this by increasing the locked memory limit on the command line using ulimit -l unlimited. If necessary, raise the hard memlock limit. You might need to consult with your system administrator to raise the hard memlock limit because it applies to the whole system, and requires administrator privileges.


Previous Next