How to build MVAPICH with Arm Compiler

Overview Before you begin Procedure Related information


How to build MVAPICH with Arm Compiler.

The MVAPICH2 software, based on the MPI 3.1 standard, delivers the best performance, scalability, and fault tolerance for high-end computing systems and servers using InfiniBand, Omni-Path, Ethernet/iWARP, and RoCE networking technologies. The MVAPICH2 software family is ABI compatible with the version of MPICH it is based on. For more information, see the MVAPICH website.

For the purposes of this build, the following components are used:

Component Form
MVAPICH Version 2.3.1
Arm Compiler for HPC
Version 19.2
Operating system RHEL 7.5
Cavium ThunderX2

Recipes for other versions of the application are available in the GitLab Packages Wiki.

Before you begin

  • Install Arm Compiler. For more information, see our Installation instructions.

  • Note:

    Arm recommends that you rebuild your MPI implementation (Open MPI, MVAPICH, MPICH) after each installation of a new version of Arm Compiler for HPC. This ensures that the Fortran interface incorporates any changes to the armflang module format, and that the correct run-time libraries are used.

  • Install the following required OS packages: ibverbs; inverbs-devel; libibmad; libibmad-devel; and byacc.
    These can be installed using the following command:
    yum -y install <package name>


  1. Download and unpack the application source code:

    tar ‑zxvf mvapich2‑2.3.1.tar.gz
  2. Change into the unpacked directory:

    cd mvapich2-2.3.1
  3. Set an install location, INSTALL_DIR, where MVAPICH will be installed to. For example:

    export INSTALL_DIR=/path/to/MVAPICH_install

    Replace /path/to/MVAPICH_install with the path to your installation.

  4. Set the compilers to use for the build:

    export CC=armclang
    export CXX=armclang++
    export FC=armflang
  5. Create a build directory and change into it:

    mkdir build
    cd build
  6. Run configure from the build directory, specifying the devices and protocols to build for, and the language interfaces to include: 

    Note: The default on Linux is OpenFabrics (OFA) IB/iWARP/RoCE with the CH3 channel. Explicitly select it with: --with-device=ch3:mrail --with-rdma=gen2.

    ../configure --prefix=$INSTALL_DIR --with-device=ch3:mrail --with-rdma=gen2 --enable-cxx --enable-fc

    Note: For more configuration options, see the MVAPICH user guide.

  7. Build, test, and install MVAPICH, using:

    make -j
    make install
    make testing
  8. By default, MVAPICH enables CPU affinity. However, in the case of multi‑threaded programs, this can lead to poor thread placement and poor performance. It is recommended to disable CPU affinity in this case:

    export MV2_ENABLE_AFFINITY=0

    You must also disable CPU affinity if you want to over-subscribe the available cores with MPI tasks, for testing and development purposes.

    MVAPICH supports the explicit control of the CPU and thread placement and binding policy via MV2_CPU_BINDING_LEVEL, MV2_CPU_BINDING_POLICY, MV2_HYBRID_BINDING_POLICY and MV2_THREADS_PER_PROCESS.

    mpirun provides details on the mapping between CPUs and processes if MV2_SHOW_CPU_BINDING is set.

    For more information, see the MVAPICH User Guide.