Open MPI with OpenUCX

How to build Open MPI with OpenUCX, using Arm Compiler for HPC.

Introduction

Both Open MPI and OpenUCX are freely available. The Open MPI project is an open source MPI-2 implementation developed and maintained by a consortium of academics, researchers and industry partners. OpenUCX is a collaboration between industry, laboratories, and academia to create an open-source production grade communication framework for data centric and high-performance applications.

More information can be found on the Open MPI and OpenUCX web pages, respectively.

Version information

For the purposes of this build, the following components are used:

Component
 Form
 Open MPI  Version 4.0.0
 Operating System
 RHEL 7.5
 Hardware  Cavium ThunderX2
 Arm Compiler for HPC
 Version 19.0
 GNU Autotools  Autoconf 2.63 or higher

Installing OpenUCX

To build and install OpenUCX, follow these steps:

  1. Ensure that your paths are set up appropriately for Arm Compiler for HPC, for example:

    export PATH=/proj/ArmCompiler/bin:$PATH
    export LD_LIBRARY_PATH=/proj/ArmCompiler/lib:$LD_LIBRARY_PATH
  2. Clone the OpenUCX repository into a new directory, ucx in the following example:

    git clone https://github.com/openucx/ucx.git ucx
    1. Optionally, checkout the release version 1.4 tag in the newly created ucx directory.

      git checkout v1.4.0 -b v1.4.0
      
  3. Change into the ucx directory and run the OpenUCX autogen.sh script:

    cd ucx
    ./autogen.sh
  4. Create a new directory called build, change into it, and run the OpenUCX configure script, choosing armclang as the C compiler, and armclang++ as C++ compiler:

    mkdir build
    cd build
    CC=armclang CXX=armclang++ ../configure --prefix=/<path_to_openucx_installation>
    Note: To strip all debugging and profiling code for the highest performance configuration, use ../contrib/configure-release
  5. Build and install OpenUCX, using:

    make
    make install

Configure and build Open MPI with OpenUCX

  1. See Building Open MPI with Arm Compiler for instructions on how to build and install Open MPI.
  2. When you invoke the configure command in Build and Install Open MPI with Arm Compiler for HPC, add the following flag:

    --with-ucx=/<path_to_openucx_installation>

Running Open MPI with OpenUCX

Example command line input for running a program using Open MPI with OpenUCX:

mpirun -np 2 -mca pml ucx -mca btl ^openib -x UCX_NET_DEVICES=mlx5_0:1 ./myprogram

UCX_NET_DEVICES is an optional flag to select the IB device mlx5_0 on port 1.

Note: If the system memory limits are set too low, you might encounter runtime memory allocation errors of the following form when using OpenUCX:

"UCX ERROR ibv_create_cq failed: Cannot allocate memory" 

You can address errors like this by increasing the locked memory limit on the command line using ulimit -l unlimited. If necessary, raise the hard memlock limit. You might need to consult with your system administrator to raise the hard memlock limit because it applies to the whole system, and requires admin privileges.

See https://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages-user for more information for more information.

Resources