Overview

Both Open MPI and OpenUCX are freely available. The Open MPI project is an open source, MPI-2 implementation, developed and maintained by a consortium of academics, researchers, and industry partners. OpenUCX is a collaboration between industry, laboratories, and academia to create an open-source production grade communication framework for data centric and high-performance applications.

You can find more information on the Open MPI and OpenUCX web pages, respectively.

Version information

The following components are used in this build:

Component
 Form
 Open MPI  Version 4.0.0
 Operating System
 RHEL 7.5
 Hardware  Cavium ThunderX2
 Arm Compiler for HPC
 Version 19.0
 GNU Autotools  Autoconf 2.63 or higher

Recipes for other versions of the application are available in the GitLab Packages Wiki.

Installing OpenUCX

To build and install OpenUCX:

  1. Ensure that your paths are set up appropriately for Arm Compiler for HPC, for example:

    export PATH=/proj/ArmCompiler/bin:$PATH
    export LD_LIBRARY_PATH=/proj/ArmCompiler/lib:$LD_LIBRARY_PATH
  2. Clone the OpenUCX repository into a new directory, ucx in the following example:

    git clone https://github.com/openucx/ucx.git ucx
    (Optional) Checkout the release version 1.4 tag in the newly created ucx directory.

    git checkout v1.4.0 -b v1.4.0
    
  3. Change into the ucx directory and run the OpenUCX autogen.sh script:

    cd ucx
    ./autogen.sh
  4. Create a new directory called build, change into it, and run the OpenUCX configure script, choosing armclang as the C compiler, and armclang++ as C++ compiler:

    mkdir build
    cd build
    CC=armclang CXX=armclang++ ../configure --prefix=/<path_to_openucx_installation>
    Note: To strip all debugging and profiling code for the highest performance configuration, use ../contrib/configure-release
  5. Build and install OpenUCX, using:

    make
    make install

Running OpenMPI with OpenUCX

Example command line input for running a program using Open MPI with OpenUCX:

mpirun -np 2 -mca pml ucx -mca btl ^openib -x UCX_NET_DEVICES=mlx5_0:1 ./myprogram

UCX_NET_DEVICES is an optional flag to select the IB device mlx5_0 on port 1.

Note: If the system memory limits are set too low, you might encounter runtime memory allocation errors of the following form when using OpenUCX:

"UCX ERROR ibv_create_cq failed: Cannot allocate memory" 

You can solve system memory errors by increasing the locked memory limit on the command line using ulimit -l unlimited. If necessary, raise the hard memlock limit. You might need to consult with your system administrator to raise the hard memlock limit because it applies to the whole system, and requires admin privileges.

See https://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages-user for more information for more information.