Both Open MPI and OpenUCX are freely available. The Open MPI project is an open source, MPI-2 implementation, developed and maintained by a consortium of academics, researchers, and industry partners. OpenUCX is a collaboration between industry, laboratories, and academia to create an open-source production grade communication framework for data centric and high-performance applications.
The following components are used in this build:
|Open MPI|| Version 4.0.0
| Operating System
|| RHEL 7.5
|Hardware|| Cavium ThunderX2
| Arm Compiler for HPC
|| Version 19.0
|GNU Autotools|| Autoconf 2.63 or higher
Recipes for other versions of the application are available in the GitLab Packages Wiki.
To build and install OpenUCX:
- Ensure that your paths are set up appropriately for Arm Compiler for HPC, for example:
- Clone the OpenUCX repository into a new directory,
ucxin the following example:
git clone https://github.com/openucx/ucx.git ucx(Optional) Checkout the release version 1.4 tag in the newly created ucx directory.
git checkout v1.4.0 -b v1.4.0
- Change into the
ucxdirectory and run the OpenUCX autogen.sh script:
cd ucx ./autogen.sh
- Create a new directory called
build, change into it, and run the OpenUCX configure script, choosing armclang as the C compiler, and armclang++ as C++ compiler:
mkdir build cd build CC=armclang CXX=armclang++ ../configure --prefix=/<path_to_openucx_installation>Note: To strip all debugging and profiling code for the highest performance configuration, use
- Build and install OpenUCX, using:
make make install
- See Building Open MPI with Arm Compiler for instructions on how to build and install Open MPI.
When you invoke the
configurecommand in Build and Install Open MPI with Arm Compiler for HPC, add the following flag:
Example command line input for running a program using Open MPI with OpenUCX:
mpirun -np 2 -mca pml ucx -mca btl ^openib -x UCX_NET_DEVICES=mlx5_0:1 ./myprogram
UCX_NET_DEVICES is an optional flag to select the IB device
mlx5_0 on port 1.
Note: If the system memory limits are set too low, you might encounter runtime memory allocation errors of the following form when using OpenUCX:
"UCX ERROR ibv_create_cq failed: Cannot allocate memory"
You can solve system memory errors by increasing the locked memory limit on the command line using
ulimit -l unlimited. If necessary, raise the hard memlock limit. You might need to consult with your system administrator to raise the hard memlock limit because it applies to the whole system, and requires admin privileges.
See https://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages-user for more information for more information.