How to build MPICH with Arm Compiler for HPC by using the CH4 device layer implementation and OpenUCX.
MPICH is a high performance, and widely portable, implementation of the Message Passing Interface (MPI) standard.
OpenUCX is a collaboration between industry, laboratories, and academia to create an open-source production grade communication framework for data centric and high-performance applications.
For the purposes of this build, the following components are used:
|Arm Compiler for HPC
|Operating system||Ubuntu 16.04
Recipes for other versions of the application are available in the GitLab Packages Wiki.
Before you begin
- Install Arm Compiler for HPC. For more information, see our instructions on Installing Arm Compiler for HPC.
Arm recommends that you rebuild your MPI implementation (Open MPI, MVAPICH, MPICH) after each installation of a new version of Arm Compiler for HPC. This ensures that the Fortran interface incorporates any changes to the armflang module format, and that the correct run-time libraries are used.
To build and install OpenUCX, follow these steps:
Ensure that your paths are set up appropriately for Arm Compiler for HPC, for example:
export PATH=/proj/ArmCompiler/bin:$PATH export LD_LIBRARY_PATH=/proj/ArmCompiler/lib:$LD_LIBRARY_PATH
Clone the OpenUCX repository into a new directory,
ucxin the following example:
git clone https://github.com/openucx/ucx.git ucx
Optionally, checkout the release version 1.4 tag in the newly created ucx directory.
git checkout v1.4.0 -b v1.4.0
Change into the
ucxdirectory and run the OpenUCX autogen.sh script:
cd ucx ./autogen.sh
Create a new directory called
build, change into it, and run the OpenUCX configure script, choosing armclang as the C compiler, and armclang++ as C++ compiler:
mkdir build cd build CC=armclang CXX=armclang++ ../configure --prefix=/<path_to_openucx_installation>Note:
To strip all debugging and profiling code for the highest performance configuration, use
Build and install OpenUCX, using:
make make install
Building MPICH with OpenUCX
Download and unpack the MPICH package:
tar -zxvf mpich-3.3.tar.gz
Choose a destination to install MPICH into, for example
Create a build directory, change into it, and run the MPICH configure script from within:
../configure CC=armclang CXX=armclang++ F77=armflang FC=armflang -prefix=$MPICH_INSTALL_DIR --enable-fast=all,O3 ‑‑with-device=ch4:ucx --with-ucx=/<path_to_openucx_installation>
Note: To build a production environment with
-O3optimization for all language bindings, append
--enable-fast=all,O3to the configure line. The default is
The configure script does not detect
armflang. To correct for this, the
libtoolscript generated by the configure script, requires some post-configuration patching:
sed -i -e 's#wl=""#wl="-Wl,"#g' libtool
sed -i -e 's#pic_flag=""#pic_flag=" -fPIC -DPIC"#g' libtool
Build and install MPICH:
Add the installation to your path:
Alternatively, setup an appropriate system environment module.
Note: If the system memory limits are set too low, you might encounter runtime memory allocation errors of the following form when using OpenUCX:
UCX ERROR ibv_create_cq failed: Cannot allocate memory
You can address errors like this by increasing the locked memory limit on the command line using
ulimit -l unlimited. If necessary, raise the
hard memlock limit. You might need to consult with your system administrator to raise the
hard memlock limit because it applies to the whole system, and requires administrator privileges.