Running OpenMPI with OpenUCX

Example command line input for running a program using Open MPI with OpenUCX:

mpirun -np 2 -mca pml ucx -mca btl ^openib -x UCX_NET_DEVICES=mlx5_0:1 ./myprogram

UCX_NET_DEVICES is an optional flag to select the IB device mlx5_0 on port 1.

Note: If the system memory limits are set too low, you might encounter runtime memory allocation errors of the following form when using OpenUCX:

"UCX ERROR ibv_create_cq failed: Cannot allocate memory" 

You can solve system memory errors by increasing the locked memory limit on the command line using ulimit -l unlimited. If necessary, raise the hard memlock limit. You might need to consult with your system administrator to raise the hard memlock limit because it applies to the whole system, and requires admin privileges.

See https://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages-user for more information for more information.

Previous Next