How to build ScaLAPACK with Arm Compiler.

ScaLAPACK is a library of high-performance linear algebra routines for parallel distributed memory machines. For more information, see the ScaLAPACK website.

The following components are used for this build:

Component Version
ScaLAPACK 2.0.2
Open MPI 4.0.1
Arm Compiler 19.2
Arm Performance Libraries (ArmPL) 19.2
Operating System RHEL 7.5
Hardware Cavium ThunderX2

Recipes for other versions of the application are available in the GitLab Packages Wiki.

Before you begin

  • Install Arm Compiler. For more information, see our Installation instructions.
  • Open MPI Versions 4.0.0+ removed support for features which were originally deprecated with the release of the MPI-2.0 standard, and later removed entirely from the MPI-3.0 standard (for more information see Removed MPI constructs in Open MPI FAQs). To enable these missing calls, you must ensure that your Open MPI installation was built with the--enable-mpi1-compatibility flag. Alternatively, it is possible to update the ScaLAPACK source to replace the obsolete calls (see Resolving issues with deprecated MPI features).

Procedure

  1. Download and unpack the library source.

    wget -O scalapack-2.0.2.tgz http://www.netlib.org/scalapack/scalapack-2.0.2.tgz
    tar -zxvf scalapack-2.0.2.tgz
    cd scalapack-2.0.2
    
  2. Make a copy of SLmake.inc.example, and rename it as SLmake.inc.

    Note: ScaLAPACK requires a SLmake.inc file which is tuned for the target machine, and is based on the example SLmake.inc.example that is included in the distribution.
  3. Modify the SLmake.inc file for the Arm HPC compiler and Arm Performance Libraries as follows:

    Set trailing underscores for calling Fortran subroutines from C:

    CDEFS = -DAdd_ 

    Set the C and Fortran flags:

    FCFLAGS = ‑O3‑mcpu=native
    CCFLAGS = ‑O3‑mcpu=native ‑fsimdmath

    Link to the ArmPL libraries for BLAS and LAPACK calls by removing the definitions of BLASLIB and LAPACKLIB, and setting LIBS:

    LIBS = $(ARMPL_DIR)/lib/libarmpl_lp64_mp.so
    Note:
    • ARMPL_DIR is the location of the ArmPL installation. For example:
      /opt/arm/armpl-19.2.0_ThunderX2CN99_RHEL-7_arm-hpc-compiler_19.2_aarch64-linux
    • These linking instructions build for 32-bit integers. To build for 64-bit integers, instead of linking against libarmpl_lp64_mp.so, link against libarmpl_ilp64_mp.so.
  4. Build the ScaLAPACK library:

    make lib
  5. Copy the libscalapack.a static library, created by the make lib command, to your preferred installation directory.

Resolving issues with deprecated MPI features

Warning:

This section only applies if you did not include the --enable-mpi1-compatibility flag when building Open MPI version 4.0.1 (see Before you begin).

This section describes how to resolve the issues with deprecated calls, if you have not built Open MPI v4.0.1+ with the --enable-mpi1-compatibility flag.

ScaLAPACK uses the MPI_Type_struct function, which was deprecated in version 2 of the MPI standard and removed entirely from version 3. To replace all occurrences of this obsolete feature, correct the ScaLAPACK source using sed:

sed -i -e 's/MPI_Type_struct/MPI_Type_create_struct/g' ./BLACS/SRC/cgamn2d_.c
sed -i -e 's/MPI_Type_struct/MPI_Type_create_struct/g' ./BLACS/SRC/dgamn2d_.c
sed -i -e 's/MPI_Type_struct/MPI_Type_create_struct/g' ./BLACS/SRC/igamn2d_.c
sed -i -e 's/MPI_Type_struct/MPI_Type_create_struct/g' ./BLACS/SRC/igamx2d_.c
sed -i -e 's/MPI_Type_struct/MPI_Type_create_struct/g' ./BLACS/SRC/sgamn2d_.c
sed -i -e 's/MPI_Type_struct/MPI_Type_create_struct/g' ./BLACS/SRC/sgamx2d_.c
sed -i -e 's/MPI_Type_struct/MPI_Type_create_struct/g' ./BLACS/SRC/zgamn2d_.c
sed -i -e 's/MPI_Type_struct/MPI_Type_create_struct/g' ./BLACS/SRC/zgamx2d_.c

Alternatively, to modify the code, download this scalapack_openmpi4_fix.patch file and apply it:

patch -p1 < scalapack_openmpi4_fix.patch

For details about the features removed in Open MPI 4.0.0+, see Removed MPI constructs on the Open MPI website.