NWChem

Building NWChem with Arm Compiler


Overview Procedure Related information

Overview

How to build NWChem with Arm Compiler for HPC.

NWChem is an open source High-Performance Computational Chemistry application. NWChem provides users with computational chemistry tools that are scalable for both efficiency and varying cluster resources.

For more information on NWChem visit the NWChem webpage.

For the purposes of this build, the following components are used:

Component
Form
NWChem Version 6.8
Arm Performance Libraries
Version 18.2
Arm Compiler for HPC
Version 18.2
Open MPI Version 3.0.2
Operating System
RHEL 7.5
Hardware Cavium ThunderX2

Recipes for other versions of the application are available in the GitLab Packages Wiki.

Procedure

  1. Download and unpack the NWChem package:

    wget https://github.com/nwchemgit/nwchem/releases/download/v6.8-release/nwchem-6.8-release.revision-v6.8-47-gdf6c956-src.2017-12-14.tar.bz2
    tar jxf nwchem-6.8-release.revision-v6.8-47-gdf6c956-src.2017-12-14.tar.bz2
  2. Set NWCHEM_TOP  to <your path>/nwchem-6.8:

    export NWCHEM_TOP=`pwd`/nwchem-6.8
  3. Configure Arm Compiler for HPC:

    export CC=armclang
    export CXX=armclang++
    export FC=armflang
    
  4. Change into the $NWCHEM_TOP directory and apply this patch:

    cd $NWCHEM_TOP
    patch -p0 <<EOF
    --- src/config/makefile.h   2018-02-02 10:30:28.399555033 +0000
    +++ src/config/makefile.h.arm   2018-02-02 10:29:47.485239331 +0000
    @@ -1592,6 +1592,9 @@
           ifeq (\$(CC),icc)
             _CC=icc
           endif
    +     ifeq (\$(CC),armclang)
    +       _CC=armclang
    +     endif
           ifeq (\$(FC),pgf90)
             _FC=pgf90
           endif
    @@ -1613,6 +1616,9 @@
              ifeq (\$(FC),xlf)
                _FC=xlf
              endif
    +     ifeq (\$(FC),armflang)
    +      _FC=armflang
    +     endif
           ifndef _FC
           FC=gfortran
           _FC=gfortran
    @@ -2141,6 +2147,92 @@
           endif
     endif

    +     ifeq (\$(_CPU),\$(findstring \$(_CPU),aarch64))
    +
    +       ifdef  USE_DEBUG
    +         FOPTIONS += -g
    +         COPTIONS += -g
    +         LDOPTIONS += -g
    +         LDFLAGS += -g
    +       endif
    +  
    +       ifeq (\$(_CC),gcc)
    +         COPTIONS   +=   -O3 -funroll-loops -ffast-math
    +         ifdef USE_OPENMP
    +           COPTIONS += -fopenmp
    +         endif
    +       endif
    +
    +       ifeq (\$(_FC),gfortran)
    + #gcc version 4.1.0 20050525 (experimental)
    +         ifdef  USE_GPROF
    +           FOPTIONS += -pg
    +           COPTIONS += -pg
    +           LDOPTIONS += -pg
    +           LDFLAGS += -pg
    +         endif
    +         LINK.f = \$(FC)  \$(LDFLAGS)
    +         FOPTIMIZE  += -O3
    +         ifeq (\$(GNU_GE_6),true)
    +           FOPTIMIZE += -fno-tree-dominator-opts # solvation/hnd_cosmo_lib breaks
    +           FOPTIONS += -fno-tree-dominator-opts # solvation/hnd_cosmo_lib breaks
    +           FDEBUG += -fno-tree-dominator-opts # solvation/hnd_cosmo_lib breaks
    +         endif
    +  
    +         FOPTIMIZE  += -fprefetch-loop-arrays #-ftree-loop-linear
    +         ifeq (\$(GNU_GE_4_8),true)
    +           FOPTIMIZE  += -ftree-vectorize   -fopt-info-vec
    +         endif
    +
    +         FDEBUG += -g -O
    +         ifdef USE_F2C
    + #possible segv with use of zdotc (e.g. with GOTO BLAS)
    + #http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20178
    +          FOPTIONS +=  -ff2c -fno-second-underscore
    +         endif
    +         ifeq (\$(GNU_GE_4_6),true)
    +           FOPTIMIZE +=  -mtune=native
    +           FOPTIONS += -finline-functions
    +         endif
    +         ifndef USE_FPE
    +           FOPTIMIZE  += -ffast-math #2nd time
    +         endif
    +         ifdef  USE_FPE
    +           FOPTIONS += -ffpe-trap=invalid,zero,overflow  -fbacktrace
    +         endif
    +       endif  # end of gfortran
    +
    +       ifeq (\$(_FC),armflang)
    +
    +         ifdef USE_SHARED
    +           FOPTIONS+= -fPIC
    +         endif
    +
    +         DEFINES   +=   -DARMFLANG
    +       LINK.f = \$(FC)  \$(LDFLAGS)
    +         #FOPTIMIZE  += -O3
    +         FOPTIMIZE  = -O3 -Mfma -ffp-contract=fast
    +         # not in armflang
    +         # FOPTIMIZE  += -fprefetch-loop-arrays #-ftree-loop-linear
    +         ifeq (\$(GNU_GE_4_8),true)
    +         # not in armflang
    +           FOPTIMIZE  += -ftree-vectorize   -fopt-info-vec
    +         endif
    +
    +         FDEBUG += -g -O
    +         FOPTIMIZE +=  -mtune=native
    +         # not in armflang
    +         # FOPTIONS += -finline-functions
    +  
    +         ifndef USE_FPE
    +           FOPTIMIZE  += -ffast-math #2nd time
    +         endif
    +         ifdef  USE_FPE
    +           FOPTIONS += -ffpe-trap=invalid,zero,overflow  -fbacktrace
    +         endif
    +       endif
    +     endif # end of aarch64
    +
     ifeq (\$(_CPU),\$(findstring \$(_CPU), ppc64 ppc64le))
     # Tested on Red Hat Enterprise Linux AS release 3 (Taroon Update 3)
     # Tested on SLES 9
    EOF
  5. Modify the src/optim/string/string.F file, using:

    sed -i 's/#if defined(DECOSF)/#if defined(ARMFLANG) || defined(DECOSF)/' $NWCHEM_TOP/src/inp/inp.F
  6. Modify the src/optim/string/string.F file, using:

    sed -i 's/logical freeze1,freezeN,projection1/logical freeze1,freezeN,projection1,tmpVar/' $NWCHEM_TOP/src/optim/string/string.F
    sed -i '/if ( .not. (task_gradient(rtdb).or.ignore))/a\ tmpVar = task_gradient(rtdb)' $NWCHEM_TOP/src/optim/string/string.F sed -i '/tmpVar = task_gradient(rtdb)/a\ if ( .not. (tmpVar.or.ignore))' $NWCHEM_TOP/src/optim/string/string.F sed -i 's/ if ( .not. (task_gradient(rtdb).or.ignore))/* if ( .not. (task_gradient(rtdb).or.ignore))/' $NWCHEM_TOP/src/optim/string/string.F
  7. Configure the build:

    1. Change into $NWCHEM_TOP:

      cd $NWCHEM_TOP
    2. Set the NWChem environment variables using $NWCHEM to define your target platform. For example:

      export NWCHEM_TARGET=LINUX64
    3. To achieve the best performance on high performance networks, define ARMCI_NETWORK. For example:

      export ARMCI_NETWORK=MPI-PR
    4. Set the MPI PATHS:

      • To indicate that NWChem should be compiled with MPI, set USE_MPI to y:
        export USE_MPI=y
      • To use MPI fortran-bindings (generally set when USE_MPI is set), set USE_MPIF to y:
        export USE_MPIF=y
      • To use MPI Integer*4 fortran-bindings (generally set when USE_MPI is set on most platforms), set USE_MPIF4 to y:
        export USE_MPIF4=y
    5. (Optional) Guess the MPI definitions. The MPI definitions may be obtained using the ${NWCHEM_TOP}/src/tools/guess-mpidefs script:

      ${NWCHEM_TOP}/src/tools/guess-mpidefs

      Note: These values may not be correct.

    6. Export the MPI definitions.

      export MPI_INCLUDE="</path/to/mpi/includes>"
      export MPI_LIB="</path/to/mpi/libs/dir>"
      export LIBMPI="<list of mpi libraries>"

      replacing </path/to/mpi/includes>, </path/to/mpi/libs/dir> and <list of mpi libraries> with your respective installation paths for each.

    7. Define the modules to be compiled:

      export NWCHEM_MODULES="all"
    8. NWChem uses 64 bit integers on a 64 bit systems. Enable ilp64 optimized armpl math libraries, by either:

      • Manually pointing to your installation of Arm Performance Libraries:
        export BLAS_LIBS="-L</path/to/arm/performance/libraries/dir> -larmpl_ilp64"
      • If you have used the provided module then you can use:
        export BLAS_LIBS="-L$ARMPL_DIR/lib -larmpl_ilp64"
        export BLASOPT=${BLAS_LIBS}
        export BLAS_SIZE=8
    9. Enable LAPACK using the same method, either:

      • Manually:
        export LAPACK="-L</path/to/arm/performance/libraries/dir> -larmpl_ilp64"
      • From using the provided module:
        export LAPACK="-L$ARMPL_DIR/lib -larmpl_ilp64"
        export LAPACK_LIBS="$LAPACK"
        export LAPACK_LIB="$LAPACK"
        export LAPACK_SIZE=8
    10. Set FFLAG_INT to -i8:

      FFLAG_INT=-i8
    11. Build and install NWChem, using:

      cd $NWCHEM_TOP/src
      make nwchem_config
      make -j 28 | tee make.log
  8. Test your build by running some QA tests:

    1. Change into the QA directory:

      cd $NWCHEM_TOP/QA
    2. Configure your environment for NWChem and run the doqmtests.mpi example:

      ./doqmtests.mpi 16 | tee doqmtests.mpi.log