How to build NEMO with Arm Compiler.

NEMO (Nucleus for European Modelling of the Ocean) is a state-of-the-art modeling framework for oceanographic research, operational oceanography seasonal forecast, and climate studies. For more information on NEMO, see the NEMO website.

For the purposes of this build, the following components are used:

Component Form
NEMO Version 3.6
Arm Compiler
Version 18.3
Open MPI Version 3.0.2
Arm Performance Libraries Version 18.3
Boost Version 1.67.0
NetCDF Version 4.6.1
HDF5 Version 1.10.1
Blitz++
XIOS Version 1.0
Operating system RHEL 7.5
Hardware
Cavium ThunderX2

Before you begin

  • Installed Arm Compiler and Arm Performance Libraries. For more information, see our installation instructions in the Installation instructions topic.

  • Configured Arm Compiler, using:

    export CC=armclang
    export CXX=armclang++
    export FC=armflang
    export F90=armflang
    export F77=armflang
  • Built and installed Open MPI. For more information on building Open MPI with Arm Compiler, see our instructions on Building OpenMPI with Arm Compiler.

  • Built and installed HDF5. For more information on building HDF5 with Arm Compiler, see our instructions on Building HDF5 with Arm Compiler.

  • Built and installed NetCDF with both the C library and the NetCDF Fortran interface. For more information on building NetCDF and the NetCDF Fortran interface with Arm Compiler, see our instructions on Building NetCDF with Arm Compiler.

  • Built and installed Boost with Arm Compiler. For more information on building Boost with Arm Compiler, see our instructions on Building Boost with Arm Compiler.

  • Configured your environment by setting the following environment variables:

    • Set HDR_DIR to the directory containing the HDF5 include and lib directories.
    • Set HDF_INCLUDE_DIR to HDF5 library include directory.
    • Set HDF_LIBRARY_DIR to HDF library lib directory.
    • Set ARMPL_DIR to Arm Performance Libraries installation directory.
    • Set NETCDF_INCLUDE_DIR to NetCDF library include directory.
    • Set NETCDF_LIBRARY_DIR to NetCDF library lib directory.
    • Set BOOST_INCLUDEDIR to Boost library include directory.
    • Set BOOST_LIBRARYDIR to Boost library lib directory.

Procedure - Build blitz++

  1. Download blitz++:

    wget https://github.com/blitzpp/blitz/archive/master.zip
    unzip master.zip
  2. Build blitz++ library, and set the BLITZ_INCLUDEDIR environment variable:

    1. Set an install location for blitz++:

      export BLITZ_INSTALL_DIR=<path/to/install/dir>
    2. From within the blitz++ top level directory, configure and build the application:

      autoreconf -fiv
      ./configure --prefix=$BLITZ_INSTALL_DIR --enable-fortran --enable-64bit
      make lib
      make install
    3. Set BLITZ_INCLUDEDIR for later use:

      export BLITZ_INCLUDEDIR=$BLITZ_INSTALL_DIR/include

Procedure - Build XIOS

  1. Checkout xios-1.0 from the Xios SVN repository:

    svn co -r 703 http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/branchs/xios-1.0 xios-1.0
  2. Within the XIOS 1.0 top-level directory (xios-1.0), navigate to the arch directory. Create three arch files for AArch64:

    • arch-AARCH64_ARM_LINUX.env

      Containing the key environment variables:

      export HDF5_INC_DIR=$HDF_INCLUDE_DIR
      export HDF5_LIB_DIR=$HDF_LIBRARY_DIR
      
      export NETCDF_INC_DIR=$NETCDF_INCLUDE_DIR
      export NETCDF_LIB_DIR=$NETCDF_LIBRARY_DIR
      
      export BOOST_INC_DIR=$BOOST_INCLUDEDIR
      export BOOST_LIB_DIR=$BOOST_LIBRARYDIR
      
      export BLITZ_INC_DIR=$BLITZ_INCLUDEDIR
      export BLITZ_LIB_DIR=""
    • arch-AARCH64_ARM_LINUX.path

      Containing the include and lib locations:

      NETCDF_INCDIR="-I $NETCDF_INC_DIR"
      NETCDF_LIBDIR="-L $NETCDF_LIB_DIR"
      NETCDF_LIB="-lnetcdff -lnetcdf"
      
      MPI_INCDIR=""
      MPI_LIBDIR=""
      MPI_LIB="-lcurl"
      
      HDF5_INCDIR="-I $HDF5_INC_DIR"
      HDF5_LIBDIR="-L $HDF5_LIB_DIR"
      HDF5_LIB="-lhdf5_hl -lhdf5 -lhdf5 -lz"
      
      BOOST_INCDIR="-I $BOOST_INC_DIR"
      BOOST_LIBDIR="-L $BOOST_LIB_DIR"
      BOOST_LIB=""
      
      BLITZ_INCDIR="-I $BLITZ_INC_DIR"
      BLITZ_LIBDIR=""
      BLITZ_LIB=""
    • arch-AARCH64_ARM_LINUX.fcm

      Containing the details of the compiler build and flags:

      #####################################################################
      #############        Projet xios - xmlioserver       ################
      #####################################################################
          %CCOMPILER      mpicc
          %FCOMPILER      mpifort
          %LINKER         mpifort
          %BASE_CFLAGS    -ansi -w -D_GLIBCXX_USE_CXX11_ABI=0
          %PROD_CFLAGS    -O3 -DBOOST_DISABLE_ASSERTS
          %DEV_CFLAGS     -g -O2
          %DEBUG_CFLAGS   -g
          %BASE_FFLAGS    -D__NONE__
          %PROD_FFLAGS    -O3
          %DEV_FFLAGS     -g -O2
          %DEBUG_FFLAGS   -g
          %BASE_INC       -D__NONE__
          %BASE_LD        -lstdc++
          %CPP            cpp
          %FPP            cpp -P
          %MAKE           gmake
  3. To allow building with external boost and blitz++ libraries, in the top-level xios-1.0 directory:

    1. Modify the bld.cfg file:

      sed -i -e 's/src::date/#src::date/' bld.cfg
      sed -i -e 's/src::blitz/#src::blitz/' bld.cfg
    2. Update the make_xios file by applying this patch:

      --- make_xios.old   2018-07-18 12:30:04.735527495 +0100
      +++ make_xios   2018-04-10 16:33:51.000000000 +0100
      @@ -3,6 +3,8 @@
       install_dir=$PWD
       compil_full="false"
       use_oasis="false"
      +use_extern_boost="false"
      +use_extern_blitz="false"
       oasis="oasis3_mct"
       use_memtrack="false"
       job="1"
      @@ -24,6 +26,8 @@
              echo "       [--avail] : to know available target architectures "
              echo "       [--full] : to generate dependencies and recompile from scratch"
              echo "       [--use_oasis 'oasis3' 'oasis3_mct' : default oasis3_mct] : to use Oasis coupler"
      +        echo "       [--use_extern_boost : to use external boost library"
      +        echo "       [--use_extern_blitz : to use external blitz library"
              echo "       [--doc] : to generate Doxygen documentation (not available yet)"
              echo "       [--job ntasks] : to use parallel compilation with ntasks"
              echo "       [--netcdf_lib 'netcdf4_par'/'netcdf4_seq'/'netcdf4_internal' : default netcdf4_par] : choice of netcdf library"
      @@ -38,6 +42,8 @@
                "--avail") ls arch/*.fcm | cut -d"-" -f2 | cut -d"." -f1 ; exit ;;
                "--full")  compil_full="true"  ; shift ;;
                "--use_oasis")  use_oasis="true" oasis=$2 ; shift ; shift  ;;
      +         "--use_extern_boost")  use_extern_boost="true" ; shift  ;;
      +         "--use_extern_blitz")  use_extern_blitz="true" ; shift  ;;
                "--doc")   doc="true"          ; shift ;;
                "--job")   job=$2              ; shift ; shift ;;
                "--netcdf_lib")   netcdf_lib=$2 ; shift ; shift ;;
      @@ -52,6 +58,10 @@
       then
           echo -e "- uncompress archives ..."        
           for tarname in `ls $install_dir/tools/archive/*.tar.gz` ; do
      +        if  ( [[ ${tarname} == "${install_dir}/tools/archive/boost.tar.gz" ]] && [[ "$use_extern_boost" == "true" ]] ) || ( [[ ${tarname} == "${install_dir}/tools/archive/blitz.tar.gz" ]] && [[ "$use_extern_blitz" == "true" ]] )
      +        then
      +                continue
      +        fi
          gunzip -f "$tarname"
          tar -xf ${tarname%.gz}
           done
      @@ -148,13 +158,38 @@
           XMLIO_LIB="$OASIS_LIBDIR $OASIS_LIB $XMLIO_LIB"
       fi

      +# Setting path for boost
      +if [[ "$use_extern_boost" == "true" ]]
      +then
      +   rm -r $PWD/extern/boost
      +   ln -s $PWD/.void_dir $PWD/extern/boost
      +else
      +   export BOOST_INCDIR="-I${PWD}/extern/boost"
      +   export BOOST_LIBDIR=""
      +   export BOOST_LIB=""
      +fi
      +
      +# Setting path for blitz
      +if [[ "$use_extern_blitz" == "true" ]]
      +then
      +   rm -r $PWD/extern/blitz
      +   ln -s $PWD/.void_dir $PWD/extern/blitz
      +else
      +   export BLITZ_INCDIR="-I${PWD}/extern/blitz"
      +   export BLITZ_LIBDIR=""
      +   export BLITZ_LIB=""
      +fi
      +
      +
      +
       if [[ "$use_memtrack" == "true" ]]
          then
          XMLIO_LIB="$ADDR2LINE_LIBDIR $ADDR2LINE_LIB $XMLIO_LIB"
          XMLIO_CPPKEY="$XMLIO_CPPKEY XIOS_MEMTRACK"
       fi

      -XMLIO_CINCDIR="$NETCDF_INCDIR $HDF5_INCDIR $MPI_INCDIR"
      +# XMLIO_CINCDIR="$NETCDF_INCDIR $HDF5_INCDIR $MPI_INCDIR"
      +XMLIO_CINCDIR="$NETCDF_INCDIR $HDF5_INCDIR $MPI_INCDIR $BOOST_INCDIR $BLITZ_INCDIR"
       XMLIO_FINCDIR="$XMLIO_FINCDIR $MPI_INCDIR"

       XMLIO_LIB="$XMLIO_LIB $NETCDF_LIBDIR $HDF5_LIBDIR $MPI_LIBDIR $NETCDF_LIB $HDF5_LIB $MPI_LIB
  4. Build the XIOS library, and set the XIOS_DIR environment variable:

    ./make_xios --dev --job 32 --full --arch AARCH64_ARM_LINUX
    export XIOS_DIR=`pwd`

Procedure - Build NEMO

  1. Download the NEMO package. Obtain the reference version from the NEMPGCM SVN repository.

    svn co http://forge.ipsl.jussieu.fr/nemo/svn/NEMO/releases/release-3.6/NEMOGCM
    cd NEMOGCM
  2. Create an arch-aarch64_arm.fcm file in the ARCH directory to contain the details of the compiler, build flags, and the locations of the various libraries required:

    # generic gfortran compiler options for linux
    # NCDF_INC    netcdf include file
    # NCDF_LIB    netcdf library
    # FC          Fortran compiler command
    # FCFLAGS     Fortran compiler flags
    # FFLAGS      Fortran 77 compiler flags
    # LD          linker
    # LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a
    # FPPFLAGS    pre-processing flags
    # AR          assembler
    # ARFLAGS     assembler flags
    # MK          make
    # USER_INC    additional include files for the compiler,  e.g. -I<include dir>
    # USER_LIB    additional libraries to pass to the linker, e.g. -l<library>
    # CC          C compiler used to compile conv for AGRIF
    # CFLAGS      compiler flags used with CC
    #
    # Note that:         
    #  - unix variables "$..." are accepted and will be evaluated before calling fcm.
    #  - fcm variables are starting with a % (and not a $)


    %NCDF_HOME           $NETCDF_DIR
    %HDF5_HOME           $HDF_DIR
    %XIOS_HOME           $XIOS_DIR

    %NCDF_INC            -I$NETCDF_INCLUDE_DIR -I$HDF_INCLUDE_DIR
    %NCDF_LIB            -L$HDF_LIBRARY_DIR -L$NETCDF_LIBRARY_DIR -lnetcdf -lnetcdff
    %XIOS_INC            -I%XIOS_HOME/inc
    %XIOS_LIB            -L%XIOS_HOME/lib -lxios

    %CPP                 cpp
    %FC                  mpifort
    %FCFLAGS             -cpp -r8 -O3 -march=armv8.1-a -mcpu=native -ffp-contract=fast -fcray-pointer
    %FFLAGS              %FCFLAGS
    %LD                  mpif90
    %LDFLAGS             -lstdc++
    %FPPFLAGS            -P -C -traditional
    %AR                  ar
    %ARFLAGS             -rs
    %MK                  gmake
    #%USER_INC            %NCDF_INC
    #%USER_LIB            %NCDF_LIB
    %USER_INC            %XIOS_INC %NCDF_INC -I$ARMPL_DIR/include
    %USER_LIB            %XIOS_LIB %NCDF_LIB -L$ARMPL_DIR/lib -lamath

    %CC                  armclang
    %CFLAGS              -O0

    Note: References to -lmath (including the ArmPL lib and include directories) are optional and can be omitted. Linking against libamath provides optimized versions of common math functions, such as exp, pow, and similar.

  3. Move to the CONFIG directory and build NEMO using the AMM12 config:

    cd CONFIG
    ./makenemo -m aarch64_arm -r AMM12 -n 'MY_AMM12' add_key key_nosignedzero
  4. Test the build:

    1. Download the AMM12 test data for NEMO 3.6 into the MY_AMM12/EXP00 directory:

      cd MY_AMM12/EXP00
      wget https://prodn.idris.fr/thredds/fileServer/ipsl_public/reee451/NEMO/AMM12_v3.6.tar
    2. Unpack and uncompress:

      tar xf AMM12_v3.6.tar
      gunzip *.gz */*.gz
    3. Run the MPI job:

      Note: Set the number of MPI tasks to suit your system by substituting 56 with the number of tasks you wish to test with.

      mpirun -np 56 ./opa