OpenFOAM

How to build OpenFOAM with Arm Compiler for HPC.

Introduction

OpenFOAM is a free, open source CFD software used to solve a variety of problems, such as complex fluid flows, chemical reactions, acoustics, solid mechanics, and electromagnetics

For more information on OpenFOAM, visit the OpenFOAM Foundation webpage.

Version information

For the purposes of this build, the following components are used:

Component
Form
OpenFOAM Version 5.0
Open MPI Version 3.0.2
Operating System
RHEL 7.5
Hardware Cavium ThunderX2
Arm Compiler for HPC
Version 18.3

Download

Create a new directory called OpenFOAM:

mkdir ~/OpenFOAM
cd ~/OpenFOAM

Download the OpenFOAM source files by cloning repositories, using:

git clone git://github.com/OpenFOAM/OpenFOAM-5.x.git
git clone git://github.com/OpenFOAM/ThirdParty-5.x.git

User configuration

To configure your environment:

  1. Setup your directory variables, using:

    BUILD_DIR=`pwd`
    BUILD_JOBS=32
  2. Configure Arm Compiler for HPC, using:

    export CC=armclang
    export CXX=armclang++
    export FC=armflang
    export OMPI_CC=$CC
    export OMPI_CXX=$CXX
    export OMPI_FC=$FC
  3. Setup the build to use Arm Compiler for HPC, using:

    echo "export WM_COMPILER=Clang" > $BUILD_DIR/OpenFOAM-5.x/etc/prefs.sh

    OF_COPT="-march=armv8.1-a -mtune=thunderx2t99 -O3 -Ofast -ffp-contract=fast"
    OF_CXXOPT="$OF_COPT -std=c++11"

    cp -r $BUILD_DIR/OpenFOAM-5.x/wmake/rules/linux64Clang $BUILD_DIR/OpenFOAM-5.x/wmake/rules/linuxAArch64Clang

Pre-build configuration

To preconfigure your environment for building OpenFOAM with Arm Compiler for HPC:

  1. Patch the newly copied build system

    patch -d $BUILD_DIR/OpenFOAM-5.x -p1 <<EOF
    --- a/etc/config.sh/settings
    +++ b/etc/config.sh/settings
    @@ -110,6 +110,16 @@
             export WM_LDFLAGS='-m64'
             ;;

    +    aarch64)
    +        WM_ARCH=linuxAArch64
    +        export WM_COMPILER_LIB_ARCH=64
    +        export WM_CC='$CC'
    +        export WM_CXX='$CXX'
    +        export WM_CFLAGS='-fPIC'
    +        export WM_CXXFLAGS='-fPIC'
    +        export WM_LDFLAGS=
    +        ;;
    +
         *)
             echo Unknown processor type `uname -m` for Linux 1>&2
             ;;
    --- a/wmake/rules/linuxAArch64Clang/c
    +++ b/wmake/rules/linuxAArch64Clang/c
    @@ -2,7 +2,7 @@

     cWARN        = -Wall

    -cc          = clang -m64
    +cc          = $CC

     include \$(RULES)/c\$(WM_COMPILE_OPTION)

    --- a/wmake/rules/linuxAArch64Clang/c++
    +++ b/wmake/rules/linuxAArch64Clang/c++
    @@ -5,7 +5,7 @@
     # Suppress some warnings for flex++ and CGAL
     c++LESSWARN = -Wno-old-style-cast -Wno-unused-local-typedefs -Wno-tautological-undefined-compare -Wno-shift-negative-value

    -CC          = clang++ -std=c++11 -m64
    +CC          = $CXX

     include \$(DEFAULT_RULES)/c++\$(WM_COMPILE_OPTION)

    --- a/wmake/rules/linuxAArch64Clang/cOpt
    +++ b/wmake/rules/linuxAArch64Clang/cOpt
    @@ -1,2 +1,2 @@
     cDBUG       =
    -cOPT        = -O3
    +cOPT        = $OF_COPT
    --- a/wmake/rules/linuxAArch64Clang/c++Opt
    +++ b/wmake/rules/linuxAArch64Clang/c++Opt
    @@ -1,2 +1,2 @@
     c++DBUG     =
    -c++OPT      = -O3
    +c++OPT      = $OF_CXXOPT
    --- a/wmake/rules/linuxAArch64Clang/mplibHPMPI
    +++ b/wmake/rules/linuxAArch64Clang/mplibHPMPI
    @@ -1,3 +1,3 @@
     PFLAGS     =
     PINC       = -isystem \$(MPI_ARCH_PATH)/include -D_MPICC_H
    -PLIBS      = -L\$(MPI_ARCH_PATH)/lib/linux_amd64 -lmpi
    +PLIBS      = -L\$(MPI_ARCH_PATH)/lib -lmpi
    EOF
  2. Configure the FOAM_INST_DIR and WM_NCOMPPROCS variables, using:

    export FOAM_INST_DIR=$BUILD_DIR
    export WM_NCOMPPROCS=$BUILD_JOBS
    source $BUILD_DIR/OpenFOAM-5.x/etc/bashrc

Optional: Build Paraview

Paraview is required for running visualizations on Arm.

Note: OpenGL drivers are needed to run. To test this, install and run glxgears.

To build Paraview:

  1. Prepare your environment, using:

    export WM_NCOMPPROCS=$BUILD_JOBS
    source $BUILD_DIR/OpenFOAM-5.x/etc/bashrc
  2. Change directories and build Paraview, using:

    cd $WM_THIRD_PARTY_DIR
    ./makeParaView
    wmRefresh
    cd ${BUILD_DIR}

Build and install OpenFOAM with Arm Compiler for HPC

To build and install OpenFOAM with Arm Compiler for HPC:

  1. Perform some minor patching, using:

    patch -d $BUILD_DIR/OpenFOAM-5.x -p1 <<EOF
    --- a/src/conversion/ensight/part/ensightPart.C
    +++ b/src/conversion/ensight/part/ensightPart.C
    @@ -27,6 +27,7 @@
     #include "dictionary.H"
     #include "ListOps.H"
     #include "addToRunTimeSelectionTable.H"
    +#include <math.h>

     // * * * * * * * * * * * * * * Static Data Members * * * * * * * * * * * * * //
     namespace Foam
    --- a/src/conversion/ensight/part/ensightPartIO.C
    +++ b/src/conversion/ensight/part/ensightPartIO.C
    @@ -29,6 +29,7 @@
     #include "ensightPart.H"
     #include "dictionary.H"
     #include "IOstreams.H"
    +#include <math.h>

     // * * * * * * * * * * * * * Private Member Functions  * * * * * * * * * * * //

    --- a/src/finiteVolume/finiteVolume/ddtSchemes/CrankNicolsonDdtScheme/CrankNicolsonDdtScheme.C
    +++ b/src/finiteVolume/finiteVolume/ddtSchemes/CrankNicolsonDdtScheme/CrankNicolsonDdtScheme.C
    @@ -101,7 +101,7 @@

     template<class Type>
     template<class GeoField>
    -CrankNicolsonDdtScheme<Type>::DDt0Field<GeoField>&
    +typename CrankNicolsonDdtScheme<Type>::template DDt0Field<GeoField>&
     CrankNicolsonDdtScheme<Type>::ddt0_
     (
         const word& name,
    EOF
  2. Configure your environment and make the build, using:

    export FOAM_INST_DIR=$BUILD_DIR
    export WM_NCOMPPROCS=$BUILD_JOBS
    echo "export WM_COMPILER=Clang" > $BUILD_DIR/OpenFOAM-5.x/etc/prefs.sh
    source $BUILD_DIR/OpenFOAM-5.x/etc/bashrc

    cd $WM_PROJECT_DIR
    ./Allwmake -j ${BUILD_JOBS}

Run a simple benchmark

To run a simple benchmark, use:

Note: This benchmark typically runs on the scale of minutes, however this will vary from system‑to‑system.

source $BUILD_DIR/OpenFOAM-5.x/etc/bashrc

mkdir -p $FOAM_RUN

cp -r $FOAM_TUTORIALS/incompressible/icoFoam/cavity/cavity $FOAM_RUN
cd $FOAM_RUN
cd cavity
blockMesh
icoFoam

More resources