Getting Started

Quickstart

This Quickstart guide will guide the user through downloading the code and building/running an Advection test case with GNUMake without MPI or OpenMP. For building with GNUMake, REMORA requires a C++ compiler that supports the C++17 standard and a C compiler that supports the C99 standard. The code is available on Github and can be accessed with git.

git clone --recursive https://github.com/seahorce-scidac/REMORA.git

Now we enter the directory for the problem we’re interested in, in this case Advection:

cd REMORA/Exec/Advection

And build, which will produce the executable REMORA.3d.gnu.TEST.ex.

make -j USE_MPI=FALSE

The executable can be run with the provided inputs file:

./REMORA.3d.gnu.TEST.ex inputs

This will produce an AMReX plotfile at the 10th time step called plt00010 which can be visualized.

A similar process can be used to build other cases within Exec, except for IdealMiniGrid, which requires MPI and PnetCDF.

Downloading the code

First, make sure that git is installed on your machine.

Then download the REMORA repository by typing:

git clone https://github.com/seahorce-scidac/REMORA.git

Or, to automatically include the AMReX submodule when downloading REMORA, type:

git clone --recursive https://github.com/seahorce-scidac/REMORA.git

Git Submodules

Ideally, when using the submodule to build it is properly updated to match what is in the repository. Depending on Git version, different commands and options to ensure these match are available. An example workflow is to run git pull to get the latest commit on your current branch, and then run git submodule update to explicitly update the submodule. This should work for all versions of git which support submodules.

The following example demonstrates a shorter form that combines both commands and requires Git 2.14 or newer:

# Replaces your git pull to use both the updated code and the updated submodule
git pull --recurse-submodules

The following example demonstrates setting defaults in the config file for the repository and requires Git 2.15 or newer:

# Set this once
git config submodule.recurse true
# When configured as true, this will use both the updated code and the updated submodule
git pull
# This option will override any configuration and not update the submodule
git pull --no-recurse-submodules

These example also apply to git checkout. For more details see Git Tools Submodules: https://git-scm.com/book/en/v2/Git-Tools-Submodules

Building

REMORA can be built using either GNU Make or CMake. The following instructions apply to building on any system. We also provide instructions for building on Perlmutter.

Minimum Requirements

REMORA requires a C++ compiler that supports the C++17 standard and a C compiler that supports the C99 standard. Building with GPU support may be done with CUDA, HIP, or SYCL. For CUDA, REMORA requires versions >= 11.0. For HIP and SYCL, only the latest compilers are supported. Prerequisites for building with GNU Make include Python (>= 2.7, including 3) and standard tools available in any Unix-like environments (e.g., Perl and sed). For building with CMake, the minimal requirement is version 3.18.

Note

While REMORA is designed to work with SYCL, we do not make any guarantees that it will build and run on your Intel platform.

Paradigm

REMORA uses the paradigm that different executables are built in different subdirectories within the Exec directory. When using gmake (see below), the user/developer should build in the directory of the selected problem. When using cmake (see below), separate executables are built for all of the problem directories listed in Exec/CMakeLists.txt. The problem directories within Exec include a number of problems, which are also used for testing.

NetCDF (Optional)

REMORA uses PnetCDF for optional NetCDF support. To build REMORA with PnetCDF, first install PnetCDF as per the instructions. Make a note of the directory where the library is installed, which we will call PNETCDF_DIR. When compiling, add the PnetCDF pkgconfig directory to the environment variable PKG_CONFIG_PATH, e.g.:

PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$PNETCDF_DIR/lib/pkgconfig

At run-time, you may need to add PnetCDF to the link path, e.g.:

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PNETCDF_DIR/lib

See sections below for compiler-specific instructions for how to enable NetCDF support.

Note

When built with some MPI implementations, PnetCDF will crash when writing files larger than 2GB. Building with MPICH v4.2.2 will likely solve this issue.

GNU Make

The GNU Make system is best for use on large computing facility machines and production runs. With the GNU Make implementation, the build system will inspect the machine and use known compiler optimizations explicit to that machine if possible. These explicit settings are kept up-to-date by the AMReX project.

Using the GNU Make build system involves first setting environment variables for the directories of the dependencies of REMORA which is the repository of AMReX. AMReX is provided as a git submodule in REMORA and can be populated by using

git submodule init; git submodule update

in the REMORA repo, or before cloning by using

git clone --recursive <remora_repo>

Although submodules of these projects are provided, they can be placed externally as long as the <REPO_HOME> environment variables for each dependency is set correctly. An example of setting the <REPO_HOME> environment variables in the user’s .bashrc is shown below:

export REMORA_HOME=${HOME}/REMORA
export AMREX_HOME=${REMORA_HOME}/Submodules/AMReX

The GNU Make system is set up to use the path to AMReX submodule by default, so it is not necessary to set these paths explicitly, unless it is desired to do so. It is also possible to use an external version of AMReX, downloaded by running

git clone https://github.com/amrex-codes/amrex.git

in which case the AMREX_HOME environment variable must point to the location where AMReX has been downloaded, which will take precedence over the default path to the submodule. If using bash shell,

export AMREX_HOME=/path/to/external/amrex

or if using tcsh,

setenv AMREX_HOME /path/to/external/amrex

cd to the desired build directory, e.g. REMORA/Exec/Upwelling/

Edit the GNUmakefile; options include

Option name	Description	Possible values	Default value
COMP	Compiler (gnu or intel)	gnu / intel	None
USE_MPI	Whether to enable MPI	TRUE / FALSE	FALSE
USE_OMP	Whether to enable OpenMP	TRUE / FALSE	FALSE
USE_CUDA	Whether to enable CUDA	TRUE / FALSE	FALSE
USE_HIP	Whether to enable HIP	TRUE / FALSE	FALSE
USE_SYCL	Whether to enable SYCL	TRUE / FALSE	FALSE
DEBUG	Whether to use DEBUG mode	TRUE / FALSE	FALSE
USE_PNETCDF	Whether to compile with PnetCDF	TRUE / FALSE	FALSE
USE_PARTICLES	Whether to compile with particle functionality enabled	TRUE / FALSE	FALSE
USE_MOAB	Whether to compile with MOAB	TRUE / FALSE	FALSE
PROFILE	Include profiling info	TRUE / FALSE	FALSE
TINY_PROFILE	Include tiny profiling info	TRUE / FALSE	FALSE
COMM_PROFILE	Include comm profiling info	TRUE / FALSE	FALSE
TRACE_PROFILE	Include trace profiling info	TRUE / FALSE	FALSE

Note

Do not set both USE_OMP and USE_CUDA to true.

Information on using other compilers can be found in the AMReX documentation at https://amrex-codes.github.io/amrex/docs_html/BuildingAMReX.html .

Make the executable by typing
```
make
```
The name of the resulting executable (generated by the GNUmake system) encodes several of the build characteristics, including dimensionality of the problem, compiler name, and whether MPI and/or OpenMP were linked with the executable. Thus, several different build configurations may coexist simultaneously in a problem folder. For example, the default build in REMORA/Exec/Upwelling will look like REMORA3d.gnu.MPI.ex, indicating that this is a 3-d version of the code, made with COMP=gnu, and USE_MPI=TRUE.

Job info

The build information can be accessed by typing

./REMORA*ex --describe

in the directory where the executable has been built.

CMake

CMake is often preferred by developers of REMORA; CMake allows for building as well as easy testing and verification of REMORA through the use of CTest which is included in CMake. CTest functionality requires additional options, described in Testing and Verification.

Using CMake involves an additional configure step before using the make command. It is also expected that the user has cloned the REMORA repo with the --recursive option or performed git submodule init; git submodule update in the REMORA repo to populate its submodules.

To build with CMake, a user typically creates a build directory in the project directory and in that directory the cmake <options> .. command is used to configure the project before building it. REMORA provides an example build directory called Build with example scripts for performing the CMake configure. Once the CMake configure step is done, then the make command will build the executable.

The options are described in the table below. Standard CMake options like CMAKE_INSTALL_PREFIX are also available.

Option name

Description

Possible values

Default

value

CMAKE_BUILD_TYPE

Build type

RelWithDebInfo /

Release / Debug

Release

REMORA_ENABLE_MPI

Whether to enable MPI

ON / OFF

OFF

REMORA_ENABLE_OMP

Whether to enable OpenMP

ON / OFF

OFF

REMORA_ENABLE_CUDA

Whether to enable CUDA

ON / OFF

OFF

REMORA_ENABLE_HIP

Whether to enable HIP

ON / OFF

OFF

REMORA_ENABLE_SYCL

Whether to enable SYCL

ON / OFF

OFF

REMORA_ENABLE_PNETCDF

Whether to compile with

PNetCDF

ON / OFF

OFF

REMORA_ENABLE_PARTICLES

Whether to compile with

particle functionality

enabled

ON / OFF

OFF

REMORA_ENABLE_MOAB

Whether to build with MOAB

ON / OFF

OFF

REMORA_ENABLE_TESTS

Whether to build tests

ON / OFF

OFF

REMORA_ENABLE_FCOMPARE

Whether to build the AMReX

utility fcompare

ON / OFF

OFF

REMORA_ENABLE_DOCUMENTATION

Whether to build

documentation

ON / OFF

OFF

An example CMake configure/build command to build REMORA without MPI. Replace the compilers with those installed on your system:

cmake -DCMAKE_BUILD_TYPE:STRING=Release \
      -DREMORA_ENABLE_MPI:BOOL=OFF \
      -DCMAKE_CXX_COMPILER:STRING=g++ \
      .. && make

An example CMake configure/build command to build REMORA with MPI is listed below:

cmake -DCMAKE_BUILD_TYPE:STRING=Release \
      -DREMORA_ENABLE_MPI:BOOL=ON \
      -DCMAKE_CXX_COMPILER:STRING=mpicxx \
      .. && make

An example CMake configure/build command to build REMORA with MPI, PnetCDF, and particles is listed below:

cmake -DCMAKE_BUILD_TYPE:STRING=Release \
      -DREMORA_ENABLE_MPI:BOOL=ON \
      -DCMAKE_CXX_COMPILER:STRING=mpicxx \
      -DREMORA_ENABLE_PARTICLES:BOOL=ON \
      -DREMORA_ENABLE_PNETCDF:BOOL=ON \
      .. && make

Note that CMake is able to generate makefiles for the Ninja build system as well which will allow for faster building of the executable(s).

Perlmutter (NERSC)

Recall the GNU Make system is best for use on large computing facility machines and production runs. With the GNU Make implementation, the build system will inspect the machine and use known compiler optimizations explicit to that machine if possible. These explicit settings are kept up-to-date by the AMReX project.

For Perlmutter at NERSC, initialize your environment by sourcing the saul-env.sh script in the Build directory. For example, from the root of the REMORA directory:

source Build/saul-env.sh

Then follow the general instructions for building REMORA using GNU Make.

Note

When building, GNU Make may complain that it cannot find NetCDF or MPICH. This is fine.

Building for and running on GPU nodes

Then build REMORA as, for example (specify your own path to the AMReX submodule in REMORA/Submodules/AMReX):

make -j 4 COMP=gnu USE_MPI=TRUE USE_OMP=FALSE USE_CUDA=TRUE AMREX_HOME=$HOME/dev-remora.REMORA/Submodules/AMReX

Finally, you can prepare your SLURM job script, using the following as a guide:

#!/bin/bash

## specify your allocation (with the _g) and that you want GPU nodes
#SBATCH -A mXXXX_g
#SBATCH -C gpu

## the job will be named "REMORA" in the queue and will save stdout to remora_[job ID].out
#SBATCH -J REMORA
#SBATCH -o remora_%j.out

## set the max walltime
#SBATCH -t 10

## specify the number of nodes you want
#SBATCH -N 2

## we use the same number of MPI ranks per node as GPUs per node
#SBATCH --ntasks-per-node=4
#SBATCH --gpus-per-node=4
#SBATCH --gpu-bind=none

# pin to closest NIC to GPU
export MPICH_OFI_NIC_POLICY=GPU

# use GPU-aware MPI
#GPU_AWARE_MPI=""
GPU_AWARE_MPI="amrex.use_gpu_aware_mpi=1"

# the -n argument is (--ntasks-per-node) * (-N) = (number of MPI ranks per node) * (number of nodes)
# set ordering of CUDA visible devices inverse to local task IDs for optimal GPU-aware MPI
srun -n 8 --cpus-per-task=32 --cpu-bind=cores bash -c "
  export CUDA_VISIBLE_DEVICES=\$((3-SLURM_LOCALID));
  ./REMORA3d.gnu.MPI.CUDA.ex inputs ${GPU_AWARE_MPI}" \
> test.out

To submit your job script, do sbatch [your job script] and you can check its status by doing squeue -u [your username].

Building for and running on CPU nodes

Then build REMORA as, for example (specify your own path to the AMReX submodule in REMORA/Submodules/AMReX):

make -j 4 COMP=gnu USE_MPI=TRUE USE_OMP=TRUE USE_CUDA=FALSE AMREX_HOME=$HOME/dev-remora.REMORA/Submodules/AMReX

Finally, you can prepare your SLURM job script, using the following as a guide:

#!/bin/bash

#SBATCH -A mXXXX
#SBATCH -C cpu
#SBATCH -q regular

## the job will be named "REMORA" in the queue and will save stdout to remora_[job ID].out
#SBATCH -J REMORA
#SBATCH -o remora_%j.out

## set the max walltime
#SBATCH -t 10

## specify the number of nodes you want
#SBATCH -N 2

## we use 4 ranks per node here as an example. This may not optimize performance
#SBATCH --ntasks-per-node=4

## This configuration assigns one OpenMP thread per physical CPU core.
## For this type of thread assignment, we want 128 total threads per node, so we should
## have (OMP_NUM_THREADS * ntasks-per-node) = 128
export OMP_PROC_BIND=spread
export OMP_PLACES=threads
export OMP_NUM_THREADS=32

# the -n argument is (--ntasks-per-node) * (-N) = (number of MPI ranks per node) * (number of nodes)
srun -n 8 ./REMORA3d.gnu.x86-milan.MPI.OMP.ex inputs > test.out

To submit your job script, do sbatch [your job script] and you can check its status by doing squeue -u [your username].

Running

The input file specified on the command line is a free-format text file, one entry per row, that specifies input data processed by the AMReX ParmParse module.

This file needs to be specified along with the executable as an argv option, for example:

mpirun -np 64 ./REMORA3d.xxx.yyy.ex inputs

Also, any entry that can be specified in the inputs file can also be specified on the command line; values specified on the command line override values in the inputs file, e.g.:

mpirun -np 64 ./REMORA3d.gnu.DEBUG.MPI.ex inputs amr.restart=chk0030 remora.use_gravity=true

See Inputs for details on run-time options that can be specified. If running on a Mac and getting errors like SIGILL Invalid, privileged, or ill-formed instruction, see the note on that page about runtime error-checking options.