Getting Started
Downloading the code
First, make sure that git is installed on your machine.
Then download the REMORA repository by typing:
git clone https://github.com/seahorce-scidac/REMORA.git
Or, to automatically include the AMReX submodule when downloading REMORA, type:
git clone --recursive https://github.com/seahorce-scidac/REMORA.git
Git Submodules
Ideally, when using the submodule to build it is properly updated to match what is in the repository. Depending on Git version, different commands and options to ensure these match are available. An example workflow is to run git pull
to get the latest commit on your current branch, and then run git submodule update
to explicitly update the submodule. This should work for all versions of git
which support submodules.
The following example demonstrates a shorter form that combines both commands and requires Git 2.14 or newer:
# Replaces your git pull to use both the updated code and the updated submodule git pull --recurse-submodules
The following example demonstrates setting defaults in the config file for the repository and requires Git 2.15 or newer:
# Set this once git config submodule.recurse true # When configured as true, this will use both the updated code and the updated submodule git pull # This option will override any configuration and not update the submodule git pull --no-recurse-submodules
These example also apply to git checkout
. For more details see Git Tools Submodules: https://git-scm.com/book/en/v2/Git-Tools-Submodules
Building
REMORA can be built using either GNU Make or CMake.
GNU Make
The GNU Make system is best for use on large computing facility machines and production runs. With the GNU Make implementation, the build system will inspect the machine and use known compiler optimizations explicit to that machine if possible. These explicit settings are kept up-to-date by the AMReX project.
Using the GNU Make build system involves first setting environment variables for the directories of the dependencies of REMORA which is the repository of AMReX. AMReX is provided as a git submodule in REMORA and can be populated by using git submodule init; git submodule update
in the REMORA repo, or before cloning by using git clone --recursive <remora_repo>
. Although submodules of these projects are provided, they can be placed externally as long as the <REPO_HOME>
environment variables for each dependency is set correctly. An example of setting the <REPO_HOME>
environment variables in the user’s .bashrc
is shown below:
export REMORA_HOME=${HOME}/REMORA
export AMREX_HOME=${REMORA_HOME}/Submodules/AMReX
The GNU Make system is set up to use the path to AMReX submodule by default, so it is not necessary to set these paths explicitly, unless it is desired to do so. It is also possible to use an external version of AMReX, downloaded by running
git clone https://github.com/amrex-codes/amrex.git
in which case the AMREX_HOME
environment variable must point to the location where AMReX has been downloaded, which will take precedence over the default path to the submodule. If using bash shell,
export AMREX_HOME=/path/to/external/amrex
or if using tcsh,
setenv AMREX_HOME /path/to/external/amrex
cd
to the desired build directory, e.g.REMORA/Exec/Upwelling/
Edit the
GNUmakefile
; options includeOption name
Description
Possible values
Default value
COMP
Compiler (gnu or intel)
gnu / intel
None
USE_MPI
Whether to enable MPI
TRUE / FALSE
FALSE
USE_OMP
Whether to enable OpenMP
TRUE / FALSE
FALSE
USE_CUDA
Whether to enable CUDA
TRUE / FALSE
FALSE
DEBUG
Whether to use DEBUG mode
TRUE / FALSE
FALSE
USE_NETCDF
Whether to compile with NETCDF
TRUE / FALSE
FALSE
USE_PARTICLES
Whether to compile with particle functionality enabled
TRUE / FALSE
FALSE
PROFILE
Include profiling info
TRUE / FALSE
FALSE
TINY_PROFILE
Include tiny profiling info
TRUE / FALSE
FALSE
COMM_PROFILE
Include comm profiling info
TRUE / FALSE
FALSE
TRACE_PROFILE
Include trace profiling info
TRUE / FALSE
FALSE
Note
Do not set both USE_OMP and USE_CUDA to true.
Information on using other compilers can be found in the AMReX documentation at https://amrex-codes.github.io/amrex/docs_html/BuildingAMReX.html .
Make the executable by typing
make
The name of the resulting executable (generated by the GNUmake system) encodes several of the build characteristics, including dimensionality of the problem, compiler name, and whether MPI and/or OpenMP were linked with the executable. Thus, several different build configurations may coexist simultaneously in a problem folder. For example, the default build in
REMORA/Exec/Upwelling
will look likeREMORA3d.gnu.MPI.ex
, indicating that this is a 3-d version of the code, made withCOMP=gnu
, andUSE_MPI=TRUE
.
Job info
The build information can be accessed by typing
./REMORA*ex --describe
in the directory where the executable has been built.
CMake
CMake is often preferred by developers of REMORA; CMake allows for building as well as easy testing and verification of REMORA through the use of CTest which is included in CMake. CTest functionality requires additional options, described in Testing and Verification.
Using CMake involves an additional configure step before using the make
command. It is also expected that the user has cloned the REMORA repo with the --recursive
option or performed git submodule init; git submodule update
in the REMORA repo to populate its submodules.
To build with CMake, a user typically creates a build
directory in the project directory and in that directory the cmake <options> ..
command is used to configure the project before building it. REMORA provides an example build directory called Build
with example scripts for performing the CMake configure. Once the CMake configure step is done, then the make
command will build the executable.
An example CMake configure command to build REMORA with MPI is listed below:
cmake -DCMAKE_BUILD_TYPE:STRING=Release \
-DREMORA_ENABLE_MPI:BOOL=ON \
-DCMAKE_CXX_COMPILER:STRING=mpicxx \
-DCMAKE_C_COMPILER:STRING=mpicc \
-DCMAKE_Fortran_COMPILER:STRING=mpifort \
.. && make
An example CMake configure command to build REMORA with MPI and particles is listed below:
cmake -DCMAKE_BUILD_TYPE:STRING=Release \
-DREMORA_ENABLE_MPI:BOOL=ON \
-DCMAKE_CXX_COMPILER:STRING=mpicxx \
-DCMAKE_C_COMPILER:STRING=mpicc \
-DCMAKE_Fortran_COMPILER:STRING=mpifort \
-DREMORA_ENABLE_PARTICLES:BOOL=ON \
.. && make
Note that CMake is able to generate makefiles for the Ninja build system as well which will allow for faster building of the executable(s).
Perlmutter (NERSC)
Recall the GNU Make system is best for use on large computing facility machines and production runs. With the GNU Make implementation, the build system will inspect the machine and use known compiler optimizations explicit to that machine if possible. These explicit settings are kept up-to-date by the AMReX project.
For Perlmutter at NERSC, look at the general instructions for building REMORA using GNU Make, and then you can initialize your environment by sourcing or running the saul-env.sh script in the Build directory. GNU Make may complain that it cannot find NetCDF. This is fine.
Then build REMORA as, for example (specify your own path to the AMReX submodule in REMORA/Submodules/AMReX):
make -j 4 COMP=gnu USE_MPI=TRUE USE_OMP=FALSE USE_CUDA=TRUE AMREX_HOME=/global/u2/d/dwillcox/dev-remora.REMORA/Submodules/AMReX USE_SUNDIALS=FALSE
Finally, you can prepare your SLURM job script, using the following as a guide:
#!/bin/bash ## specify your allocation (with the _g) and that you want GPU nodes #SBATCH -A mXXXX_g #SBATCH -C gpu ## the job will be named "REMORA" in the queue and will save stdout to remora_[job ID].out #SBATCH -J REMORA #SBATCH -o remora_%j.out ## set the max walltime #SBATCH -t 10 ## specify the number of nodes you want #SBATCH -N 2 ## we use the same number of MPI ranks per node as GPUs per node #SBATCH --ntasks-per-node=4 #SBATCH --gpus-per-node=4 #SBATCH --gpu-bind=none # pin to closest NIC to GPU export MPICH_OFI_NIC_POLICY=GPU # use GPU-aware MPI #GPU_AWARE_MPI="" GPU_AWARE_MPI="amrex.use_gpu_aware_mpi=1" # the -n argument is (--ntasks-per-node) * (-N) = (number of MPI ranks per node) * (number of nodes) # set ordering of CUDA visible devices inverse to local task IDs for optimal GPU-aware MPI srun -n 8 --cpus-per-task=32 --cpu-bind=cores bash -c " export CUDA_VISIBLE_DEVICES=\$((3-SLURM_LOCALID)); ./REMORA3d.gnu.MPI.CUDA.ex inputs ${GPU_AWARE_MPI}" \ > test.out
To submit your job script, do sbatch [your job script] and you can check its status by doing squeue -u [your username].
Running
The input file specified on the command line is a free-format text file, one entry per row, that specifies input data processed by the AMReX ParmParse
module.
This file needs to be specified along with the executable as an argv
option, for example:
mpirun -np 64 ./REMORA3d.xxx.yyy.ex inputs
Also, any entry that can be specified in the inputs file can also be specified on the command line; values specified on the command line override values in the inputs file, e.g.:
mpirun -np 64 ./REMORA3d.gnu.DEBUG.MPI.ex inputs amr.restart=chk0030 remora.use_gravity=true
See Inputs for details on run-time options that can be specified.
Testing and Verification
Testing and verification of REMORA can be performed using CTest, which is included in the CMake build system. If one builds REMORA with CMake, the testing suite, and the verification suite, can be enabled during the CMake configure step. A nightly test is reflected on the dashboard here .
An example cmake
configure command performed in the Build
directory in REMORA is shown below with options relevant to the testing suite:
cmake -DCMAKE_INSTALL_PREFIX:PATH=./install \
-DCMAKE_BUILD_TYPE:STRING=Release \
-DREMORA_ENABLE_MPI:BOOL=ON \
-DCMAKE_CXX_COMPILER:STRING=mpicxx \
-DCMAKE_C_COMPILER:STRING=mpicc \
-DCMAKE_Fortran_COMPILER:STRING=mpifort \
-DREMORA_ENABLE_FCOMPARE:BOOL=ON \
-DREMORA_ENABLE_TESTS:BOOL=ON \
-DREMORA_USE_CPP:BOOL=ON \
..
While performing a cmake -LAH ..
command will give descriptions of every option for the CMake project. Descriptions of particular options regarding the testing suite are listed below:
REMORA_ENABLE_FCOMPARE – builds the fcompare
utility from AMReX as well as the executable(s), to allow for testing differences between plot files
REMORA_ENABLE_TESTS – enables the base level regression test suite that will check whether each test will run its executable to completion successfully
Building the Tests
Once the user has performed the CMake configure step, the make
command will build
every executable required for each test.
In this step, it is highly beneficial for the user to use the -j
option for make
to build source files in parallel.
Running the Tests
Once the test executables are built, CTest also creates working directories for each test within the Build
directory
where plot files will be output, etc. This directory is analogous to the source location of the tests in Tests/test_files
.
To run the test suite, run ctest
in the Build
directory. CTest will run the tests and report their exit status.
Useful options for CTest are -VV
which runs in a verbose mode where the output of each test can be seen. -R
where a regex string can be used to run specific sets of tests. -j
where CTest will bin pack and run tests in
parallel based on how many processes each test is specified to use and fit them into the amount of cores available
on the machine. -L
where the subset of tests containing a particular label will be run.
(We note that using -L 'regression'
will run all the tests that do not use SUNDIALS.)
Output for the last set of tests run is available in the Build
directory in Testing/Temporary/LastTest.log
.
Adding Tests
Developers are encouraged to add tests to REMORA and in this section we describe how the tests are organized in the
CTest framework. The locations (relative to the REMORA code base) of the tests are in Tests
. To add a test, first
create a problem directory with a name in Exec/<prob_name>
. This problem directory is meant for a production
run where the simulation is run until convergence or a solution is developed. This problem setup could comprise
of a more complex physics than the corresponding tests for regression at Tests/test_files/<test_name>
. Prepare
toned down versions of the input file(s) for each combination of physics that a regression test is desired.
For example, Upwelling
problem with input file Exec/Upwelling/inputs
solves the double gyre problem. The corresponding regression test is driven by the input files
Tests/test_files/Upwelling/Upwelling.i
.
Any file in the test directory will be copied during CMake configure to the test’s working directory.
The input files meant for regression test run only until a few time steps. The reference solution that the
regression test will refer to should be placed in Tests/REMORA-Gold-Files/<test_name>
. Next, edit the
Exec/CMakeLists.txt
and Tests/CTestList.cmake
files, add the problem and the corresponding tests
to the list. Note that there are different categories of tests and if your test falls outside of these
categories, a new function to add the test will need to be created. After these steps, your test will be
automatically added to the test suite database when doing the CMake configure with the testing suite enabled.