.. _Testing:

Testing and Verification
------------------------

Testing and verification of REMORA can be performed using CTest, which is included in the CMake build system. If one builds REMORA with CMake, the testing suite, and the verification suite, can be enabled during the CMake configure step.

An example ``cmake`` configure/build command performed in the ``Build`` directory in REMORA is shown below with options relevant to the testing suite:

::

  cmake -DCMAKE_INSTALL_PREFIX:PATH=./install \
        -DCMAKE_BUILD_TYPE:STRING=Release \
        -DREMORA_ENABLE_MPI:BOOL=ON \
        -DCMAKE_CXX_COMPILER:STRING=mpicxx \
        -DREMORA_ENABLE_FCOMPARE:BOOL=ON \
        -DREMORA_ENABLE_TESTS:BOOL=ON \
        -DREMORA_USE_CPP:BOOL=ON \
        .. && make

To run CTest on GPU, add one of the options: ``REMORA_ENABLE_CUDA``, ``REMORA_ENABLE_HIP``, or ``REMORA_ENABLE_SYCL``, depending on the type of GPU on your system. For example:

::

  cmake -DCMAKE_INSTALL_PREFIX:PATH=./install \
        -DCMAKE_BUILD_TYPE:STRING=Release \
        -DREMORA_ENABLE_MPI:BOOL=ON \
        -DREMORA_ENABLE_CUDA:BOOL=ON \
        -DCMAKE_CXX_COMPILER:STRING=mpicxx \
        -DREMORA_ENABLE_FCOMPARE:BOOL=ON \
        -DREMORA_ENABLE_TESTS:BOOL=ON \
        -DREMORA_USE_CPP:BOOL=ON \
        .. && make

While performing a ``cmake -LAH ..`` command will give descriptions of every option for the CMake project. Descriptions of particular options regarding the testing suite are listed below:

**REMORA_ENABLE_FCOMPARE** -- builds the ``fcompare`` utility from AMReX as well as the executable(s), to allow for testing differences between plot files

**REMORA_ENABLE_TESTS** -- enables the base level regression test suite that will check whether each test will run its executable to completion successfully


Building the Tests
~~~~~~~~~~~~~~~~~~

Once the user has performed the CMake configure step, the ``make`` command will build
every executable required for each test.
In this step, it is highly beneficial for the user to use the ``-j`` option for ``make``
to build source files in parallel. For example:

   .. code:: shell

        make -j8

Running the Tests
~~~~~~~~~~~~~~~~~

Once the test executables are built, CTest also creates working directories for each test within the ``Build`` directory
where plot files will be output, etc. This directory is analogous to the source location of the tests in ``Tests/test_files``.

To run the test suite, run ``ctest`` in the ``Build`` directory. CTest will run the tests and report their exit status.
Useful options for CTest are ``-VV`` which runs in a verbose mode where the output of each test can be seen. ``-R``
where a regex string can be used to run specific sets of tests. ``-j`` where CTest will bin pack and run tests in
parallel based on how many processes each test is specified to use and fit them into the amount of cores available
on the machine.
Output for the last set of tests run is available in the ``Build`` directory in ``Testing/Temporary/LastTest.log``.

Adding Tests
~~~~~~~~~~~~

Developers are encouraged to add tests to REMORA and in this section we describe how the tests are organized in the
CTest framework. The locations (relative to the REMORA code base) of the tests are in ``Tests``. To add a test, first
create a problem directory with a name in ``Exec/<prob_name>``. This problem directory is meant for a production
run where the simulation is run until convergence or a solution is developed. This problem setup could comprise
of a more complex physics than the corresponding tests for regression at ``Tests/test_files/<test_name>``. Prepare
toned down versions of the input file(s) for each combination of physics that a regression test is desired.
For example, ``Upwelling`` problem with input file ``Exec/Upwelling/inputs`` solves the double gyre problem. The corresponding regression test is driven by the input files
``Tests/test_files/Upwelling/Upwelling.i``.

Any file in the test directory will be copied during CMake configure to the test's working directory.
The input files meant for regression test run only until a few time steps. The reference solution that the
regression test will refer to should be placed in ``Tests/REMORA_Gold_Files/<test_name>``. Next, edit the
``Exec/CMakeLists.txt`` and ``Tests/CTestList.cmake`` files, add the problem and the corresponding tests
to the list. Note that there are different categories of tests and if your test falls outside of these
categories, a new function to add the test will need to be created. After these steps, your test will be
automatically added to the test suite database when doing the CMake configure with the testing suite enabled.