Performance benchmarks and regression tests for the ExCALIBUR project.
These benchmarks are based on a similar project by StackHPC.
Note: at the moment the ExCALIBUR benchmarks are a work-in-progress.
It is recommended to install the excalibur-tests package with pip by
pip install .On most systems, it is recommended to install the package in a virtual environment.
For example, using the python3 built-in virtual environment tool venv,
create an environment called my_environment with
python3 -m venv ./my_environmentand activate it with
source ./my_environment/bin/activateFor development,
pass the -e/--editable flag to pip to link the installed package to the files in the local
directory, instead of copying, to be able to make changes to the installed package.
The pip install will install a compatible version of ReFrame from PyPi. However, you will have to manually provide an installation of Spack.
Spack is a package manager specifically designed for HPC facilities. In some HPC facilities there may be already a central Spack installation available. However, the version installed is most likely too old to support all the features used by this package. Therefore we recommend you install the latest version locally, following the instructions below.
Note: if you have already installed spack locally and you want to upgrade to
a newer version, you might first have to clear the cache to avoid conflicts:
spack clean -m
Follow the official instructions to install the latest version of Spack (summarised here for convenience, but not guaranteed to be up-to-date):
- git clone spack:
git clone -c feature.manyFiles=true https://github.com/spack/spack.git - run spack setup script:
source ./spack/share/spack/setup-env.sh - check spack is in
$PATH, for examplespack --version
In order to use Spack in ReFrame, the framework we use to run the benchmarks
(see below), the directory where the spack program is installed needs to be in
the PATH environment variable. This is taken care of by the setup-env.sh
script as above, and you can have your shell init script (e.g. .bashrc)
do that automatically in every session, by adding the following lines to it:
export SPACK_ROOT="/path/to/spack"
if [ -f "${SPACK_ROOT}/share/spack/setup-env.sh" ]; then
source "${SPACK_ROOT}/share/spack/setup-env.sh"
fireplacing /path/to/spack with the actual path to your Spack installation.
ReFrame also requires a Spack
Environment. We
provide Spack environments for some of the systems that are part of the
ExCALIBUR and DiRAC projects in
https://github.com/ukri-excalibur/excalibur-tests/tree/main/benchmarks/spack/.
If you want to use a different Spack environment,
set the environment variable EXCALIBUR_SPACK_ENV to the path of the directory
where the environment is. If this is not set, ReFrame will try to use the
environment for the current system if known, otherwise it will automatically
create a very basic environment (see "Usage on unsupported systems" section below).
ReFrame is a high-level framework for writing regression tests for HPC systems. For our tests we require ReFrame v4.1.3.
If you need to manually install ReFrame, follow the official instructions to install this package. Note that ReFrame requires Python 3.6: in your HPC system you may need to load a specific module to have this version of Python available.
We provide a ReFrame configuration file with the settings of some systems that
are part of the ExCALIBUR or DiRAC projects. You can point ReFrame to this file by
setting the
RFM_CONFIG_FILES
environment variable:
export RFM_CONFIG_FILES="${PWD}/benchmarks/reframe_config.py"If you want to use a different ReFrame configuration file, for example because you use a different system, you can set this environment variable to the path of that file.
Note: in order to use the Spack build system in ReFrame, the spack
executable must be in the PATH also on the compute nodes of a cluster, if
you want to run your benchmarks on them. This is taken care of by adding it
to your init file (see spack section above).
However, you will also need to set the
RFM_USE_LOGIN_SHELL
environment variable (export RFM_USE_LOGIN_SHELL="true") in order to make ReFrame use
!#/bin/bash -las shebang line, which would load the user's init script.
Once you have set up Spack and ReFrame, you can execute a benchmark with
reframe -c benchmarks/apps/BENCH_NAME -r --performance-reportwhere benchmarks/apps/BENCH_NAME is the directory where the benchmark is. The command
above assumes you have the program reframe in your PATH. If you have followed the instructions
to install using pip into the default directory, it should have been automatically added.
If it is not the case, call reframe with its relative or absolute path.
For example, to run the Sombrero benchmark in the benchmarks/apps/sombrero directory you can
use
reframe -c benchmarks/apps/sombrero -r --performance-reportFor benchmarks that use the Spack build system, the tests define a default Spack specification
to be installed in the environment, but users can change it when invoking ReFrame on the
command line with the
-S option to set
the spack_spec variable:
reframe -c benchmarks/apps/sombrero -r --performance-report -S spack_spec='sombrero@2021-08-16%intel'
All the built-in fields of ReFrame regression classes can be set on a per-job basis using the
-S command-line option. One useful such field is
env_vars,
which controls the environment variables used in a job.
The syntax to set dictionary items, like for env_vars, is a comma-separated list of key:value pairs: -S dict=key_1:value_1,key_2:value_2.
For example
reframe -c benchmarks/apps/sombrero -r --performance-report -S env_vars=OMP_PLACES:threads
runs the benchmarks/apps/sombrero benchmark setting the environment variable OMP_PLACES
to threads.
The provided ReFrame configuration file contains the settings for multiple systems. If you
use it, the automatic detection of the system may fail, as some systems may use clashing
hostnames. To avoid this, you can use the flag --system NAME:PARTITION
to specify the system (and optionally the partition) to use.
Additionally, if submitting jobs to the compute nodes requires additional options, like for
example the resource group you belong to (for example --account=... for Slurm), you have
to pass the command line flag
--job-option=...
to reframe (e.g., --job-option='--account=...').
The configuration provided in reframe_config.py lets you run the
benchmarks on pre-configured HPC systems. However you
can use this framework on any system by choosing the "generic" system with --system generic, or by using your own ReFrame configuration. You can use the "generic" system to run
benchmarks in ReFrame without using a queue manager or an MPI launcher (e.g. on a personal workstation).
If you choose the "generic" system and a benchmark using the Spack build system,
a new empty Spack environment will be automatically created in
spack-environments/generic when ReFrame is launched for the first time.
You should populate the environment with the packages already installed on your system
before running Spack to avoid excessively rebuilding system packages. See the
Spack configuration section of CONTRIBUTING.md for instructions on how
to set up a Spack environment.
In particular, make sure that at least a compiler and an MPI library are added into the environment.
After the Spack environment is set up, tell ReFrame to use it by setting the environment
variable EXCALIBUR_SPACK_ENV, as described above.
While the aim is to automate as much system-specific configuration as possible, there are some options that have to be provided by the user, such as accounting details, and unfortunately the syntax can vary.
The file SYSTEMS.md contains information about the use of this framework on specific systems.
Feel free to add new benchmark apps or support new systems that are part of the
ExCALIBUR benchmarking collaboration. Read CONTRIBUTING.md for more details.