HPC-ME: HPC Portable Containers for Model Environments

What is HPC-ME

HPC Portable Container - Model Environments is a set of Dockerfiles, Singularity Definition files, and containers to provide portable model environments for scientific applications that require the same set of libraries. The ultimate goal is to have a community-based list of libraries that are needed for compiling, executing, and post-processing earth science models. We all use many of the same underlying libraries, and by working together we can agree upon a community-based approach to making container usage as standardized as possible.

List of current compilers/MPI/OS

For each container, there is a full version that contains the programming environment and a smaller runtime environment that can be used to run compiled executables. (The runtime container definition files will be added soon.) #- gcc 8/mpich/ubuntu 20.04

List of current libraries

This is the current list of most of the libraries used in the HPC-ME containers (We are trying to keep this up-to-date). The complete lit should be found in the respective YAML file.

How to build

We plan to make this step optional soon. In order to build the Docker images, you will need access to a computer with root-like access, and either docker or singularity installed. If you do not have root-like access to a suitable machine, you can still run images that were already created (e.g. on Docker hub), and we plan on hosting runnable Docker images along with the Dockerfiles in this repository soon. If you have root-like access and docker, start by choosing one of the currently supported model environments from the list above. Then build the Docker container from the Dockerfile using docker build; for example, to build the gcc8/mpich/ubuntu18 container:

docker build --file Dockerfile_gnu_ubuntu20.04 . --tag hpc-me.ubuntu.gnu

The build process takes approximately 2-3 hours, as the packages are downloaded and compiled using Spack. After a successful build, you will see that the image was built and tagged successfully:

Successfully built 90a878af77b4
Successfully tagged hpc-me.rhel8.gnu:latest

Then, you may run the container using docker or singularity on the same host. To run the image on a different machine, pushing the image to Docker Hub is recommended. Note that you will need a DockerHub account to do this (replace USER with your Docker user ID in the examples below). For example:

docker tag hpc-me.rhel8.gnu USER/hpc-me.rhel8.gnu
docker login
docker push USER/hpc-me.rhel8.gnu:latest

How to use

We plan to make improvements on this process. Also, while we plan on making Docker images available on the GitHub container registry, currently you must build the images yourself. Please start with the Build instructions to generate a Docker image with your desired OS/compiler HPC-ME environment. Then you may run the container using docker or singularity; singularity is more likely than docker to be available on HPC environments.

The usage documentation consists of some general notes on serial/parallel usage, files inside and outside the container, downloading the containers, and then specific usage scenarios:

Serial applications using docker
Serial applications using singularity
Parallel applications using singularity

Serial and parallel usage

HPC-ME containers are intended for both serial and parallel applications. Serial applications include compiling model executables, generating input grids, and post-processing model output. Earth system, climate, and weather models require parallelism to run efficiently, and use one of the Message Passage Interface (MPI) implementations OpenMPI, Intel MPI, or mpich. GCC-based HPC-ME containers use the mpich-based MPI library, which is widely available on most HPC sites, and the Intel-based containers contain both mpich and Intel MPI.

Notes on filesystems and writing files

We recommend not saving or modifying files within the environment container, and instead create and modify files on your regular filesystem. To do this, you will need to connect your filesystem to your container using bind mounts.

Downloading containers and managing images on the filesystem

Once you have pushed your images to DockerHub, you will need to download them before using. In the examples below, replace USER with your Docker Hub ID. If using docker,

docker pull USER/hpc-me.rhel8.gnu:latest

If using singularity,

singularity pull docker://USER/hpc-me.rhel8.gnu:latest

If using singularity, the image file (SIF format) is saved to the current working directory

> ls *.sif
-rwxr-xr-x 532M Dec 10 16:09 hpc-me.rhel8.gnu_latest.sif*

If using docker, the downloaded image is handled by the central docker service.

Serial applications using docker

You may activate an interactive shell within the desired HPC-ME container using docker. After running the container, the compilers and tools available within the container will be accessible in your PATH; e.g.

> docker run -it hpc-me.rhel8.gnu:latest

[root@0d2cf64e1175 /]# which nf-config
/opt/view/bin/nf-config

[root@0d2cf64e1175 /]# nf-config --version
netCDF-Fortran 4.5.3

[root@0d2cf64e1175 /]# nf-config --cflags
-I/opt/software/linux-rhel8-x86_64/gcc-8.4.1/netcdf-fortran-4.5.3-g5qfkdlp36unt2s4j4wyrc6heh2sa64n/include

Serial applications using singularity

Singularity can run Docker images and is more likely to be available on HPC environments. As with docker run, the HPC-ME tools and compilers are available in the shell, somewhat similar to loading a set of Environment Modules prepared by site administrators.

>singularity run hpc-me.rhel8.gnu_latest.sif

Singularity> which nf-config
/opt/view/bin/nf-config

Singularity> nf-config --version
netCDF-Fortran 4.5.3

Parallel applications using singularity

HPC-ME containers can provide the runtime environment for MPI applications. For instance, one could compile an MPI application using the instructions above using one of the HPC-ME development containers; and then run the application using the corresponding runtime HPC-ME container.

Please note that we are continuing to improve the usability of HPC-ME containers as well as provide more usage examples.

Usually, GFDL climate models are run on gaea by submitting a runscript to the Slurm scheduler. The runscript loads needed runtime Environment Modules, prepares input directories and files, and executes the MPI executable using srun. The HPC-ME containers provide the necessary runtime environment, obviating the need for loading Environment Modules. Currently, our approach for using the HPC-ME containers is as follows:

Create a new container, starting with the desired HPC-ME runtime container
Add the MPI-compiled executable to the container filesystem
Set the MPI-compiled executable to as the container's command (so that when the container is run the MPI executable within the container runs)
Run the singularity container SIF file using srun within the runscript, replacing the traditional MPI executable.

Replace "srun executable.x" with "srun singularity run container.SIF"
Add --mpi=pmi2 to the srun call, which connects the system MPI to the container MPI to the singularity run call
Bind the working directory so that the container has access to the input files and can write output files (singularity run -B=/path/to/workdir)

Submit the modified runscript to the scheduler

We plan to provide more examples and usage scenarios, such as using the HPC-ME containers as-is (i.e. not creating a new container as described above)

GFDL example

An example of using an HPC-ME container with the GFDL FRE workflow can be found here

Planned improvements

HPC-ME is a work in progress under active development, so please check back or follow the repository for more updates.

Build cache

We are working to create a build cache for the libraries listed so that building the containers is quick and easy.

Github container registry

We are working to add CI capability to this repository, so that the containers will be automatically built and stored in the github container registry. This will make building unnecessary for most cases, though users may build the containers themselves if they wish (e.g. for custom modifications).

Disclaimer

The United States Department of Commerce (DOC) GitHub project code is provided on an 'as is' basis and the user assumes responsibility for its use. DOC has relinquished control of the information and no longer has responsibility to protect the integrity, confidentiality, or availability of the information. Any claims against the Department of Commerce stemming from the use of its GitHub project will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government.

This project code is made available through GitHub but is managed by NOAA-GFDL at https://gitlab.gfdl.noaa.gov.

About

Languages

Language:Python 83.2%Language:Roff 11.2%Language:Dockerfile 5.0%Language:Shell 0.6%

NOAA-GFDL / HPC-ME