Hands-on examples for the PRACE Training Course "Scalable Parallel Computations with PETSc" held at CSC - IT Center for Science, Espoo, Finland on 2-3 May 2019
Welcome to the tutorial!
You can find tutorial information and slides at https://events.prace-ri.eu/event/871/.
During the tutorial, please open the latest version of slides: https://bit.ly/2VCmLih.
Login to Taito
ssh trainingXXX@taito.csc.fi
Start interactive session at Taito
Before running any example or compiling PETSc, start an interactive session on a compute node:
salloc -n 4 -t01:00:00 --reservation=training
and at the end of each hands-on session
exit
Some GIT basics
First of all, load newer GIT (since the default one causes troubles):
module add git
(I suggest adding this line to the end of your ~/.bashrc
file.)
To make a local copy of this repository:
# make sure we are at $HOME
cd
# clone into new petsc-tut-cscs-2019 directory
git clone https://github.com/haplav/petsc-tut-cscs-2019.git
To update your local copy:
cd ~/petsc-tut-cscs-2019 # path to local clone (contains .git subdirectory)
git pull
If git pull
fails because of local changes, stash them first:
git stash
git pull
git stash pop
Sometimes you will have to resolve merge conflicts (let me know if you struggle with that). Then try to compile the example again.
Enable PETSc
Option 1: module
module spider petsc
module spider petsc/3.10.0
module load intel/18.0.1 intelmpi/18.0.1 mkl/18.0.1 hdf5-par/1.10.2 petsc/3.10.0
Option 2 (recommended): use my PETSc installation
source /homeappl/home/training026/appl_taito/petsc/env-arch-taito-impi18-icc-mkl-dbg
# one of
# env-arch-taito-impi18-icc-mkl-dbg
# env-arch-taito-impi19-gcc-mkl-dbg
# env-arch-taito-mpich3-gcc-fblaslapack-dbg
# env-arch-taito-openmpi3-gcc-mkl-dbg
Look what these scripts do.
PETSc was configured differently (mainly with different MPI implementation) in each case using the corresponding arch-$PETSC_ARCH.py
scripts;
we will talk about them in detail later.
I have added all those files also into this repository.
Test it!
cd ~/petsc-tut-cscs-2019/solutions
git pull
make ex1 && srun -n 4 ./ex1
Tips & tricks
Filter the error output - only from rank 0
srun -n 4 ./ex3 2>&1 | grep '^\[0\]'
VecScatter
Convert a distributed vector to a sequential one using Vec vin; /* distributed vector */
Vec vout; /* sequential vector (to be created) */
VecScatter ctx;
...
ierr = VecScatterCreateToZero(vin,&ctx,&vout);CHKERRQ(ierr);
ierr = VecScatterBegin(ctx,vin,vout,INSERT_VALUES,SCATTER_FORWARD);CHKERRQ(ierr);
ierr = VecScatterEnd(ctx,vin,vout,INSERT_VALUES,SCATTER_FORWARD);CHKERRQ(ierr);
ierr = VecScatterDestroy(&ctx);CHKERRQ(ierr);
...
ierr = VecDestroy(&vout);CHKERRQ(ierr);