QE-GPU: GPU-Accelerated Quantum ESPRESSO
----------------------------------------
Quantum ESPRESSO is an integrated suite of Open-Source computer codes for
electronic-structure calculations and materials modeling at the nanoscale.
The aim of QE-GPU is to create a "plugin-like" component for the standard
Quantum ESPRESSO package that allows to exploit the capabilities of NVIDIA
GPU graphics cards in order to allow materials scientists to do better and
fast science. GPU acceleration is currently available for the Plane-Wave
Self-Consistent Field (PWscf) code and the energy barriers and reaction
pathways through the Nudged Elastic Band method (NEB) package.
QE-GPU is provided "as is" and with no warranty. This software is distributed
under the GNU General Public License, please see the files LICENSE and
DISCLAIMER for details. This README presents an introduction to compiling,
installing, and using QE-GPU.
* * * * *
Note from the project maintainer
--------------------------------
I do not earn a salary for my work on QE-GPU which is pro bono and performed
during my spare time. If you have benefited from this project or you are simply
interested to leave a feedback, please consider to get in contact with me by
filling this form (it does not require a google account):
http://goo.gl/forms/xCyW5WesJe
QE-GPU 14.10.0 is the last version scheduled this year, it includes very few
changes and it has been tested extensively using CUDA 5.5. It is not guarantee
that compiles and run with more recent version of CUDA. Please read the file
RELEASE_NOTES for few additional details.
Since mid October I am working on a completely new version, build from scratch
based on strenghts and weakness of QE-GPU. This version will be based on QE
5.1.1, will not work with previous version of QE and it will be released
immediately after the next GPU Technology Conference (late March 2015)
-- Filippo
* * * * *
How to compile
--------------
1. Copy QE-GPU in espresso directory
The package includes a directory called "GPU" that has to be copied under the
Quantum ESPRESSO main directory.
Run a command like:
$ rsync -r --exclude=.git ./GPU <path-to-espresso>/
2. Apply QE-GPU patch to espresso
To apply the patch first copy the appropriate file
(according to the version of both QE and QE-GPU) in the main espresso directory
and run the command:
$ patch -p1 < name-of-the-patch.patch
Please refer to "qe-patches/README" for further information.
3. Run QE-GPU configure
3.1 SERIAL
$ cd GPU
$ ./configure --disable-parallel --enable-openmp \
--enable-cuda --with-gpu-arch=sm_35 \
--with-cuda-dir=<full-path-where-CUDA-is-installed> \
--with-magma --with-phigemm
$ cd ..
$ make -f Makefile.gpu pw-gpu
3.2 PARALLEL
$ cd GPU
$ ./configure --enable-parallel --enable-openmp --with-scalapack \
--enable-cuda --with-gpu-arch=sm_35 \
--with-cuda-dir=<full-path-where-CUDA-is-installed> \
--without-magma --with-phigemm
$ cd ..
$ make -f Makefile.gpu pw-gpu
For additional options for QE-GPU see "./configure --help"
IMPORTANT NOTE (1):
- specify always the right GPU compute capability.
- specify the correct PATH of CUDA (where CUDA is installed). Usually HPC
systems have the env variable $CUDA_HOME or $CUDA_ROOT
- "pw-gpu.x" will be placed under "bin/" (and also under "GPU/PW/")
* * * * *
How to run
----------
Nothing has changed. If the support to the GPU is enables you MUST remember to
specify few additional environmental variables required by the phiGEMM library
used by Quantum EPSRESSO.
There are 2 recognized variables, one for each *GEMM routine implemented. This
factor, a value strictly between 0 and 1, tells to the library how much of the
calculation has to be performed on the CPU and how much on the GPU. In detail,
if PHI_DGEMM_SPLIT is 0.9 it means that the 95% of the computation will be
performed by the GPU, the 5% by the CPU.
Here an example:
export PHI_DGEMM_SPLIT=0.975
export PHI_ZGEMM_SPLIT=0.975
IMPORTANT NOTE (1): suggested values are between 0.925 and 0.975
IMPORTANT NOTE (2): some system are equipped with different cards. For
example, you may have 2 Tesla Cxxxx and another card, that
you use for the video output. The video card may have not
enough CUDA capabilities for computation (below 1.3) or
different technical specs. There is a way to hide a
specific GPU card, using the environmental variable
"CUDA_VISIBLE_DEVICES".
IMPORTANT NOTE (3): the library auto-tune the split factor at runtime. You just
have to declare a initial good guess. If you are not sure
about the guess value leave the default.
* * * * *
Trouble-makers, inconsistencies, Known bugs, etc.
-------------------------------------------------
- The current version of MAGMA provided with QE-GPU does not support CUDA 6.0
- if you are using Intel MPI, please add to DFLAGS "-DMPICH_SKIP_MPICXX" to
ignore MPI C++ bindings.
- if the calculation fail due to "S non definite positive" or other errors
related to MAGMA then recompile by disabling MAGMA support.
* * * * *
In case of CUDA/GPU code errors...
----------------------------------
It may happen that some ported routines fail. If this happen you can easily
decide to run the code skipping that specific "GPU" section that may produce
some issues. To do that it is possible to compile the code with few additional
flags that exclude the GPU code related to ADDUSDENS, NEWD or VLOC_PSI.
These flags are:
-D__DISABLE_CUDA_ADDUSDENS
-D__DISABLE_CUDA_VLOCPSI
-D__DISABLE_CUDA_NEWD
Without editing the make.sys, you can run make in this way:
$ make -f Makefile.gpu MANUAL_DFLAGS="-D__DISABLE_CUDA_ADDUSDENS" pw-gpu
You can also specify more of these flags together.
* * * * *
References
----------
[1] P. Giannozzi et al., "QUANTUM ESPRESSO: a modular and open-source software
project for quantum simulations of materials", J. Phys.
Condens. Matter 21 395502 (2009)
[2] URL: http://www.quantum-espresso.org
[3] F. Spiga and I. Girotto, "phiGEMM: a CPU-GPU library for porting
Quantum ESPRESSO on hybrid systems.", 20th Euromicro International
Conference on Parallel, Distributed and Network-Based Processing
(PDP), 2012