- A compiled version of CPL library (https://github.com/Crompulence/cpl-library) which has been added to the user's path using "source SOURCEME.sh"
- A copy of OpenFOAM
- The same GCC and MPICH version used to build CPL library
The following environment variables:
FOAM_CPL_SOCKET_LIBBIN = $FOAM_INST_DIR/cpl-socket/lib
LD_LIBRARY_PATH = $LD_LIBRARY_PATH:$FOAM_CPL_LIBBIN
must be defined in order for a) the compilation to work and b) the library to be found by the ld linker. They are conveniently defined in the config file SOURCEME, which can be used as follows:
$ source SOURCEME.sh
The install process assumes that you have OpenFOAM fully installed with all source code. The process then simply builds a top level solver, using all of the OpenFOAM shared libraries and CPL library's shared library, which is designed for coupled simulation.
First, change directory to CPL_APP_OPENFOAM-DEV,
cd CPL_APP_OPENFOAM-DEV
and create a file called CODE_INST_DIR which will tell the APP which OPENFOAM to install against,
echo "/path/to/openfoam/directory/" > CODE_INST_DIR
Note that the folder in this directory is expected to be called OpenFOAM-3.0.1. This is the currently supported OpenFOAM version.
Now, to build the various solvers, it may be as simple as,
$ make
N.B.: warnings from the included MPI headers may be ignored. This makes a range of OpenFOAM solvers, including
- CPLIcoFOAM - A basic fluid solver for MD-CFD coupling which partially overlaps an MD simulation
- CPLSediFOAM - SediFOAM (https://github.com/xiaoh/sediFoam) adapted to use the CPL library philosophy of minimal linked library
- CPLCFDDEMFOAM - CFDDEM (https://www.cfdem.com/) adapted to use the CPL library philosophy of minimal linked library
CPL_APP_OpenFOAM is released under the GNU GPL v3 license. Details are found in the file LICENSE that is included with the release.
This application repository is structured as follows:
-
src - source files which include the following
- CPLPstream: OpenFOAM's parallel-computing communications supports a number of different paradigms, including MPI. In order to support all of them, it wraps all communication functions in a library called "Pstream". CPL Library is based on MPI, but requires the MPI_COMM_WORLD communicator to be split into two "realm" communicators - one for MD, and the other for CFD. All subsequent operations that would have been on MPI_COMM_WORLD in the MD and CFD coupled programs must now be called on the realm communicator CPL_REALM_COMM. CPLPstream replicates OpenFOAM's MPI Pstream, where CPL_REALM_COMM has been substituted in place of MPI_COMM_WORLD.
- CPLSocket: The interface to CPL Library is implemented in a single class named CPLSocket. This directory holds its source code.
- solvers: OpenFOAM is implemented as a set of libraries, and it is up to the user to develop their own main-level applications that use these libraries. This directory contains the source code for a coupled incompressible solver CPLIcoFoam that is based on OpenFOAM's icoFoam.
-
examples - some input examples for various cases
-
test - a range of test cases run automatically on Travis CI
-
config - scripts to specify version of MPI to build OpenFOAM with and patches for scotch.
New folders created by building process
- lib - dynamic-link library binaries are created in a new folder
./lib/. - bin - Executable solver applications are created in the a folder
./bin.
Why do you need to do this? If you want to run two codes with the colon MPMD syntax,
mpiexec -n 4 CPLIcoFoam : -n 16 ./MD
where they have a shared MPI_COMM_WORLD, which we will call the "shared" paradigm of coupling (as opposed to the distinct paradigm where both codes are started individually and join together using the not always functional MPI_Open_port and MPI_Comm_accept style linking to create an intercommunicator between the MPI_COMM_WORLD intracommunicators of both codes). The sharing of MPI_COMM_WORLD means that any use of MPI_COMM_WORLD in any MPI communications will now cause errors or deadlock in the coupled code, so these have to be replaced with a local comm. Patching Pstream, the location where all MPI communication is contained, is the easiest way to do that for OpenFOAM. The steps are as follows (given in general terms to account for future OpenFOAM changes but specifically done for v2112 in the current patch.
The Pstream which is used can be replaced for all codes using LD_LIBRARY_PATH, it goes from version in
/home/USERNAME/codes/CFD/OpenFOAM/openfoam-OpenFOAM-v2112.220610/platforms/linux64GccDPInt32Opt/lib/
to
CPL_APP_OPENFOAM/lib/libPstream.so
The first step in patching requires copying all of the Pstream/mpi file from the OpenFOAM code diretory. e.g. in my case
/home/USERNAME/codes/CFD/openfoam-OpenFOAM-v2112.220610/src/Pstream/mpi/
to a folder called CPLPstream in the CPL_APP_OPENFOAM/src directory, replacing whatever files are there.
You can try building this with wmake, which should add PStream to CPL_APP_OPENFOAM/lib and
by changing LD_LIBRARY_PATH you should see included CPL_APP_OPENFOAM/lib/Pstream.so
replacing the default Pstream.so on your OpenFOAM executables, e.g.
ldd /home/USERNAME/codes/CFD/OpenFOAM/openfoam-OpenFOAM-v2112.220610/platforms/linux64GccDPInt32Opt/bin/icoFoam
Now, you can edit this Pstream as needed to make sure the shared CPL paradigm works. Then adding the following line
extern MPI_Comm CPLRealmComm;
in the file PstreamGlobals.H and setting it to default to MPI_COMM_WORLD for default case which is nothing to do with coupling.
MPI_Comm Foam::PstreamGlobals::CPLRealmComm = MPI_COMM_WORLD;
in PstreamGlobals.C. At this stage, you can rebuild Pstream again and functionality should be identical (we have just add a new variable so far).
Next, find and replace every instance of MPI_COMM_WORLD in UPStream.C with Foam::PstreamGlobals::CPLRealmComm. You can rebuild Pstream and test again as nothing has changed, as the new Foam::PstreamGlobals::CPLRealmComm is still MPI_COMM_WORLD.
Finally, in order to allow a "shared" MPI run, we need to define the Foam::PstreamGlobals::CPLRealmComm to be the value returned by CPL.init. In an example script this looks like:
#include "PstreamGlobals.H"
#include "mpi.h"
#include "cpl.h"
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
int main(int argc, char *argv[])
{
//Define variables
int CFD_realm = 1;
MPI_Comm CFD_COMM;
int flag = 0;
int ierr = MPI_Initialized(&flag);
if (flag == 0)
MPI_Init(&argc, &argv);
//Initialise CPL library
CPL::init(CFD_realm, CFD_COMM);
Foam::PstreamGlobals::CPLRealmComm = CFD_COMM;
A few notes, the PstreamGlobals.h must be included so the variable introduced above can be set (and replace MPI_COMM_WORLD). Also, this should be called as early as possible in the main function of an OpenFOAM solver. The function which starts MPI Foam::"UPstream::init" and uses Foam::PstreamGlobals::CPLRealmComm is called by one of the include statements #include "setRootCase.H", #include "createTime.H" or #include "createMesh.H"