This is a copy of Chombo version 3.2 available at https://commons.lbl.gov/display/chombo with updates and tweaks for GRChombo. Please see the Chombo readme below. The GRChombo collaboration is only a a user of Chombo. We are not affiliated to or endorsed by LBNL.
Chombo
Applied Numerical Algorithms Group
National Energy Research Scientific Computing Center
Lawrence Berkeley National Laboratory
chombo@lbl.gov
http://chombo.lbl.gov
See "Copyright.txt" in this directory.
This README file contains an extremely terse description of the Chombo software distribution. We describe
(1) What Chombo is. (2) User feedback mechanisms. (3) Where to find further documentation. (4) Software requirements for Chombo installation. (5) Installation and porting procedures. (6) Examples.
More complete documentation is contained within the distribution, either in the doc subdirectory of this directory, or in the doxygen documentation which may be generated by running "make doxygen" in the Chombo/lib directory, or online, at http://Chombo.lbl.gov.
The Chombo package provides a set of tools for implementing finite difference and finite-volume methods for the solution of partial differential equations on block-structured adaptively refined logically rectangular (i.e. Cartesian) grids. Both elliptic and time-dependent modules are included. Support for parallel platforms and standardized self-describing file formats are included.
Chombo's design is uniquely flexible and accessible. Any collaborators will be able to develop parallel applications to solve the partial differential equations in which they are interested with far shorter development times than would be possible without the infrastructure. Very careful design and documentation allows entry to the software at many levels, enabling the use of Chombo to investigate deep technical issues of adaptive mesh refinement algorithms or to simply adapt the example applications to solve scientific problems of interest.
User feedback should be directed to chombo@lbl.gov. We are happy to field all questions and comments.
Also, there is a Chombo users e-mail list at chombousers@lbl.gov, which has been an excellent resource for sharing ideas and questions with other Chombo users, many of whom are quite expert users of Chombo. (The Chombo developers do follow the chombousers mailing list as well.)
The Chombo Design documents are included in the Chombo source distribution in the Chombo/doc/ directory, and may also be found on the Chombo website (http://Chombo.lbl.gov). These documents include descriptions of both algorithms and the APIs of the Chombo package. There are also detailed installation and requirements descriptions.
If the doxygen utility is installed on your system, the Chombo html reference manual may be generated by running "make doxygen" in the Chombo/lib directory and will then installed at:
Chombo/lib/doc/doxygen/html/index.html
The HTML files are produced directly from the source code using the doxygen software, and document every class and function within the public interface of the Chombo distribution. A link to the doxygen-generated documentation may also be found online at the Chombo website.
GNU C++ (g++) 3.4 or higher -- stable releases only.
Intel C++ (icc/icpc) for Linux, Versions 10,11.
IBM C++ (xlC) version 5 and 6. Recently tested on IBM SP2 with AIX 5.2.
SGI/MIPSpro C++ (CC) version 7. Not recently tested in serial on a IRIX64 platform.
Chombo has built at various times in the past (but not recently) on compilers from: Compaq, KCC, Sun, Fujitsu.
Chombo has been compatible with all Fortran 77 compilers that we have tried. These include g77, gfortran, Intel ifc/ifort, IBM xlf*, PGI pgf77/pgf90, Sun f77, DEC/Compaq f77/90/95, Cray T3E f90, Fujitsu f90.
Chombo works with all the MPI implementations we have tried.
We strive to make the Chombo code compliant with the ISO/IEC 14882 C++ Standard (1998). Some C++ compilers are still not fully compliant and may not be able to build Chombo. In particular, building templates is known to be a problem in some compilers (e.g. pgCC).
We would like to hear about successful builds with other compiler/platforms. Please send email to chombo@lbl.gov.
Installation is described in more detail in doc/chomboDesign.pdf. Chombo uses GNU make as its build engine. Chombo can be built with several options which are specified by setting make variables on the command line if the defaults are not desirable. The resulting library names reflect the option values, so libraries with different sets of options may be built and used without conflicting with each other.
The main options are: choice of C++ and Fortran compilers, number of dimensions (1,2,3,4,5, or 6 ), floating point precision (float or double), with optimization or debugging, serial or parallel, with or without performance profiling, with HDF5 files or not, with memory tracking or not, with 32 or 64bit pointers on systems that support both.
Many packages address these options with a configure procedure. The problem is that the options then tend to be mutually exclusive for the installation. Our approach requires some additional vigilance from the installer (since configure is not going to auto-magically figure everything out for you), but has the advantage of allowing multiple versions (e.g. 2d and 3d, serial and MPI, etc) to be used simultaneously.
If you want to use the Chombo facilities for writing data files, you will need the HDF5 package installed on your machine. See:
Chombo works with versions 1.4.1 through 1.8.x. To run with HDF5 1.8 you need to compile with the 1.6 compatibility flags.
We would suggest that HDF5 be installed with the configure options "--enable-production".
If you are going to run Chombo in parallel, you will need a functioning MPI-1.2 compliant C-binding and a parallel version of HDF5 built with that MPI.
Use the "--enable-parallel" configure option for HDF5. Install the parallel libraries into a separate directory from the serial libs.
Chombo should build with any standard-compliant implementation of MPI.
The Argonne Lab version of MPI, called MPICH, is freely available from:
http://www.mpich.org
Chombo has been successfully built with MPICH and the IBM, SGI and Cray implementations of MPI.
You will need the GNU version of make to build Chombo. Any non-ancient version should work.
You can check to see if you have GNU make by running the command:
make --version
If this doesn't say it is GNU make, try
gmake --version
If this works, you should always use 'gmake' on Chombo code. Nothing else needs to be changed.
If this doesn't say it is GNU make, you need help from a local expert to find GNU make on your system or install it. It can be downloaded from: http://directory.fsf.org/wiki/Make
You will need perl to build Chombo. Any version 5.x should work. Available from:
http://www.perl.org/get.html
-
Download and untar the source. This will give you a directory named "Chombo".
-
Customizing for the local computer
Customizing the Chombo make system is done by setting make variables. You should not have to change any of the Chombo makefiles except one: "Chombo/lib/mk/Make.defs.local". The command 'make setup' will create this file from a template that contains instructions for setting the make variables that Chombo uses. These variables specify what configuration to build by default, what compiler and compiler options to use, where to find the HDF library and include files, etc.
The Chombo makefiles already know how to set the compiler options for a variety of compilers, so you may not need to customize these. The default values for the configuration variables are in the file: "Chombo/lib/mk/Make.def.defaults", and you may wish to reset these to different values in "Make.defs.local". It is likely that you will have to set the HDF variables, since they vary from system to system.
You can see some examples of setting the make variables in the file Chombo/lib/mk/local/Make.defs.ANAG, which is what we use here on our local network. This subdirectory also contains several other files for specific supercomputers that we use (eg. The NERSC computers "hopper" and "carver", the OLCF computer "jaguar", etc.). You can use these files by copying or symlinking them to "Chombo/lib/mk/Make.defs.local".
If you must customize the makefile variables differently for multiple computers that share the same copy of the Chombo files, you can do this by creating a file in "Chombo/lib/mk/local" for each different computer. The filename must be "Make.defs." where "" is the same as the output from the "uname -n" command (or the "hostname" command if uname does not work). The Chombo make system will load this file only when it is running on the computer with that name.
Every time you modify "Make.defs.local", the main makefile in the "Chombo/lib" directory will check the makefile variables to see that they have valid values. A frequent mistake is to use invalid values for the HDF* variables, especially the HDFMPI* variables. If you do not have MPI installed, you should not set the MPICXX and HDFMPI* variables.
Most of the major Chombo make variables can be set on the 'make' command line, so once you learn which variables you want to change, it is easy to build multiple configurations of the Chombo code without modifying the makefiles.
More documentation for using the Chombo make system can be found in:
- the file Chombo/lib/mk/README
- the "Requirements" and "Installation" sections of "Chombo Software Package for AMR Applications Design Document"
- Compiling the Chombo code into libraries and executables
Enter the Chombo/lib directory. Type: "make all"
This will check the variables that are set in "Make.defs.local" and complain if they are invalid. This check will be run every time "Make.defs.local" is changed. If the variables are ok, it will build the default version of the Chombo libraries and test programs.
The top level GNUMakefile has a description of the other targets.
When make completes, it will have built a set of archive libraries in Chombo/lib/. The library names are different for each version of Chombo that you can build. The pattern is:
libd..<c++compiler>.<.DEBUG><.PROF><.MPI><.64>
where: corresponds to the subdirectories of Chombo/lib/src is the value of the DIM variable, (the number of dimensions) (1,2,3,4,5,6) is the operating system name (computed automatically) <c++compiler> is the name of the C++ compiler, taken from the CXX variable. is the name of the Fortran compiler, taken from the FC variable. <.DEBUG> is present if the DEBUG variable is TRUE and absent otherwise <.PROF> is present if the PROFILE variable is TRUE and absent otherwise <.MPI> is present if the MPI variable is TRUE and absent otherwise <.64> is present if the USE_64 variable is present and absent otherwise
Note: the compiler variables (CXX and FC) may include a path and options, but only the command name is stored in the library filename.
The Chombo makefiles support parallel build using the GNU make '-j' option. The libraries and test and example programs can be built in parallel but must be built separately using the commands: make -j 6 all cd ../example ; make -j 6 example
The number of processes (6 above) should be set to whatever makes the build go the quickest. Using too many processes will slow things down. The optimal value is usually around 2-3 times the number of physical CPUs in the computer.
There are various kinds of errors that might occur in making Chombo. Read the documentation for details of how to diagnose and correct these errors.
- Running the Chombo test programs
Type: "make run"
This will try to run all the Chombo tests. Several results are possible.
- none of the programs run
- all of the programs run and complete successfully with no errors
- one or more programs do not run or abort or run with errors
If 1) happens, it might be because there is a dynamic library directory missing from the LD_LIBRARY_PATH environment variable. This is usually indicated by an error message that says something about being unable to load (or find) a file (or library) named lib.so (possibly with numbers appended).
The HDF5 library is a likely culprit. The path set in the HDFLIBFLAGS and HDFMPILIBFLAGS make variables probably should be in the LD_LIBRARY_PATH environment variable too.
The canonical output from a Chombo test program looks like:
Running broadcastTest for configuration 2d.Linux.g++.g77 ... broadcastTest: passed all tests ... broadcastTest finished with status 0
If the status value is not 0, the test failed. A failed test may print other messages. If many tests fail, there's probably something wrong with the compiler options that were used. Try changing the CXXFLAGS and FFLAGS variables in Chombo/lib/mk/Make.defs.local.
If a test fails with a segmentation violation or a stack overflow, it may mean you need to increase the stack size available to your processes. How to do this varies depending on the operating system and shell, but the command is usually something like 'limit stacksize 25600' or 'ulimit -s 25600'.
To recompile Chombo after changing compiler options, you must first type "make realclean" then "make all". Alternatively, if you don't want to destroy the existing copy of the libraries, you can use the XTRACONFIG make variable to produce a set of Chombo libraries with different filenames. For example:
make all XTRACONFIG=.O3 CXXFLAGS=-O3 FFLAGS=-O3
will compile Chombo with -O3 and name all the library files with the pattern: lib.O3.a.
If you continue to get errors while building or testing, send email to chombo@lbl.gov with the output from running the commands 'make vars' and 'make all'. If the problem happens while running the test programs, also include the output from 'make run'. Also include a copy of Chombo/lib/mk/Make.defs.local. It is also helpful to list the versions of the compilers and operating system used. (e.g., "g++ -dumpversion" and "g77 -v" and "uname -a").
If you are endeavoring to make our code run in a new environment you should know the following from Chombo:
-
Chombo neither throws nor catches any exceptions. Exception handling can safely be turned off. Chombo does makes use of STL, and most vendors have implemented their STL with exceptions, hence most compilers will complain about their own STL if you turn exceptions off.
-
Chombo is not currently MT-safe. There are only a very small number of routines that are known MT-unsafe, but the code has not been scrubbed. This work will depend on demand.
-
HDF5 currently does not supply an MT-safe version, although the next major release does make this promise.