longcongduoi/RaftLib

RaftLib is a C++ Library for enabling stream/data-flow parallel computation. Using simple right shift operators (just like the C++ streams that you would use for string manipulation), you can link parallel compute kernels together. With RaftLib, we do away with explicit use of pthreads, std::thread, OpenMP, or any other parallel "threading" library. These are often mis-used, creating non-deterministic behavior. RaftLib's model allows lock-free FIFO-like access to the communications channels connecting each compute kernel. The full system has many auto-parallelization, optimization, and convenience features that enable relatively simple authoring of performant applications. This project is currently in the alpha stage (recently emerging from a PhD thesis). The beta release will bring back multi-node support, along with (planned) container support for the remote machines. Feel free to give it a shot, if you have any issues, also feel free to send the authors an e-mail.

User Group / Mailing List: Google Group

Direct message to authors: mail

=============

Build status

CI Test environment:

Linux - kernel v. 3.13, Ubuntu 5.2.1-23ubuntu1~12.04, (gcc-5.2.1/Clang 3.7.1), CMake 3.5

Offline testing:

OS X - El Capitan, Apple LLVM version 7.0.2, CMake 3.6
Linux - kernel v. 4.4, gcc-6.1/clang 3.8

Pre-requisites

OS X & Linux

Compiler: c++14 capable -> Clang, GNU GCC 5.0+, or Intel icc Libraries:

Boost, if not installed, needed headers automatically downloaded with cmake

Windows

Latest merge from pull request to main should enable compilation on VS on Win10.

Install

Make a build directory (for the instructions below, we'll write [build]). If you want to build the OpenCV example, then you'll need to add to your cmake invocation:

-DBUILD_WOPENCV=true

To use the QThreads User space HPC threading library you will need to add the following (NOTE: The qthread library currently uses its own partitioner and does not work with Scotch, it also has issues with OpenCV, will fix in next release iteration):

-DUSEQTHREAD=1

To build:

mkdir [build]
cd [build]
cmake ..
make && make test
sudo make install

NOTE: The default prefix in the makefile is:

PREFIX ?= /usr/local

Feel free to substitute your favorite build tool. I use Ninja and make depending on which machine I'm on. To change out, use cmake to generate the appropriate build files with the -Gxxx flag.

Citation

If you use this framework for something that gets published, please cite it as:

@article{blc16,
  author = {Beard, Jonathan C and Li, Peng and Chamberlain, Roger D},
  title = {RaftLib: A C++ Template Library for High Performance Stream Parallel Processing},
  year = {2016},
  doi = {http://dx.doi.org/10.1177/1094342016672542},
  eprint = {http://hpc.sagepub.com/content/early/2016/10/18/1094342016672542.full.pdf+html},
  journal = {International Journal of High Performance Computing Applications}
}

Other Info Sources

Project web page
Project wiki page
Blog post intro
Jonathan Beard's thesis
Views on parallel computing, general philosphy
Feel free to e-mail one of the authors of the repo

Random Notes

The old Makefile had an uninstall script, I need to add an object to the cmake file so that we can have similar functionality. Theres also a bit of cleanup to do as I transition fully from Make to CMake.

A lot of the auto-optimization stuff has been pulled out temporarily while I'm working on cross-platform compatibility. A lot of the low level API calls are well, low level and Linux/Unix/OS X specific so I'm working on building in Windows versions of those specific calls. Should be done in a few weeks. ( 5 Jan 2016, jcb )

Added in prefetch for object types larger than a single cache line. Profiling shows that it's really not beneficial on most platforms with good prefetch algorithms. (25 March 2016, jcb )

longcongduoi / RaftLib