Zylatis / daliuge

The DALiuGE Execution Engine

Data Activated 流 Graph Engine

DALiuGE is a workflow graph execution framework, specifically designed to support very large scale processing graphs for the reduction of interferometric radio astronomy data sets. DALiuGE has already been used for processing large astronomical datasets in existing radio astronomy projects. It originated from a prototyping activity as part of the SDP Consortium called Data Flow Management System (DFMS). DFMS aimed to prototype the execution framework of the proposed SDP architecture.

Development and maintenance of DALiuGE is currently hosted at ICRAR and is performed by the DIA team.

See the docs/ directory for more information, or visit our online documentation

Installation

docker

The easiest way to get started is to use the docker container installation procedures provided to build and run the daliuge-runtime and the daliuge-translator. Please refer to the README files in the subdirectories daliuge-runtime and daliuge-translator, respectively. Depending on what you want to deploy you may need to build and run only the runtime or the translator.

PyPi

It is also possible to install the latest stable version of the full package using PyPi:

pip install daliuge

If you only want the translator engine and don't need the runtime, or vice-versa, you can install them separately:

pip install daliuge-translator
pip install daliuge-runtime

Or if you plan to develop DALiuGE:

git clone https://github.com/ICRAR/daliuge
cd daliuge
pip install -e daliuge-common
pip install -e daliuge-translator # optional
pip install -e daliuge-runtime    # optional

Porting from DALiuGE 0.X

With the release of DALiuGE 1.0.0 the code has been broken down into separate packages to accommodate leaner and easier installations when only a subset of the functionality is required. In doing so we tried to maintain as much backward compatibility as possible, but there are some minor exceptions:

• Code doing from dlg import delayed or similar must be changed to from dlg.runtime import delayed.
• Scripts finding the include directory path for daliuge headers using code like python -c 'import dlg; print(dlg.get_include_dir()) should switch to invoke dlg include_dir instead.

[1]	流 (pronounced Liu) is the Chinese character for "flow".