Rainbow worker

Quick Start Instructions

In order to use drttor.jar you should ensure that you have:

A 64 bit Linux OS (tested on Ubuntu 14.04)
Oracle JDK 8

To launch the jar file use the command line java -Djava.library.path=dist -jar drttor.jar <rtorrent> (for example java -Djava.library.path=dist -jar drttor.jar ffmpeg_wannaworktogether.rbw). Downloaded/generated files are created in \tmp by default. You can however use the -o options to change the output directory.

Source and sample rbw files

usage: Client [options]

Available options: -h,--help Show this help and exit. -o,--output DIR Read/write data to directory DIR. -i,--iface IFACE Bind to interface IFACE. -s,--seed SECONDS Time to seed after downloading (default: infinitely). -d,--max-download KB/SEC Max download rate (default: unlimited). -u,--max-upload KB/SEC Max upload rate (default: unlimited).

Compiling the software

Step 1: This project requires Oracle Java 8 to be installed on the development environment. Make sure the environment variables Path and Java are set accordingly.
Step 2: Import the project into Eclipse or another IDE that supports Java.
Step 3: Edit the project’s build path to include all the libraries found in “lib” folder at the root of the repository.

What is rainbow worker?

This project enables distributed computing over a peer-to-peer network. Similarly to bittorrent, peers connects to a swarm using tracker address (centralized) or info-hash (using DHT, decentralized) inside a beencoded file. This file uses a specification close to the original Bittorrent .torrent file and contains complete definition of the computational task. Therefore we refer to these beencoded files as .rbw files.

A complete task is splitted into many pieces, which are generated by participating peers. A piece is the smallest unit of work a participant can produce and can be the result of any Unix command. Pieces can be the result of any processing, e.g. encoded video segment, rainbow table parts or more generally a partial result of a greater process. Pieces are created from unix (bash) shell commands. These commands are specified inside the .rbw file. Once all pieces are generated, pieces can (optionally) be merged using the complete command. Examples of completion merging process includes raw concatenation of pieces or application specific processing.

A task is attributed by defining a generation command key<command> which will be evaluated for each piece. While a .rbw file specify only one command for the whole content, it is parametrized so that each piece can be computed using different parameters. Thanks to the power of bash, it is possible to generate complex commands using few parameters.

How to create a task

Using Rainbow Worker, a distributed task is created by specifying beencoded dictionary parameters inside a .rbw file. A .rbw file is very similar to a .torrent file. Like a .torrent file, it contains an announce(-list) (optional using DHT) and a list of files. Unlike a torrent file, it does not contain any hash list as the content is not known in advance.

Ideally, pieces sizes range from 256kB to 4MB bytes, these limits are the recommendations for the bittorrent network. If the output file size is deterministic (all outputs have the same size and the size is known in advance), the pieces size should be equal to the output size. If the output size is nondeterministic, the specified pieces size should be superior or equal to the maximum output size.

Pieces content are evaluated using stdout bash shell commands (using /bin/bash). The command key specify the actual shell command. The $PIECE_IDX, $FILE_IDX and $PART_IDX environment variables can be used to differentiate command parameters between the files. E.g. we can create a task calculating square of natural numbers using the command echo (($PIECE_IDX*$PIECE_IDX)). Using this command, the piece content of the N^th piece will be N*N (ASCII encoded).

Current Status

As of version 0.9:

Rainbow Worker is currently a proof of concept project. It allows in the current version to distribute over a decentralized P2P network large tasks. It consists in two parts: (1) a bittorrent-like client and (2) beencoded .rbw files. Sample beencoded files are provided to compute either rainbow tables or video transcoding. The included examples can be modified with a beencode editor (e.g: beencode editor)[https://sites.google.com/site/ultimasites/bencode-editor#download].

The Rainbow Worker itself is a bittorrent-like client coded with Java 8. It requires no installation (can be copied) and features a command-line interface to review tasks progress. The client supports both trackers and DHT operations.

License

This Rainbow Worker software is distributed under the terms of the Apache Software License version 2.0. See COPYING file for more details.

2014-10-11

Authors

Arnaud Durand<arnaud.durand@unifr.ch> Rainbow Worker developer.
Mikael Gasparian<mikael.gasparian@unifr.ch> Rainbow Worker developer.
Thomas Rouvinez<thomas.rouvinez@unifr.ch> Rainbow Worker developer.

Contributors

Maxime Petazzoni <maxime.petazzoni@bulix.org> (Software Engineer at SignalFuse, Inc)
Original author, main developer and maintainer of Ttorrent Java Library.
David Giffin <david@etsy.com>
Multi-file torrent support in the Ttorrent Java Library.

DurandA / bitworker