Dchan is a server that exposes channels for inter-process communications (IPC) over a file tree interface. These channels are much like Go channels and can be used in the same way but between processes. Instead of implementing a new protocol for data exchange or use a old one like AMQP, Dchan uses a simple file interface. There’s no need of client libraries for each language (every language knows how to read and write from files).
Dchan is able to share those files in the network with the help of the 9P protocol, but it is transparent for clients.
This project uses the concept of Literate Programming of Donald Knuth.
Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do. \cite{Knuth:1984:LiterateProgramming}
This book is the only source for design ideas, code, documentation, tests and project management. From it we build everything.
You can download the book here or read the html version here.
Dchan was created as an alternative technology for data processing pipelines. The classic approach to data processing is to write a program that reads in data, transforms it in some desired way, and outputs new data. Such programs, when chained, forms an architecture called “data pipeline”. An obvious example is UNIX pipes, that works very well when all programs share the same address space. For distributed architectures, a more elaborate solution must be used to perform message passing between different machines.
After some time using RabbitMQ, and then studying other solutions, we discovered that majority of current technologies aren’t good enough to solve our specific use cases.
The problem is that, when the worker queue grow indefinitely, because of slow consumers throughput compared to publishers or because some consumers crashed, two things can happen:
- Memory starvation (in case of no persistence)
- Abruptly performance degradation
In a high volume data processing, the difference of throughput between publishers and consumers can lead to lots of terabytes waiting to be processed if not properly handled.
Dchan is an alternative to current brokered solutions but providing ad-hoc synchronization. Dchan helps micro-services to communicate transparently. It’s much like a rendezvous point in your network, where different services met in order to exchange data directly (no need to store messages in-fly). Dchan uses the CSP (Communicating Sequential Processes) style of concurrency to create channels of data, enabling direct inter-process communication without requiring service discovery.
To build dchan you’ll need the following dependencies installed:
Tests dependencies (optional):
- plan9port
- Bash >= 4
Book dependencies (optional):
- Latex
- Ditaa
On archlinux the following command installs almost everything needed:
sudo pacman -Syy emacs24 make go \ # software dependencies
plan9port bash \ # test dependencies
texlive-most ditaa # book dependencies
To install orgmk jump to Orgmk section.
After that, if you want execute dchan tests, you need configure your PATH and GOPATH environment variables.
PLAN9=<path-to-plan9port> # commonly /opt/plan9
export PATH=$PATH:$PLAN9 $PLAN9For GOPATH, read the Go Getting Started.
We don’t use the default orgmk elisp, but a modified version configured for our project. To build orgmk, then use the commands below:
cd <some-place>
git clone https://github.com/fniessen/orgmk.git
cd orgmk
ORGMK_EL=<path-to-dchan>/scripts/orgmk.el make -e
sudo make installThe commands above will install orgmk on /usr/local/bin but using the emacs init elisp from dchan directory. If you use orgmk for other projects then you should install orgmk on another place.
Executing make will build the software, execute tests and build the book (pdf, html and txt). But you can run make build or make test independently.