demux-js

Demux is a backend infrastructure pattern for sourcing blockchain events to deterministically update queryable datastores and trigger side effects. This library serves as a reference implementation of that pattern for use with Node applications.

Installation

# Using yarn
yarn add demux

# Using npm
npm install demux --save

Overview

Taking inspiration from the Flux Architecture pattern and Redux, Demux was born out of the following qualifications:

A separation of concerns between how state exists on the blockchain and how it is queried by the client front-end
Client front-end not solely responsible for determining derived, reduced, and/or accumulated state
Ability for blockchain events to trigger new transactions, as well as other side effects outside of the blockchain
The blockchain as the single source of truth for all application state

Separated Persistence Layer

Storing data in indexed state on blockchains can be useful for three reasons: decentralized consensus of computation results, usage of state from within other blockchain computations, and for retrieval of state for use in client front-ends. When building more complicated front-ends, you run into a few problems when retrieving directly from indexed blockchain state:

The query interface used to retrieve the indexed data is limited. Complex data requirements can mean you either have to make an excess number of queries and process the data on the client, or you must store additional derivative data on the blockchain itself.
Scaling your query load means creating more blockchain endpoint nodes, which can be very expensive.

Demux solves these problems by off-loading queries to any persistence layer that you want. As blockchain events happen, your chosen persistence layer is updated by updater functions, which deterministically process an array of Action objects. The persistence layer can then be queried by your front-end through a suitable API (for example, REST or GraphQL).

This means that we can separate our concerns: for data that needs decentralized consensus of computation or access from other blockchain events, we can still store the data in indexed blockchain state, without having to worry about tailoring to front-end queries. For data required by our front-end, we can pre-process and index data in a way that makes it easy for it to be queried, in a horizontally scalable persistence layer of our choice. The end result is that both systems can serve their purpose more effectively.

Side Effects

Since we have a system for acting upon specific blockchain events deterministically, we can utilize this system to manage non-deterministic events as well. These effect functions work almost exactly the same as updater functions, except they run asynchronously, are not run during replays, and modifying the deterministic datastore is off-limits. Examples include: signing and broadcasting a transaction, sending an email, and initiating a traditional fiat payment.

Single Source of Truth

There are other solutions to the above problems that involve legacy persistence layers that are their own sources of truth. By deriving all state from the blockchain, however, we gain the following benefits:

If the accumulated datastore is lost or deleted, it may be regenerated by replaying blockchain actions
As long as application code is open source, and the blockchain is public, all application state can be audited
No need to maintain multiple ways of updating state (submitting transactions is the sole way)

Data Flow

Client sends transaction to blockchain
Action Watcher invokes Action Reader to check for new blocks
Action Reader sees transaction in new block, parses actions
Action Watcher sends actions to Action Handler
Action Handler processes actions through Updaters and Effects
Actions run their corresponding Updaters, updating the state of the Datastore
Actions run their corresponding Effects, triggering external events
Client queries API for updated data

Class Implementations

Repository	Description
EOSIO / demux-js-eos *	Action Reader implementations for EOSIO blockchains
EOSIO / demux-js-postgres *	Action Handler implementation for Postgres databases
Zapata / demux-js-bitshares	Action Reader implementations for BitShares blockchain

* Officially supported by Block.one

To get your project listed, add it here and submit a PR!

Usage

This library provides the following classes:

AbstractActionReader: Abstract class used for implementing your own Action Readers
AbstractActionHandler: Abstract class used for implementing your own Action Handlers
BaseActionWatcher: Base class that implements a ready-to-use Action Watcher
ExpressActionWatcher: Exposes the API methods from the BaseActionWatcher through an Express server

In order to process actions, we need the following things:

An implementation of an AbstractActionReader
An implementation of an AbstractActionHandler
At least one HandlerVersion, which contain Updater and Effect arrays

After we have these things, we need to:

Instantiate the implemented AbstractActionReader with any needed configuration
Instantiate the implemented AbstractActionHandler, passing in the HandlerVersion and any other needed configuration
Instantiate the BaseActionWatcher (or a subclass), passing in the Action Handler and Action Watcher instances
Start indexing via the Action Watcher's watch() method (by either calling it directly or otherwise)

Example

const { BaseActionWatcher, ExpressActionWatcher } = require("demux")
const { MyActionReader } = require("./MyActionReader")
const { MyActionHandler } = require("./MyActionHandler")
const { handlerVersions } = require("./handlerVersions")
const { readerConfig, handlerConfig, pollInterval, portNumber } = require("./config")

const actionReader = new MyActionReader(readerConfig)
const actioHandler = new MyActionHandler(handlerVersions, handlerConfig)

Then, either

const watcher = new BaseActionWatcher(
  actionReader,
  actionHandler,
  pollInterval,
)

watcher.watch()

Or,

const expressWatcher = new ExpressActionWatcher(
  actionReader,
  actionHandler,
  pollInterval,
  portNumber,
)

expressWatcher.listen()

// You can then make a POST request to `/start` on your configured endpoint

API documentation

Learn from a full example

Contributing

Contributing Guide

Code of Conduct

License

MIT

Important

See LICENSE for copyright and license terms. Block.one makes its contribution on a voluntary basis as a member of the EOSIO community and is not responsible for ensuring the overall performance of the software or any related applications. We make no representation, warranty, guarantee or undertaking in respect of the software or any related documentation, whether expressed or implied, including but not limited to the warranties or merchantability, fitness for a particular purpose and noninfringement. In no event shall we be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the software or documentation or the use or other dealings in the software or documentation. Any test results or performance figures are indicative and will not reflect performance under all conditions. Any reference to any third party or third-party product, service or other resource is not an endorsement or recommendation by Block.one. We are not responsible, and disclaim any and all responsibility and liability, for your use of or reliance on any of these resources. Third-party resources may be updated, changed or terminated at any time, so the information here may be out of date or inaccurate.

vlbos / demux-js