alex-dixon / clj-3df

Clojure(Script) client for Declarative Differential Dataflows

clj-3df

This is currently a research project, not feature-complete in any way, and overall not ready for use in production services.

3DF is best thought of as a pub/sub system in which subscriptions can be arbitrary Datalog expressions. Subscribers register queries with the broker, and data sources (such as Kafka, Datomic, or any other source-of-truth) publish new data to it. All subscriber queries affected by incoming data will be notified with a diff, describing how their results have changed. The Datalog implementation is modeled after Datomic's query language and aims to support the same set of features.

3DF does this efficiently, thanks to being built on top of differential dataflows. In particular, for any given transaction asserting / retracting n facts, 3DF will at worst do work in O(n * # of registered queries). Often it can be even less than that, because dataflows can be shared between similar queries.

This repository contains the Clojure client for 3DF. The broker is written in Rust and can be found in the Declarative Differential Dataflows repository.

How it works

The 3DF client compiles Datalog expressions into an intermediate representation that can be synthesised into a differential dataflow on the server. This dataflow is then registered and executed across any number of workers. Whenever query results change due to new data entering the system, the server will push the neccessary changes via a WebSocket connection.

For example, consider a subscriber created the following subscription:

(register-query! conn db "user inbox" 
  '[:find ?msg ?content
    :where 
	[?msg :msg/recipient "[e-mail]"]
	[?msg :msg/content ?content]])

and a new message arrives in the system.

[{:msg/receipient "[e-mail]"
  :msg/content    "Hello!"}]

Then the server will push the following results to the subscriber:

[[[<msg-id> "Hello!"] +1]]

If at some later point in time, this message was retracted

[[:db/retractEntity <msg-id>]]

the server would again notify the subscriber, this time indicating the retraction:

[[[<msg-id> "Hello!"] -1]]

This guarantees, that subscribers maintaining any form of functionally derived information will always have a consistent view of the data.

Query Language Features

• Basic datalog expressions (e.g. implicit joins)
• not, not-join, or, or-join clauses
• Parameterized queries
• Rules, self-referential / mutually recursive rules
• Aggregates
• As-of queries
• More find specifications (e.g. collection, scalar)
• Pull queries

Non-Features

3DF is neither concerned with durability nor with consistency in the ACID sense. It is intended to be used in combination with a consistent, durable source-of-truth such as Datomic or Kafka.

Consequently, 3DF will accept whatever tuples it is supplied with. For example, whereas in Datomic two subsequent transactions on an empty database

(d/transact conn [[:db/add 123 :user/balance 1000]])
...
(d/transact conn [[:db/add 123 :user/balance 1500]])

would result in the following sets of datoms being added into the database:

[[123 :user/balance 1000 <tx1> true]]
...
[[123 :user/balance 1000 <tx2> false]
 [123 :user/balance 1500 <tx2> true]]

3DF will by itself not take any previous information into account on transactions. Again, 3DF is intended to be fed data from a system like Datomic, which would ensure that transactions produce consistent tuples.

License

Copyright © 2018 Nikolas Göbel

Licensed under Eclipse Public License (see LICENSE).