codax
Codax is an idiomatic transactional embedded database for clojure. A codax database functions as a single (potentially enormous) clojure map, with a similar access & modification api.
Version 1.3.1 is no longer AOT compiled
Version 1.3.0 improves compaction to reduce disk space usage. It also enables custom path type definitions. See CHANGELOG for more details.
The Why
Even simple programs frequently benefit from saving data to disk. Unfortunately, there is generally a large semantic leap once you go from values in memory to values on disk. Codax aims to close that semantic gap. While it won't win any speed contests, it is designed to be performant enough for applications serving thousands of users. Most importantly, it is designed to make data persistance as low friction as possible. It is also designed to be effortless to get started with. There are no external libraries to install, and the underlying B+ tree is written directly in clojure.
Fundamentally, I wrote this library for myself in an ongoing effort to make my own life simpler and my own programming more fun. I wanted to share it with the community in the hopes that others may find it does the same for them.
ACID Compliance
Codax provides the following guarantees:
- Atomic - Every transaction is completed fully, or not at all
- Consistent - The database always represents a valid clojure map
- Isolated - No reader or writer will ever see data from an incomplete transaction
- Durable - All writes are synced to disk before returning
Production Ready?
I have successfully used this library in production environments. That said, there are probably a few rough edges that could use smoothing.
Usage
Basic API
Database Functions
open-database!
- Opens or creates a database, or returns an existing database connection if it's already openclose-database!
- Safely closes an open databaseclose-all-databases!
- Safely closes all open databasesis-open?
- Checks if a database is opendestroy-database!
- Deletes a database and all its data irretrievably (intended for use in tests).
Transaction Macros
These take a database argument and a transaction-symbol and bind the symbol to a newly created transaction. Transactions are isolated from each other. Read-transactions evaluate to the value of their body, but a (successful) write-transaction evaluates to nil.
with-read-transaction
- creates a read transactionwith-write-transaction
- creates a write transaction (body must evaluate to a transaction or an exception will be thrown)
In-Transaction Functions
These are all similar to the clojure.core map *-in
(e.g. assoc-in
) with the following exceptions:
- their first argument is a transaction instead of a map
- their second argument is a path (see below)
- their value argument or result (in the case of update) must be conformant
These must be called within a with-write-transaction
or a with-read-transaction
expression. Changes will only be persistent if with-write-transaction
is used.
get-at
assoc-at
update-at
merge-at
dissoc-at
Shortcut Functions
These are the same as the transactional-functions except that their first argument is a database instead of a transaction. These are convenience functions which automatically create and execute transactions. The write variants will also return the result of the modification.
get-at!
assoc-at!
update-at!
merge-at!
dissoc-at!
Seek Functions added in 1.2.0
These allow you to get ordered subsets of data from the database "map" in the form of ordered key-value pairs. Each accepts optional :limit
and :reverse
keyword parameters. They follow the same naming conventions of the other functions (plain variants expect a tx
argument and !
variants expect a db
argument).
seek-at
&seek-at!
- get ordered key-value pairs from the map at the providedpath
seek-from
&seek-from!
- get ordered key-value pairs from the map at the provided path for keys >=start-val
seek-to
&seek-to!
- get ordered key-value pairs from the map at the provided path for keys <=end-val
seek-range
&seek-range!
- get ordered key-value pairs from the map at the provided path for keys >=start-val
and <=end-val
seek-prefix
&seek-prefix!
- get ordered key-value pairs from the map at the provided path for string or keyword keys which begin withval-prefix
seek-prefix-range
&seek-prefix-range!
- get ordered key-value pairs from the map at the provided path for string or keyword keys which begin with a value between (inclusive)start-prefix
&end-prefix
See Seek Examples
Paths
A path
is a vector of keys similar to the [k & ks]
used in function like assoc-in
with a few exceptions:
- they are limited to the following types:
- Symbols
- Keywords
- Strings
- Numbers (float/double use is strongly discouraged)
- true
- false
- nil
- java.time.Instant
- org.joda.time.DateTime
- the path can only target nested maps, and cannot be used to descend into other data structures (e.g. vectors).
- you can get the empty path (e.g.
(get-at db [])
returns the full database) but you cannot modify it (e.g.(assoc-at [] :foo)
throws an error)
If you are you need support for additional types, please review doc/types.md.
Conformant Values
- non-map values of any type serializable by nippy
- this will only be relevant to you if you are storing custom records or exotic datatypes. Out of the box, virtually all standard clojure datatypes are supported (i.e. you don't need to do anything special to store lists/vectors/sets/etc.)
- the serialization is performed automatically, you do not need to serialize values manually
- maps and nested maps whose keys conform to the valid path types listed above
Transactions
Immutability
Transactions are immutable. Each transformation (e.g. assoc-at
, update-at
) returns a new transaction, it does not modify the transaction. Essentially you should treat them as you would a standard clojure map, one that you interact with using the *-at
functions.
Example:
(c/with-write-transaction [db tx-original]
(let [tx-a (c/assoc-at tx-original [:letter] "a")
tx-b (c/assoc-at tx-original [:letter] "b")]
tx-a))
(c/get-at! db [:letter]) ; "a"
See the FAQ for examples of potential pitfalls.
Visibility
Changes in a transaction are only visible to subsequent transformations on that transaction. They are not visible anywhere else until committed (by being the final result in the body of a with-write-transaction
expression). The changes are also not visible in any read transaction opened before the write transaction is committed.
Example:
(c/with-write-transaction [db tx]
(-> tx
(c/assoc-at [:number] 1000)
(c/update-at [:number] inc)))
(c/get-at! db [:number]) ; 1001
Exceptions
If an Exception is thrown within a with-write-transaction
expression, the transaction is aborted and no changes are persisted.
Locking
Write transactions block other write transactions (though they do not block read transactions). It is best to avoid doing any computationally complex or IO heavy tasks (such as fetching remote data) inside a with-write-transaction
block. See Performance for more details.
Examples
(require [codax.core :as c])
Simple Use
(def db (c/open-database! "data/demo-database")) ;
(c/assoc-at! db [:assets :people] {0 {:name "Alice"
:occupation "Programmer"
:age 42}
1 {:name "Bob"
:occupation "Writer"
:age 27}}) ; {0 {:age 42, :name "Alice", ...}, 1 {:age 27, :name "Bob", ...}}
(c/get-at! db [:assets :people 0]) ; {:name "Alice" :occupation "Programmer" :age 42}
(c/update-at! db [:assets :people 1 :age] inc) ; 28
(c/merge-at! db [:assets] {:tools {"hammer" true
"keyboard" true}}) ; {:people {...} :tools {"hammer" true, "keyboard" true}}
(c/get-at! db [:assets])
;; {:people {0 {:name "Alice"
;; :occupation "Programmer"
;; :age 42}
;; 1 {:name "Bob"
;; :occupation "Writer"
;; :age 28}}
;; :tools {"hammer" true
;; "keyboard" true}}
(c/close-database! db)
Transaction Example
(def db (c/open-database! "data/demo-database"))
;;;; init
(c/with-write-transaction [db tx]
(c/assoc-at tx [:counters] {:id 0 :users 0}))
;;;; user fns
(defn add-user
"create a user and assign them an id"
[username]
(c/with-write-transaction [db tx]
(when (c/get-at tx [:usernames username] )
(throw (Exception. "username already exists")))
(let [user-id (c/get-at tx [:counters :id])
user {:id user-id
:username username
:timestamp (System/currentTimeMillis)}]
(-> tx
(c/assoc-at [:users user-id] user)
(c/assoc-at [:usernames username] user-id)
(c/update-at [:counters :id] inc)
(c/update-at [:counters :users] inc)))))
(defn get-user
"fetch a user by their username"
[username]
(c/with-read-transaction [db tx]
(when-let [user-id (c/get-at tx [:usernames username])]
(c/get-at tx [:users user-id]))))
(defn rename-user
"change a username"
[username new-username]
(c/with-write-transaction [db tx]
(when (c/get-at tx [:usernames new-username] )
(throw (Exception. "username already exists")))
(when-let [user-id (c/get-at tx [:usernames username])]
(-> tx
(c/dissoc-at [:usernames username])
(c/assoc-at [:usernames new-username] user-id)
(c/assoc-at [:users user-id :username] new-username)))))
(defn remove-user
"remove a user"
[username]
(c/with-write-transaction [db tx]
(when-let [user-id (c/get-at tx [:usernames username])]
(-> tx
(c/dissoc-at [:username username])
(c/dissoc-at [:users user-id])
(c/update-at [:counters :users] dec)))))
;;;;; edit users
(c/get-at! db) ; {:counters {:id 0, :users 0}}
(add-user "charlie") ; nil
(c/get-at! db)
;; {:counters {:id 1, :users 1},
;; :usernames {"charlie" 0},
;; :users {0 {:id 0, :timestamp 1484529469567, :username "charlie"}}}
(add-user "diane") ; nil
(c/get-at! db)
;; {:counters {:id 2, :users 2},
;; :usernames {"charlie" 0, "diane" 1},
;; :users
;; {0 {:id 0, :timestamp 1484529469567, :username "charlie"},
;; 1 {:id 1, :timestamp 1484529603444, :username "diane"}}}
(rename-user "charlie" "chuck") ; nil
(c/get-at! db)
;; {:counters {:id 2, :users 2},
;; :usernames {"chuck" 0, "diane" 1},
;; :users
;; {0 {:id 0, :timestamp 1484529469567, :username "chuck"},
;; 1 {:id 1, :timestamp 1484529603444, :username "diane"}}}
(remove-user "diane") ; nil
(c/get-at! db)
;; {:counters {:id 2, :users 1},
;; :usernames {"chuck" 0, "diane" 1},
;; :users {0 {:id 0, :timestamp 1484529469567, :username "chuck"}}}
(c/close-database! db)
Seek Examples
Directory Example
(def db (c/open-database! "data/example-database"))
(c/assoc-at! db [:directory]
{"Alice" {:ext 247, :dept "qa"}
"Barbara" {:ext 228, :dept "qa"}
"Damian" {:ext 476, :dept "hr"}
"Adam" {:ext 357, :dept "hr"}
"Frank" {:ext 113, :dept "hr"}
"Bill" {:ext 234, :dept "sales"}
"Evelyn" {:ext 337, :dept "dev"}
"Chuck" {:ext 482, :dept "sales"}
"Emily" {:ext 435, :dept "dev"}
"Diane" {:ext 245, :dept "dev"}
"Chelsea" {:ext 345, :dept "qa"}
"Bob" {:ext 326, :dept "sales"}})
;; - seek-at -
(c/seek-at! db [:directory])
;;[["Adam" {:dept "hr", :ext 357}]
;; ["Alice" {:dept "qa", :ext 247}]
;; ["Barbara" {:dept "qa", :ext 228}]
;; ["Bill" {:dept "sales", :ext 234}]
;; ["Bob" {:dept "sales", :ext 326}]
;; ["Chelsea" {:dept "qa", :ext 345}]
;; ["Chuck" {:dept "sales", :ext 482}]
;; ["Damian" {:dept "hr", :ext 476}]
;; ["Diane" {:dept "dev", :ext 245}]
;; ["Emily" {:dept "dev", :ext 435}]
;; ["Evelyn" {:dept "dev", :ext 337}]
;; ["Frank" {:dept "hr", :ext 113}]]
(c/seek-at! db [:directory] :limit 3)
;;[["Adam" {:dept "hr", :ext 357}]
;; ["Alice" {:dept "qa", :ext 247}]
;; ["Barbara" {:dept "qa", :ext 228}]]
(c/seek-at! db [:directory] :limit 3 :reverse true)
;;[["Frank" {:ext 113, :dept "hr"}]
;; ["Evelyn" {:ext 337, :dept "dev"}]
;; ["Emily" {:ext 435, :dept "dev"}]]
;; - seek-prefix -
(c/seek-prefix! db [:directory] "B")
;;[["Barbara" {:dept "qa", :ext 228}]
;; ["Bill" {:dept "sales", :ext 234}]
;; ["Bob" {:dept "sales", :ext 326}]]
;; - seek-prefix-range -
(c/seek-prefix-range! db [:directory] "B" "D")
;;[["Barbara" {:dept "qa", :ext 228}]
;; ["Bill" {:dept "sales", :ext 234}]
;; ["Bob" {:dept "sales", :ext 326}]
;; ["Chelsea" {:dept "qa", :ext 345}]
;; ["Chuck" {:dept "sales", :ext 482}]
;; ["Damian" {:dept "hr", :ext 476}]
;; ["Diane" {:dept "dev", :ext 245}]]
(c/close-database! db)
Messaging Example
(def db (c/open-database! "data/example-database")) ;
(defn to-instant [s]
(java.time.Instant/ofEpochMilli (java.util.Date/parse s)))
(defn post-message!
([user body]
(post-message! (java.time.Instant/now) user body))
([inst user body]
(c/assoc-at! db [:messages inst] {:user user
:body body})))
(defn process-messages
[messages]
(map (fn [[inst m]] (assoc m :time (str inst))) messages))
(defn get-messages-before [ts]
(process-messages
(c/seek-to! db [:messages] ts)))
(defn get-messages-after [ts]
(process-messages
(c/seek-from! db [:messages] ts)))
(defn get-messages-between [start-ts end-ts]
(process-messages
(c/seek-range! db [:messages] start-ts end-ts)))
(defn get-recent-messages [n]
(-> (c/seek-at! db [:messages] :limit n :reverse true)
process-messages ;;
reverse)); we reverse the result because we want the messages to be in chronological order
; but we needed to use the :reverse seek parameter to prevent collecting all
; of the messages from the beginning of time (there could be many thousands!)
(defn simulate-message! [date-time-string user body]
(post-message! (to-instant date-time-string) user body))
(simulate-message! "June 6, 2020 11:01" "Bobby" "Hello")
(simulate-message! "June 6, 2020 11:02" "Alice" "Welcome, Bobby")
(simulate-message! "June 6, 2020 11:03" "Bobby" "I was wondering how codax seeking works?")
(simulate-message! "June 6, 2020 11:07" "Alice" "Please be more specific, have you read the docs/examples?")
(simulate-message! "June 6, 2020 11:08" "Bobby" "Oh, I guess I should do that.")
(simulate-message! "June 7, 2020 14:30" "Chuck" "Anybody here?")
(simulate-message! "June 7, 2020 14:35" "Chuck" "Guess not...")
(simulate-message! "June 8, 2020 16:50" "Bobby" "Okay, so I read the docs. What is the :reverse param for?")
(simulate-message! "June 8, 2020 16:55" "Alice" "Basically, it seeks from the end and works backwards")
(simulate-message! "June 8, 2020 16:56" "Bobby" "Why would I do that?")
(simulate-message! "June 8, 2020 16:57" "Alice" "Well, generally it is used to grab just the end of a long dataset.")
(get-recent-messages 3)
;;({:user "Alice" :time "2020-06-08T22:55:00Z" :body "Basically, it seeks from the end and works backwards"}
;; {:user "Bobby" :time "2020-06-08T22:56:00Z" :body "Why would I do that?"}
;; {:user "Alice" :time "2020-06-08T22:57:00Z" :body "Well, generally it is used to grab just the end of a long dataset." })
(get-messages-after (to-instant "June 7, 2020 14:32"))
;;({:user "Chuck" :time "2020-06-07T20:35:00Z" :body "Guess not..."}
;; {:user "Bobby" :time "2020-06-08T22:50:00Z" :body "Okay, so I read the docs. What is the :reverse param for?"}
;; {:user "Alice" :time "2020-06-08T22:55:00Z" :body "Basically, it seeks from the end and works backwards"}
;; {:user "Bobby" :time "2020-06-08T22:56:00Z" :body "Why would I do that?"}
;; {:user "Alice" :time "2020-06-08T22:57:00Z" :body "Well, generally it is used to grab just the end of a long dataset." })
(get-messages-before (to-instant "June 6, 2020 11:05"))
;;({:user "Bobby" :time "2020-06-06T17:01:00Z" :body "Hello"}
;; {:user "Alice" :time "2020-06-06T17:02:00Z" :body "Welcome, Bobby"}
;; {:user "Bobby" :time "2020-06-06T17:03:00Z" :body "I was wondering how codax seeking works?"})
(get-messages-between (to-instant "June 7, 2020")
(to-instant "June 7, 2020 23:59"))
;;({:user "Chuck" :time "2020-06-07T20:30:00Z" :body "Anybody here?"}
;; {:user "Chuck" :time "2020-06-07T20:35:00Z" :body "Guess not..."})
(c/close-database! db)
Frequently Asked Questions
Why aren't all my changes being saved?
Because transactions are immutable, if an updated transaction is discarded, the transformations it contains will not be committed.
Incorrect:
(c/with-write-transaction [db tx]
(c/assoc-at tx [:users 1] "Alice") ; this write is "lost"
(c/assoc-at tx [:users 2] "Bob"))
(c/get-at! db [:users]) ; {2 "Bob"}
Correct:
(c/with-write-transaction [db tx]
(-> tx ; thread the transaction through multiple transformations
(c/assoc-at [:users 1] "Alice")
(c/assoc-at [:users 2] "Bob")))
(c/get-at! db [:users]) ; {1 "Alice" 2 "Bob"}
Why am I getting a NullPointerException in my Write Transaction?
A common cause is that the body of the with-write-transaction
form is not evaluating to (returning) a transaction.
Incorrect:
(defn init-counter! []
(c/with-write-transaction [db tx]
(when-not (c/get-at tx [:counter])
(c/assoc-at tx [:counter] 0))))
(init-counter!) ; nil (it works the first time)
(init-counter!) ; java.lang.NullPointerException (the body evaluates to nil)
Correct:
(defn init-counter! []
(c/with-write-transaction [db tx]
(if-not (c/get-at tx [:counter])
(c/assoc-at tx [:counter] 0)
tx))) ;; if the counter is already initialized, return the unmodified transaction
(init-counter!) ; nil
(init-counter!) ; nil
Performance
Codax is geared towards read-heavy workloads.
- Read-Transactions block nothing
- Write-Transactions block other Write-Transactions
- Stage-1 Compaction blocks Write-Transactions (slow)
- Stage-2 Compaction blocks both Reader-Transactions and Write-Transactions (fast)
Benchmark Results
Jan 14, 2017
The following figures are for a database populated with 16,000,000 (map-leaf) values running on a Digital Ocean 2-core 2GB RAM instance. The write transactions have an average "path" length of 6 and an average 7 leaf values.
- ~320 write-transaction/second
- ~1640 read-transactions/second
- ~2700ms per compaction (compaction happens automatically every 10,000 writes)
This benchmark is a bit dated, but a similar benchmarking function is available as codax.bench.performance/run-benchmark
.
Bugs
...
Contributing
Insights, suggestions, and PRs are very welcome.
License
Copyright © 2018 David Scarpetti
Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.