Transaction support

[Casey Link]

A library like this isn't complete without some support for transactions.

One first step, that might also be sufficient, is to expose a with-transaction macro that wraps the next.jdbc's with-transaction macro.

Reading:

• https://hexdocs.pm/ecto/Ecto.Multi.html
• https://curiosum.dev/blog/elixir-ecto-database-transactions

[Casey Link]

From Slack:

kwrooijen
6:21 PM
I think we need to do the following things. what are your thoughts?

• Think of a nice structure for aggregating the executions in data. If a more functional way is desired we can create honeysql-like wrapper functions to build the structure
• We will need a transaction-connection dynamic definition which will take precedence over any other connection (for the binding in transact)
• In the exception handler we need to check if transaction is set, if it is then we need to throw the data instead of returning it
• Create a transaction function in factory. This function will take the datastructure and execute each one in sequence. If any fail, extract the data with ex-data and return that with the pairing identifier key. (edited)

Casey
6:36 PM
Sounds good

6:37
I find it helpful to write an example of what I think the user api would look like, I suppose we could do that with the elixir account transfer example

[Casey Link]

Going straight for the complicated case: running multiple step pipelines in a transaction.

I modeled this example after the Elixir Ecto article linked above.

It represents a multi step pipeline in a domain where each step could produce its own errors. It represents the stages, results, and errors as data with functions, no macros necessary.

;; given this model 
(def model-account
  [:map
   [:account/id {:primary-key true} int?]
   [:account/balance int?]
   ])

;; define the functions in the pipeline
(defn retrieve-accounts [a1-id a2-id]
  (fn []
    (let [accounts {a1-id (q/find-by :account/id a1-id)
                    a2-id (q/find-by :account/id a2-id)}]
      (if (every? some? accounts)
        (vals accounts)
        {:error [:account-not-found (mapv first (filter (fn [[_ v]] (nil? v)) accounts))]}))))

(defn verify-balances [transfer-amount]
  ;; each pipeline fn is passed a map of steps executed thus far
  (fn [{[sender recipient] :retrieve-accounts}]
    (if (< (:account/balance sender) transfer-amount)
      [sender recipient transfer-amount]
      {:error [:balance-too-low sender]})))

(defn subtract-from-sender [{[sender recipient verified-amount] :verify-balances}]
  (q/save! (changeset sender
                      (update sender :account/balance #(- % verified-amount)))))

(defn add-to-recipient [{[sender recipient verified-amount] :verify-balances}]
  (q/save! (changeset recipient
                      (update recipient :account/balance #(+ % verified-amount)))))

(defn transfer-money [sender-id recipient-id amount]
  [[:retrieve-accounts (retrieve-accounts sender-id recipient-id)]
   [:verify-balances (verify-balances amount)]
   [:subtract-from-sender subtract-from-sender]
   [:add-to-recipient add-to-recipient]])

;; execute the pipeline in a transaction
(exec-tx-pipeline (transfer-money 1 2 100))

;; => returns an ok map with a vector containing the results of each step in order
{:ok [[:retrieve-accounts [{:account/id 1 :account/balance 500} {:account/id 2 :account/balance 10}]]
      [:verify-balances [{:account/id 1 :account/balance 500} {:account/id 2 :account/balance 10} 100]]
      [:subtract-from-sender ...]
      [:add-to-recipient ...]]}


;; => unless an error occurs, in which case the pipeline short circuits and returns a map containing the error, and the results of all previous steps
{:error [:verify-balances :balance-too-low]
 :results [:retrieve-accounts [{:account/id 1 :account/balance 50} {:account/id 2 :account/balance 10}]]}

As you can see the convention is that the step functions return their result or on error a map keyed with :error. Except for the functions that return the (q/save!) directly, on error here the :changeset/error will be present which the pipeline executor should handle.

I'm not totally pleased with the data format here, but I do think vectors make more sense than maps considering order is important. If you've used re-frame on the front end, you'll see a similarity between the result vectors and re-frame event vectors.

[Casey Link]

For the simpler case, i.e., execute this single function in a transaction with no special error handling ceremony we can wrap next.jdbc's transact function. To the user it would look something like:

(defn handle-create-user [user-name email]
  (let [result] (q/transact
                 (fn []
                   (-> (changeset {:user/name name :user/email email})
                       q/save!
                       send-welcome-email!)))
       (if (some? (:tx/rollback result))
         ;; handle error
         ;; success!
         )))

For this to work q/save would throw on :changeset/errors and on sql execution errors.

[Kevin van Rooijen]

Fram Slack:

kwrooijen 11:45 AM
Then I think it's a good setup. I agree that vectors are a bit strange instead of maps, but as you pointed out we need to hold on to the order of execution. The only thing I would change is making the keywords qualified. e.g. :transaction/error :transaction/results (and I would replace :ok with :transaction/results as well, just don't include :transaction/error)

Casey 11:53 AM
{:transaction/results {:retrieve-accounts [{:account/id 1 ...}...], :verify-balances ... }
:transaction/steps [:retrieve-accounts :verify-balances :subtract-from-sender ...]}

kwrooijen 11:54 AM
That's also a nice option actually
Kind of reminds me of the changesets

Casey 11:55 AM
just had the same thought hah 😀

kwrooijen 11:55 AM
Maybe rename steps to pipeline ?

---

Allso the simple use case looks good too. All of these functions should be defined in gungnir.transaction. We can look at function naming later.