zhiwei-w-luo / dtm

A service orchestrator for distributed transactions, supports workflow, saga, tcc, xa, 2-phase message, outbox patterns, supports many languages.

English | 简体中文

Distributed Transactions Manager

What is DTM

DTM is a distributed transaction framework which provides cross-service eventual data consistency. It provides saga, tcc, xa, 2-phase message, outbox, workflow patterns for a variety of application scenarios. It also supports multiple languages and multiple store engine to form up a transaction as following:

Who's using DTM (partial)

Tencent

Bytedance

Ivydad

More

Features

• Multiple languages support: SDK for Go, Java, PHP, C#, Python, Nodejs
• Support for multiple transaction modes: SAGA, TCC, XA, Workflow, Outbox
• Better Outbox: 2-phase messages, a more elegant solution than Outbox, support multi-databases
• Multiple database transaction support: Mysql, Redis, MongoDB, Postgres, TDSQL, etc.
• Support for multiple storage engines: Mysql (common), Redis (high performance), MongoDB (under planning)
• Support for multiple microservices architectures: go-zero, go-kratos/kratos, polarismesh/polaris
• Support for high availability and easy horizontal scaling

Application scenarios.

DTM can be applied to data consistency issues in a large number of scenarios, here are a few common ones

• cache management: thoroughly guarantee the cache final consistency and strong consistency
• flash-sales to deduct inventory: in extreme cases, it is also possible to ensure that the precise inventory in Redis is exactly the same as the final order created, without the need for manual adjustment
• Non-monolithic order system: Dramatically simplifies the architecture
• Event publishing/subscription: better outbox model

Cook Book

Quick start

run dtm

git clone https://github.com/dtm-labs/dtm && cd dtm
go run main.go

Start an example

Suppose we want to perform an inter-bank transfer. The operations of transfer out (TransOut) and transfer in (TransIn) are coded in separate micro-services.

Here is an example to illustrate a solution of dtm to this problem:

git clone https://github.com/dtm-labs/dtmcli-go-sample && cd dtmcli-go-sample
go run main.go

Code

Use

  // business micro-service address
  const qsBusi = "http://localhost:8081/api/busi_saga"
  // The address where DtmServer serves DTM, which is a url
  DtmServer := "http://localhost:36789/api/dtmsvr"
  req := &gin.H{"amount": 30} // micro-service payload
  // DtmServer is the address of DTM micro-service
  saga := dtmcli.NewSaga(DtmServer, shortuuid.New()).
	// add a TransOut sub-transaction，forward operation with url: qsBusi+"/TransOut", reverse compensation operation with url: qsBusi+"/TransOutCom"
	Add(qsBusi+"/TransOut", qsBusi+"/TransOutCom", req).
	// add a TransIn sub-transaction, forward operation with url: qsBusi+"/TransIn", reverse compensation operation with url: qsBusi+"/TransInCom"
	Add(qsBusi+"/TransIn", qsBusi+"/TransInCom", req)
  // submit the created saga transaction，dtm ensures all sub-transactions either complete or get revoked
  err := saga.Submit()

When the above code runs, we can see in the console that services TransOut, TransIn has been called.

Timing diagram

A timing diagram for a successfully completed SAGA transaction would be as follows:

Rollback upon failure

If any forward operation fails, DTM invokes the corresponding compensating operation of each sub-transaction to roll back, after which the transaction is successfully rolled back.

Let's purposely fail the forward operation of the second sub-transaction and watch what happens

app.POST(qsBusiAPI+"/TransIn", func(c *gin.Context) {
  log.Printf("TransIn")
  // c.JSON(200, "")
  c.JSON(409, "") // Status 409 for Failure. Won't be retried
})

The timing diagram for the intended failure is as follows:

More examples

If you want more quick start examples, please refer to dtm-labs/quick-start-sample

The above example mainly demonstrates the flow of a distributed transaction. More on this, including practical examples of how to interface with an actual database, how to do compensation, how to do rollback, etc. please refer to dtm-examples for more examples.

Chat Group

Join the chat via https://discord.gg/dV9jS5Rb33.

Give a star! ⭐

If you think this project is interesting, or helpful to you, please give a star!