rstonkus/Paxos

Paxos

This is an implementation of the basic Paxos algoritm originally described by Leslie Lamport in The Part-Time Parliament. See also Wikipedia, for a more accessible explanation.

Functionality

It is implemented using the following messages and roles directly

Client   Proposer      Acceptor     Learner
   |         |          |  |  |       |  |
   X-------->|          |  |  |       |  |  Request
   |         X--------->|->|->|       |  |  Prepare(1)
   |         |<---------X--X--X       |  |  Promise(1,{Va,Vb,Vc})
   |         X--------->|->|->|       |  |  Accept!(1,Vn)
   |         |<---------X--X--X------>|->|  Accepted(1,Vn)
   |<---------------------------------X--X  Response
   |         |          |  |  |       |  |

An operation is a named function

  type Value = string option
  type ValueLastTouched = Value * (Sender * Guid) list
  type Operation = Operation of string * (ValueLastTouched option -> ValueLastTouched)

Paxos does not guarantee idempotency. The list attached to each value: (Sender * Guid) list is useful for implementing it on top. It is a list of (proposer, request-guid), and will not grow longer than the amount of proposers. Idempotency is implemented using

  let idempotent ((fname,f):string * (Value -> Value)) (sender:Sender) (g:System.Guid) : Operation =
    let op vo = 
      match vo with
      | None -> (f None,[(sender,g)])
      | Some (v,xs) -> 
        let xo = List.tryFind (fun (_,guid) -> guid = g) xs
        match xo with
        | None -> let xs' = List.filter (fst >> (<>) sender) xs
                  in (f v, (sender,g)::xs')
        | Some _ -> (v,xs)
    Operation (fname, op)

If you are using the idempotency wrapper, the operation signature is

  f :: string * (Value -> Value)

Even though people claim that paxos does not get stuck, I fail to see how it is possible (messages can be lost). So, it is crucial to support retries. The core algorithm defines time as

  type Time = int64

And it is the user of the library that defines the timeout

  hastimedout:Time -> bool

If the timeout is set too low, it might cause an infinite loop of retries!

Leader election

I have included an operator that can be used as leader election between servies using the library.

The idea is (courtesy of ildhesten - he very much ignited this idea)

Represent leaderStatus as (leader,lease)
Each participant who is not the leader, will decrement lease by 1
The leader will renew its lease to lease. I.e. lease will function as the timeout.

Example: 3 participants leaderElection is applied every 1s by each participant lease is 4

Then: Initially, the value will be None, and the first come will be the leader, and set its lease to lease All other participants will decrease his lease by 1, => lease is now 2 If the leader goes down, he will not be able to renew his lease. And so the other two participants will decrease his lease. The participant who sets the current lease to 0, will obtain leadership, and initialize his lease to lease

Whenever the current leader fails to renew his lease, he has to shutdown. The effective timeout (potential downtime) is 2 rounds

  //Untested !
  let leaderElection (participant:string) (lease:int) : string * (Value -> Value) =
    let f (vo:Value) : string = 
      match vo with
      | None -> 
        participant + "," + (lease.ToString())
      | Some v -> 
        let s = v.Split(',');
        let leader = s.[0]
        let coins = System.Int32.Parse(s.[1])
        if (leader = participant)
        then leader + "," + (lease.ToString()) //preserve leader as me
        else if (coins = 0)
             then participant + "," + (lease.ToString()) //choose new leader
             else leader + "," + ((coins-1).ToString())//preserve other leader
    in ("leaderElection",f >> Some)

Test

The core algorithm implemented in BasicPaxos.fs has been thouroughly tested by emulating a distributed system. The emulator runs in a single thread and implements random interleavings, processor crashes and message losses.

The following pseudo fsharp shows the essence of the emulator

  let runToEnd (debugIn:bool) (random:System.Random) emulateMessageLosses quorumSize participants result = 
    while noMoreMessages
      do
        match random.Next 5 with
        |0|1|2|3 -> //progress
          match random.Next 6 with
          | 0 | 1 | 2 -> ``consume 1 message by 1 randomly chosen participant``
          | 3 | 4 -> ``send``
          | _ -> ``checkTimeout``
        | _ ->
          if (random.Next 50 <> 0)  // P(evil) = 1/5 * 1/20 = 1/100
          then () 
          else if (``possible to crash without breaking the quorum``) //TODO This should not be necessary
		       then ``crash random participant for x rounds + clear input messages``

Since the processes are randomly interleaved, a crash might not manifest as "not receiving new messages", and as such it also models pure message loss (albeit with a somewhat low probability).

The test runs 2*2000 requests for incrementing a common value by two clients targeting each their proposer.

I have not yet been able to produce an inconsistence in the test.

Performance

I have not tested the performance, but I do expect it to be poor. It has the problem of dualing proposers. If one wanted to use it for high-throughput, once must either implement Multi-Paxos or at least find a way of directing the requests to one proposer at a time.

TODOs (and known bugs)

The emulator does not currently duplicate messages, and in fact the algorithm does not yet handle this scenario. It could however easily be extended to do so.
The quorum check before crashing a process in the emulator should not be necessary

rstonkus / Paxos