Improve sync to bring new-view
tock-ibm opened this issue · comments
If a leader gets partitioned and the partition heals, but after the view change there were no transactions, the current Synchronizer.Sync() will not inform the node on the new view.
The core of the problem is because our state-transfer is partial - compared to pbft and bft-smart - we do not bring the most recent part of the "log" (in the general sense), which in our case is the WAL, in the sync. We only bring what was decided and logged in the app's log (i.e. get the blocks form Fabric).
Therefore, one way to solve problem, and possibly other problems that we did not tackle yet, is to improve the sync process in general. We leave the Syc() implemented in Fabric as is, to do catch up on decided blocks.
On top of that, we add a "state-transfer" protocol in the library, that would bring to the requesting node the part of the state in peer nodes that is not provided in Synchronizer.Sync() - that is, information about view changes.
In particular, the "ViewChange" & "NewView" messages written to the WAL are of interest to this protocol.
An outline of the protocol is:
- The controller sync() would call the Fabric (dependency Synchronizer) Sync() that catches up decided blocks and brings the last decision.
- Then (or in parallel), it could call a "State-transfer-request" (for lack of a better term). Peers respond to that by sending in a "State-transfer-response" message, that includes some description of the latest messages written to the WAL (from memory, actually).
- Nodes don't have to send the propose+prepare or commit in full, as that is delivered by Sync(); they have to send only an indication of the View+Sequence an the phase they are in; e.g.: <'NewView', v, s> or <'Prepare', v, s> or <'Commit', v, s>, etc.
- The requesting node can then wait for f+1 identical responses and make progress from there.
So, all we need is to get the NewView message.
I suggest that the "state transfer request" will simply contain a view sequence number, and the response will contain a NewView message if the view number is greater than the number of the request, or simply an acknowledgement that the view number is the same.
We also don't need to wait for f+1 identical responses when we get a response containing a NewView message because it's signed.
Right now the NewView message is not actually written to the WAL, just the view and sequence agreed on, but it is easy to add it if we want to check signatures.
And remember that even if one node sent a NewView message it doesn't mean that it is the latest view. You need at least f+1 to know what is the latest you are expecting.
Right now the NewView message is not actually written to the WAL, just the view and sequence agreed on, but it is easy to add it if we want to check signatures.
Good point, so we need to change it to persist the NewView message.
And remember that even if one node sent a NewView message it doesn't mean that it is the latest view. You need at least f+1 to know what is the latest you are expecting.
I think that if we do the following:
- Request sends the view number and proposal sequence that Sync returned.
- Response sends back either an ACK with the matching view number, or a NewView message in case the new view is higher than the view number in the request.
If we see f+1 ACKs we don't do anything. Else, there is at least one NewView message that has a higher view number. If the proposal sequence in the NewView matches ours, it means we just need to adopt the new view. Else, we still have some proposals to sync, so we don't do anything, and we will sync again some time later.
So if Sync finished we have one of the 3 possible outcomes:
- We got to the latest view - so in the requests we send this view number, and in that case we wait for f+1 identical messages that say that we got to the latest view, and no need to send NewView as a response.
- We got to a view that is behind of the latest view, and no proposals have been made since our view. In such a case we get some NewView message that has a proposal sequence that matches ours, and we then just adopt this NewView.
- The complement of 1+2: We got to a view that is behind the latest view, and some proposals have been made, in which case we can just sync to that view.
By the way - I opened an issue about this very item before... https://github.com/SmartBFT-Go/consensus/issues/325 I just forgot it exists :-)
I think we just need the view metadata that is already written to the WAL when we see a newView message.
We need to wait for a quorum of view metadata and if we see f+1 identical view number that is higher then what we have from syncing then we know that there was a view change and this is the latest one.
Given that we persist the view metadata currently, it makes sense to indeed do just that.