Avoid diff cache double-write with leader election

[Jonathan Chan Kwan Yin]

Description

Currently diff controller runs a multi-leader elector to ensure high availability. However this puts a lot of stress on the diff cache backend.

To optimize this, we can introduce a new elector for writing to diff cache. While the original lease continues to restrict the number of watch streams, the new elector only restricts the number of diff cache writers.

stateDiagram
  A --> B: acquire watch lease
  B --> C: acquire cache writer lease
  C --> D: lose cache writer lease
  D --> B: maintained state\nfor more than\n15 seconds

Loading

State	Watch lease	Cache writer lease	Behavior
A	Not acquired	Do not participate in election	Try to acquire watch lease. Do not watch objects.
B	Acquired	Not acquired	Try to acquire cache writer lease. Watch objects and maintain a local cache for 15 seconds. Do not write to remote diff cache.
B -> C	Acquired	Not acquired -> acquired	Write the local cache for the previous 15 seconds to the remote diff cache.
C	Acquired	Acquired	Watch objects and write to remote diff cache.
D	Acquired	Acquired -> lost election for less than 15 seconds	Watch objects and write to remote diff cache.
B	Acquired	Lost election for more than 15 seconds	Try to acquire cache writer lease. Watch objects and maintain a local cache for 15 seconds. Do not write to remote diff cache.

User story

No response