Upgrade preflight check should fail if machines in cctl state file is different from Kubernetes nodes

[Puneet Gupta]

Let us define the following two sets

S = machines in state file with master role
C = node objects with label master as reported by kubectl get nodes

Currently preflight check checks if the nodes in set C are ready

• In most cases S == C, and preflight check does the right thing
• However if there are out of band operations like node being deleted by kubectl delete node or state file being edited then S-C or C-S can be non empty sets. In that case
  • The upgrade is bound to fail (S-C not empty)
  • or Succeed only partially (C-S not empty)
  • Both can be avoided by adding this to the preflight check.

Update preflight check to do this set difference and fail fast if S != C

[Puneet Gupta]

Discussed with Arun this is needs to be fixed reopening the issue