Idea: from first batch of ref points in each swap step, only keep track of best medoid to flip with (probably the one it's assigned to?)

[Mo Tiwari]

Then this becomes a form of local search