chdsbd / kodiak

Recently we've discovered that kodiak is updating all branches that have automerge set, regardless of their place in the queue. The status that's set on the commits is "⛴ merging PR (updating branch)". This causes quite a lot of extra CI jobs, especially when we've got long queues, leading to way longer merge times.

Reading through the source code and recent changes I'm not seeing any obvious candidates that might be causing this.
We don't know when this started, could have been some time ago.
The only configs we're changing from the defaults are merge.message.title and merge.priority_merge_label.

Here are some random 3 pull requests:
island-is/island.is#5749
island-is/island.is#5742
island-is/island.is#5721

We can provide more examples or information as needed.

Hey @sindrig, thanks for the report.

This is weird behavior. I'm not sure why for example this PR is marked as missing reviews: island-is/island.is#5717
It seems like it was merging but then Kodiak said it was missing reviews

Also, looking in the logs, it seems like there were two Kodiak repo workers running at the same time merging 5750 and 5742

https://github.com/chdsbd/kodiak/blob/0c6cda5169e2f311ed654877b11a15feba65ad05/bot/kodiak/queue.py#L430:11

I'm going to try eliminating those incorrect "missing required reviews" status messages in #762

Thanks for the quick response @chdsbd.

I did suspect some kind of race condition or error due to parallelism but didn't have anything to go by. When you say two Kodiak repo workers running, you mean there were two runners running in your systems, right? Is there anything we could have spotted?

I'll try to keep an eye on pull requests once developers start pushing changes tomorrow and update here if we see this happening still.

Yeah, I meant two internal workers in Kodiak (actually asyncio tasks) processing the same repo.

Kodiak starts repo worker tasks with this code:

kodiak/bot/kodiak/queue.py

Lines 519 to 525 in 0c6cda5

    
           def start_repo_worker(self, *, queue_name: str) -> None: 
        
               self._start_worker( 
        
                   queue_name, 
        
                   "repo", 
        
                   repo_queue_consumer(queue_name=queue_name, connection=self.connection), 
        
               )

I'm honestly not sure yet how it's possible to create two duplicate tasks. 😬

Pretty sure this is still happening. See this one for example:

island-is/island.is#5733

island-is/island.is@ecefbbf
island-is/island.is@a7c236b
island-is/island.is@d647687

Looking in the logs, it seems like the issue of having two workers at the same time isn't happening anymore, but, it seems like I introduced a small bug via e60988d, where if changes are requested, the bot doesn't update the branch correctly.

I think this can cause some unnecessary updates.

I've deployed #765 which should help with some of the unnecessary updates when changes are requested

	def start_repo_worker(self, *, queue_name: str) -> None:
	self._start_worker(
	queue_name,
	"repo",
	repo_queue_consumer(queue_name=queue_name, connection=self.connection),
	)

Unnecessary branch updates