Giters

openHPI / poseidon

Scalable task execution orchestrator for CodeOcean

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

New connections to draining Runners

mpass99 opened this issue · comments

commented

Our current drain_on_shutdown strategy for stopping Nomad agents is:

  • On Shutdown the Nomad agent gets ineligible and no new runners are being scheduled.
  • In the drain-on-shutdown deadline all running executions have time to finish.
    • ⚡ We still start new Executions in runners on the draining agent that may not have enough time to finish
  • After that the Nomad Agent shuts down

The executions that don't have enough time to finish result in a user-visible error.

We might need to "exclude" some runners for new executions as soon as the respective Nomad agent is about to shut down.

See #651

Unfortunately, we currently don't have any metric to count how often this issue occurs.

commented

ToDo: Let's identify which error / log information / ... we get when above issue occurs.

commented

We've conducted a local reproduction of this scenario: nomadEventLog-ExecuteDraining.txt.
It shows that POSEIDON-3W (#590) with the sub-error the allocation was rescheduled indicates this error. This has not happened for at least 90 days.

If we consider a fix for this necessary in the future, we might consider listening to Nomad's Node events to receive drain updates, fetch all allocations of this node, and block new executions for these allocations/runners. Further, we should ensure that the drain deadline matches the maximum of all allowed execution timeouts (of CodeOcean).

commented

This issue is still valid and could be a nice improvement. However, we don't expect that this problem occurs many times, so that it doesn't have a high priority.