kubernetes-retired / kube-batch

A batch scheduler of kubernetes for high performance workload, e.g. AI/ML, BigData, HPC

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Statement unevict method doesn't update nodeInfo.Releasing field as expected

yodarshafrir1 opened this issue · comments

Is this a BUG REPORT or FEATURE REQUEST?:

kind bug

What happened:
unevict method in file: statement.go doesn't reset the field Releasing of the task's nodeInfo to the expected value (the value of Releasing which was changed by evict method will remain as if the task was released from the node).

This means that when a task is running on a node, evict will be called and then unevict will be called, the Releasing value of the current nodeInfo will be as if the task was evicted from the node.

This happens because unevict calls AddTask. In file: node_info.go AddTask there's a condition to check if the task exists on this node and if so we return an error and don't set the Releasing value to the right value.

Instead the method unevict in file: statement.go should call UpdateTask (this is the fix). In UpdateTask the Releasing value will be updated as if the task was never evicted from the node.

What you expected to happen:
The Releasing value of the node should be as if the task was never evicted from its node.

How to reproduce it (as minimally and precisely as possible):
Create a use case where preempt evicts a task but fails to pipeline a task, then statement.Discard will be called.
Print the value Releasing field of the nodeinfo and see that it has a wrong value.

is there anyway to trigger this case by workload?

This happened to me after changing the order of the actions, so preempt will run before allocate.

When preempt will run, we just need an attempt to evict a task and yet, not have enough resources to pipeline the preemptor. On this case stmt.Discard() is called.
When Allocate will run after this case, due to the following if:

// Allocate releasing resource to the task if any.
task.InitResreq.LessEqual(node.Releasing) {

A pending task that fits the node.Releasing will be pipelined to the node, even though that the running task was never really evcited.

By the way, reading the comments in the code suggests that it was intended to update the task and not add a task.

// Update task in node.
if node, found := s.ssn.Nodes[reclaimee.NodeName]; found {
    node.AddTask(reclaimee)
}

thanks very much for your contribution :)

Issue was solved in: #912