Think of a better env refresh when instance count is 1
partouf opened this issue · comments
Often when refreshing the GPU environment, there's only 1 instance active. And when you do ce --env gpu environment refresh it will launch a new instance, but it does not wait long enough before shutting down the original instance, causing the gpu environment to return errors for a while until the new instance is healthy.
Figure out if this is a solved problem on AWS, or otherwise change the logic to Increase instance count to 2 first and wait for that 2nd instance to be alive, then shutdown the original instance and return the count to 1.
Unless its staging/beta, no need to do it there