CreateCommandsInvoker update hangs when stack is in state UPDATE_ROLLBACK_COMPLETE
FilipeAleixo opened this issue · comments
I've noticed that sam deploy hangs in this specific situation:
I do an unsuccessful deploy which has to be rolled back due to an error, which then puts the stack in state UPDATE_ROLLBACK_COMPLETE.
When I subsequently correct the error and call sam deploy again it hangs in this stage:
This is what I see in the stack:
Even deleting the stack becomes impossible after this because there's no way to stop the custom resource (at least I didn't find one), so I just have to wait for it to time out and delete it.
Any clue on how to fix this?
Just pinging you to show what I've built using your repo: https://github.com/EsteFilipe/discord-ethereum-authentication
Thanks for the clean code :)
As far as I understand, this error occurs when a bug is introduced into CreateCommandsFunction that prevents CfnLambda to run as intended and post the custom resource status back to CloudFormation via a pre-signed URL that is provided to the Lambda function. (For more details of how CloudFormation custom resources work under the hood, refer to here).
As long as the CloudFormation resource update is running, it's not possible to update or delete the stack.
Unfortunately, the timeout period for custom resource status updates is quite long (2 hours?) and as far as I can tell, there is currently no way to change this.
The only options I see are:
- Wait for the custom resource to timeout and create another stack for development in the meantime
- Try to send the status update via HTTP yourself (link) - note: I didn't try this myself
I agree that this is very annoying during development but I do not see any way how to make things easier here. This would be AWS`s turn. If you find out something new with regeard to this issue or if it works out well to send the expected status update yourself - feel free to share!
By the way: Your project looks awesome! Thanks for sharing!
Thanks for the thorough explanation!
The only way I've found to overcome this for development was to just keep creating stacks with new names, while leaving the failed ones so that they time out and I delete them later. This seems like quite a shortcoming from Cloudformation - they even have videos like this one suggesting a hack of send a signal to the lambda in order to be able to delete it https://www.youtube.com/watch?v=hlJkMoCxR-I
