BlockProviderExecutor executor blocks are not scaled in if bad_state is set

[Ben Clifford]

Describe the bug

If a BlockProviderExecutor is in bad state due to set_bad_state_and_fail_all, Parsl does not scale in the blocks for that executor at DFK shutdown.

It probably should - block management is part of the job subsystem, not part of the executor, I think.

Expected behavior
parsl runs should not leave orphaned blocks.

Environment

• Parsl version 2023.10.23

[Ben Clifford]

It's likely that if this behaviour was naively changed to try to scale_in blocks at exit, #2627 would apply and so still stop some blocks being scaled in - this is the case in the user's log file that I encountered this issue #2961.