Task concurrency slot availability check improvements

[Jay Hack]

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar request and didn't find it.
• I searched the Prefect documentation for this feature.

Prefect Version

2.x

Describe the current behavior

Currently per this line prefect waits 30 seconds before checking for an open task run concurrency slot. This can result in a large slowdown. One example is in the community slack here.

Describe the proposed behavior

@madkinsz and I discussed a few different options to make task concurrency slot availability checks faster, more flexible, or more intelligent.

• Drop the delay_transition seconds arg naively, to something like one or five seconds.
• Expose a setting for the concurrency slot check delay length
• Use historic task run length to inform the delay for a given task. Worth noting that in the linked case of many tasks that take one second to run, this would effectively drop the delay length significantly.
• Compute the check delay to be some constant (1 second?) times the tasks place in a queue - the next up task would poll every second for an available concurrency slot, the nth would wait n seconds.
• Enable an exponential backoff behavior to each tasks concurrency check delay

Example Use

No response

Additional context

No response