AsyncMethodsRateLimiter does not handle ratelimitted errors properly

[Greg Unrein]

Thanks again for providing this client!

We're seeing deep recursion in enqueueThenRun (example below) and no backoff when the Slack API responds with HTTP 429 with the async methods client (specifically the call to enqueueThenRun in the catch (SlackApiException e) block). In the example, the time between the requests made by each enqueueThenRun is usually in the 100-300ms range which is effectively the API response latency so the requests are happening about as fast as possible (no backoff).

We're using the default executors with a pool size of 15 and a custom rate limit of 30 for users.profile.set (we added this custom limit recently while trying to mitigate this issue and were seeing the issue before adding the rate limit).

In this case it seems like respecting the server's retry-after header value and waiting for that long to retry requests for that specific team ID+method combination would be an improvement for both the client and the load on the Slack API.

I'm happy to help with a fix if you'd like, but would need some guidance on how you'd like it done.

Reproducible in:

The Slack SDK version

+--- com.slack.api:slack-api-client:1.37.0
|    +--- com.slack.api:slack-api-model:1.37.0

Expected result:

API calls to the Slack API back off when the server responds with an HTTP 429 and recursionm

Actual result:

These calls to enqueueThenRun each take around 100-300ms even though the Slack API is sending back 429 responses with retry-after set to several seconds (it varies).

at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:149)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.enqueueThenRun(AsyncRateLimitExecutor.java:169)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor.lambda$execute$0(AsyncRateLimitExecutor.java:77)
at app//com.slack.api.methods.impl.AsyncRateLimitExecutor$$Lambda$4429/0x00007f27e9db7bc0.get(Unknown Source)
at java.base@17.0.9/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)

[Kazuhiro Sera]

Hi @gunrein, thank you for sharing this. Indeed, the part could be even more rubost while the async client does not expect getting ratelimited as long as it's properly used.

The sync client does handle the pattern properly here: https://github.com/slackapi/java-slack-sdk/blob/v1.37.0/slack-api-client/src/main/java/com/slack/api/methods/impl/MethodsClientImpl.java#L3532-L3541 So, adding something similar to all the async clients can improve the async client side. I will work on it later.

[Kazuhiro Sera]

@gunrein Thank you so much again for reporting this issue. I've figured the root cause out and came up with the PR #1274 to resolve the issue. A new version including the fix will be released shortly.

[Greg Unrein]

Thank you very much @seratch ! That was an amazingly quick turnaround.