Retrofit is stopped handling requests after back from background for about 30 mins

[Hieu Truong]

Version:

• Retrofit 2.9.0
• OkHttp version 4.11.0

After putting the app in the background for a long time (around 30mins) and bringing it back to the foreground, my retrofit library only enqueues requests but not executing any of them and making the app hang because of waiting for API responses.

I tried to check inside Dispatcher.kt and saw that it triggered fun enqueue(call: AsyncCall). And the readyAsyncCalls keep saving my next requests (over 20) but runningAsyncCalls always stay with 5 items without any processing to clear the queue and handle the next requests.

If you have any idea that helps me debug and able to provide more information, it would be great. Thanks.

[Muaaz Farooq]

Try again by disabling battery optimization (^-^).

[Hieu Truong]

@m-muaaz-farooq Battery optimization is already off. I tried to create an API caller using HttpUrlConnection to check if the app has internet connection or not. But result is only Retrofit cannot call APIs while HttpUrlConnection still giving me API result.

coroutineScope.launch(Dispatchers.IO) {
            val url = URL("https://myip.directadmin.com/")

            val httpURLConnection = url.openConnection() as HttpURLConnection
            try {
                val responseCode = httpURLConnection.responseCode
                Timber.d("===> directAdmin: responseCode $responseCode")
                if (responseCode == HttpURLConnection.HTTP_OK) {
                    val response = httpURLConnection.inputStream.bufferedReader()
                        .use { it.readText() }  // defaults to UTF-8
                    Timber.d("===> directAdmin: response $response")
                } else {
                    Timber.e("===> directAdmin: err $responseCode")
                }
            } finally {
                httpURLConnection.disconnect()
            }

        }

I assume that somehow Retrofit stopped working after the application resume for a long time but not throwing any error, just sitting there and not giving me any hint.

[Jake Wharton]

Retrofit's only goal is to issue the formed request to OkHttp and await the response for processing. Once the request is sent to OkHttp, this library does not play any part in the actual execution of the HTTP request. We have no control over that layer of the stack–it's entirely a black box to Retrofit. Since you have traced the request down into OkHttp, that means that Retrofit cannot really do anything to assist. Or, put another way, if you issued raw HTTP requests directly to OkHttp you would experience the same behavior. Thus, there's no action for Retrofit to take here. We simply don't have any levers to pull to assist you, even in debugging, because the library isn't involved when the problem is occurring. You can try asking on the OkHttp discussions or StackOverflow with the 'okhttp' tag, but there's also not a whole lot of information to go on here so I suspect most people will have to guess (similar to the battery optimization suggestion).

[Hieu Truong]

I found the problem, so that I will share it here. I hope people who are searching for it will have an idea to fix your issue.

Backgrounds:

• Authentication Interceptor that handles a call request with HTTP Code 401. It will contain a refresh token right there.
• Call 5 API requests onResume() parallelly of an activity.

How to reproduce:

1. Put the app in the background and back in the foreground.
2. Assume that all five requests above will return 401 Unauthorized (Using Charles Proxy for example).
3. Single refresh token API will be enqueued to refresh token while the requests above await a new access token.

Result:
Refresh token API never be called to the server. All APIs inside the app are stopped requesting to the server.

Problem:

• When 401 Unauthorized is triggered. All 5 API requests will wait for single Refresh token API response.
• According to the OkHttp source code, all enqueued requests will be placed inside okhttp3.Dispatcher.kt -> readyAsyncCalls. They will only be processed when the number of requests with the same host is lower than 5.

  /**
   * The maximum number of requests for each host to execute concurrently. This limits requests by
   * the URL's host name. Note that concurrent requests to a single IP address may still exceed this
   * limit: multiple hostnames may share an IP address or be routed through the same HTTP proxy.
   *
   * If more than [maxRequestsPerHost] requests are in flight when this is invoked, those requests
   * will remain in flight.
   *
   * WebSocket connections to hosts **do not** count against this limit.
   */
  @get:Synchronized var maxRequestsPerHost = 5

Because 5 API requests are taking place inside the runningAsyncCalls list, so Refresh token API will be put in 6th in the queue.
The problem is here. 5 API requests wait for Refresh token but Refresh token won't be processed until one of the 5 requests above is completed.

Fix:
I searched a little bit and find out single OkHttpClient shared the queue. And it is recommended to use single OkHttpClient in whole app to reduce latency and save memory.

 * ## OkHttpClients Should Be Shared
 *
 * OkHttp performs best when you create a single `OkHttpClient` instance and reuse it for all of
 * your HTTP calls. This is because each client holds its own connection pool and thread pools.
 * Reusing connections and threads reduces latency and saves memory. Conversely, creating a client
 * for each request wastes resources on idle pools.
 *
 * Use `new OkHttpClient()` to create a shared instance with the default settings:
 *
 * ```
 * // The singleton HTTP client.
 * public final OkHttpClient client = new OkHttpClient();
 *

But if the refresh token uses the same pool with the rest, it will cause a problem above.

So my current solution is creating different OkHttpClient for Refresh token API only. Like an emergency lane on the road, separate it from others so it won't be blocked.

Although it is fixed for now, I wonder if there is another way to fix it without creating a new OkHttpClient, because the new OkHttpClient will always take memory, even if the case is rare. Like removing 401 requests from the queue so Refresh token API can process.

This is my AuthInterceptor:

override fun intercept(chain: Chain): Response {
        val request = chain.request()
        val originRequest = buildCustomRequest(chain.request().newBuilder())
        // Proceed the request then check if token is expired
        val response = chain.proceed(originRequest)

        when (response.code) {
            HttpURLConnection.HTTP_UNAVAILABLE -> {
                runBlocking { serviceStateManager.updateState(ServiceState.Unavailable) }
            }

            HttpURLConnection.HTTP_UNAUTHORIZED -> {
                return handleOnUnauthorized(chain, request, response)
            }
        }

        return response
    }

    private fun handleOnUnauthorized(
        chain: Chain,
        request: Request,
        response: Response,
    ): Response {
        // Do not try to refresh access token if current refresh token is unavailable
        val accessToken = authManager.getAccessToken()
        val refreshToken = runBlocking { authManager.getRefreshToken() }
        if (accessToken.isBlank() || refreshToken.isNullOrBlank()) {
            return response
        }

        val shouldRetryRequestWithNewToken = synchronized(this) {
            runBlocking { Mutex().withLock { refreshToken() } }
        }

        return if (shouldRetryRequestWithNewToken) {
            response.close()
            val newRequest = buildCustomRequest(request.newBuilder())
            chain.proceed(newRequest)
        } else {
            response
        }
    }