wvlet.airframe.http.HttpClientException
Atry opened this issue · comments
Describe the bug
Recently I found publishing fails around 20 minutes with a wvlet.airframe.http.HttpClientException, even when the timeout is 3 hours:
How to reproduce the issue
See build log at https://github.com/ThoughtWorksInc/bindable.scala/actions/runs/4004136273/jobs/6884891378
2023-01-25 20:21:05.853Z info [SonatypeClient] The close stage is in progress. Waiting for 15.00 sec. - (SonatypeClient.scala:164)
2023-01-25 20:21:20.978Z info [SonatypeClient] repositoryClosed: id:comthoughtworksbinding-2491 - (SonatypeClient.scala:384)
2023-01-25 20:21:20.979Z info [SonatypeClient] [close] Finished successfully - (SonatypeClient.scala:173)
2023-01-25 20:21:20.979Z info [SonatypeClient] Promoting staging repository [comthoughtworksbinding-2491] status:closed, profile:com.thoughtworks.binding(581d7a4338d65c) description: Scala CI 4004136273 183 6 2.13.4 - (SonatypeClient.scala:210)
2023-01-25 20:21:21.031Z warn [HttpClient] [1/100] Request to /service/local/staging/profiles/581d7a4338d65c/promote is failed: [500: Internal Server Error] Request failed: {"errors":[{"id":"*","msg":"Unhandled: Staging repository is already transitioning: comthoughtworksbinding-2491"}]}. Retry the request in 1.279 sec. - (HttpClient.scala:180)
2023-01-25 20:21:22.357Z warn [HttpClient] [2/100] Request to /service/local/staging/profiles/581d7a4338d65c/promote is failed: [500: Internal Server Error] Request failed: {"errors":[{"id":"*","msg":"Unhandled: Staging repository is already transitioning: comthoughtworksbinding-2491"}]}. Retry the request in 1.908 sec. - (HttpClient.scala:180)
2023-01-25 20:21:24.314Z warn [HttpClient] [3/100] Request to /service/local/staging/profiles/581d7a4338d65c/promote is failed: [500: Internal Server Error] Request failed: {"errors":[{"id":"*","msg":"Unhandled: Staging repository is already transitioning: comthoughtworksbinding-2491"}]}. Retry the request in 3.026 sec. - (HttpClient.scala:180)
2023-01-25 20:21:27.392Z warn [HttpClient] [4/100] Request to /service/local/staging/profiles/581d7a4338d65c/promote is failed: [500: Internal Server Error] Request failed: {"errors":[{"id":"*","msg":"Unhandled: Staging repository is already transitioning: comthoughtworksbinding-2491"}]}. Retry the request in 4.223 sec. - (HttpClient.scala:180)
2023-01-25 20:21:31.666Z info [CircuitBreaker] CircuitBreaker[default] is changed to OPEN - (CircuitBreaker.scala:73)
2023-01-25 20:21:31.666Z warn [HttpClient] [5/100] Request to /service/local/staging/profiles/581d7a4338d65c/promote is failed: [500: Internal Server Error] Request failed: {"errors":[{"id":"*","msg":"Unhandled: Staging repository is already transitioning: comthoughtworksbinding-2491"}]}. Retry the request in 1.300 sec. - (HttpClient.scala:180)
2023-01-25 20:21:32.969Z info [CircuitBreaker] CircuitBreaker[default] is changed to HALF_OPEN - (CircuitBreaker.scala:73)
2023-01-25 20:21:32.973Z error [Sonatype]
wvlet.airframe.http.HttpClientException
at wvlet.airframe.http.client.HttpClients$$anonfun$defaultHttpClientErrorHandler$1.applyOrElse(HttpClients.scala:262)
at wvlet.airframe.http.client.HttpClients$$anonfun$defaultHttpClientErrorHandler$1.applyOrElse(HttpClients.scala:252)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
at wvlet.airframe.http.client.SyncClient.send(HttpClients.scala:80)
at wvlet.airframe.http.client.SyncClient.send$(HttpClients.scala:69)
at wvlet.airframe.http.client.SyncClientImpl.send(HttpClients.scala:28)
at wvlet.airframe.http.client.SyncClient.callInternal(HttpClients.scala:115)
at wvlet.airframe.http.client.SyncClient.callInternal$(HttpClients.scala:108)
at wvlet.airframe.http.client.SyncClientImpl.callInternal(HttpClients.scala:28)
at xerial.sbt.sonatype.SonatypeClient.promoteStage(SonatypeClient.scala:213)
at xerial.sbt.sonatype.SonatypeService.promoteStage(SonatypeService.scala:184)
at xerial.sbt.sonatype.SonatypeService.closeAndPromote(SonatypeService.scala:[198](https://github.com/ThoughtWorksInc/bindable.scala/actions/runs/4004136273/jobs/6884891378#step:6:199))
at xerial.sbt.Sonatype$.$anonfun$sonatypeBundleRelease$2(Sonatype.scala:182)
at xerial.sbt.Sonatype$.withSonatypeService(Sonatype.scala:437)
at xerial.sbt.Sonatype$.$anonfun$sonatypeBundleRelease$1(Sonatype.scala:177)
at scala.Option.getOrElse(Option.scala:189)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at xsbt.boot.Launch$.$anonfun$run$1(Launch.scala:149)
at xsbt.boot.Launch$.withContextLoader(Launch.scala:176)
at xsbt.boot.Launch$.run(Launch.scala:149)
at xsbt.boot.Launch$.$anonfun$apply$1(Launch.scala:44)
at xsbt.boot.Launch$.launch(Launch.scala:159)
at xsbt.boot.Launch$.apply(Launch.scala:44)
at xsbt.boot.Launch$.apply(Launch.scala:21)
at xsbt.boot.Boot$.runImpl(Boot.scala:78)
at xsbt.boot.Boot$.run(Boot.scala:73)
at xsbt.boot.Boot$.main(Boot.scala:21)
at xsbt.boot.Boot.main(Boot.scala)
Caused by: wvlet.airframe.control.CircuitBreakerOpenException
at wvlet.airframe.control.CircuitBreaker$.$anonfun$throwOpenException$1(CircuitBreaker.scala:69)
at wvlet.airframe.control.CircuitBreaker.verifyConnection(CircuitBreaker.scala:272)
at wvlet.airframe.control.Retry$RetryContext.$anonfun$runInternal$1(Retry.scala:287)
at scala.util.Try$.apply(Try.scala:213)
at wvlet.airframe.control.Retry$RetryContext.runInternal(Retry.scala:286)
at wvlet.airframe.control.Retry$RetryContext.runWithContext(Retry.scala:259)
at wvlet.airframe.http.client.SyncClient.send(HttpClients.scala:74)
... 52 more - (Sonatype.scala:443)
- sbt version: 1.8.2
- sbt-sonatype version: 3.9.15
- Show your build settings: See https://github.com/ThoughtWorksInc/bindable.scala/blob/8ac54bf713459a636bdfdca1792ff07af0ecec00/.github/workflows/scala.yml#L48
- GitHub repository: https://github.com/ThoughtWorksInc/bindable.scala
Additional context
I expect sbt-sonatype should retry until reaching the 3-hour timeout
According to https://status.maven.org/, sonatype is unstable recently. Let's retry more times.
I've also hit this issue. I think sbt-sonatype should be fixed to wait for a longer time if the circuit breaker opens.
Currently, when a circuit breaker is open, it's marked as non-retryable for fail-fast behavior. https://github.com/wvlet/airframe/blob/master/airframe-http/src/main/scala/wvlet/airframe/http/HttpClientException.scala#L162
So, we need to wrap HTTP requests with double retry blocks when the Sonatype service is totally down for a while.
Unfortunately createStage can also fail due to connection reset. See https://github.com/ThoughtWorksInc/Binding.scala/actions/runs/4021048510/jobs/6909569228
https://github.com/ThoughtWorksInc/LatestEvent.scala/actions/runs/4021150036/jobs/6911289377
Still get the same error after upgrading to sbt-sonatype v3.9.16
There is another http client only for createStage with a long timeout configuration. We also need to disable the circuit breaker for it
But this exception is thrown from promoteStage: https://github.com/ThoughtWorksInc/LatestEvent.scala/actions/runs/4021150036/jobs/6911289377
ok. That's a bit of unexpected behavior 🤔
@Atry It seems the error message says the max retry count is 100, which means it's using an older version before your change or the latest version might not include the latest change.
[1/100] Request to /service/local/staging/profiles/581d7a4338d65c/promote is failed: [500: Internal Server Error] Request failed: {"errors":[{"id":"*","msg":"Unhandled: Staging repository is already transitioning: comthoughtworksbinding-2789"}]}. Retry the request in 1.410 sec.
Adding a version log to reduce the debug time #366
It seems 3.9.16 does not include #364
https://github.com/xerial/sbt-sonatype/commits/v3.9.16
Yeah. Now releasing 3.9.17