increasing timeout above 100s still ends request after 100s

[Springy]

I'm using IFlurlRequest.WithTimeout(TimeSpan.FromMinutes(5)) (followed by .PostAsync().ReceiveJson<bool>()) to call a long running api endpoint (both client and server using NET7).

But after 100s the client gives up:

System.OperationCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
 ---> Flurl.Http.FlurlHttpTimeoutException: Call timed out: POST https://hostwithlongrunningoperation
 ---> System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
 ---> System.TimeoutException: The operation was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
 ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   --- End of inner exception stack trace ---
   at Flurl.Http.FlurlRequest.HandleExceptionAsync(FlurlCall call, Exception ex, CancellationToken token)
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   at Flurl.Http.ResponseExtensions.ReceiveJson[T](Task`1 response)
   --- End of inner exception stack trace ---

At server side I can see that ControllerBase.HttpContext.RequestAborted (of type CancellationToken) transissions to cancelled state at same time.

What could I have done wrong? (I only found similar issues with shorter timeouts than 100s so far)

[Todd Menier]

In your app, are you providing your own HttpClient, either by passing it to the FlurlClient constructor, or by implementing a custom factory? If so, you'll need to set a greater value on HttpClient.Timeout, which defaults to 100s. Flurl takes care of it for you when it's in control of HttpClient construction, but you'll need to do it yourself when you're providing your own.

[Springy]

@tmenier thanks for the hint, I found a call to IFlurlRequest.WithClient(new FlurlClient(precreatedHttpClient)) and now added an inifinite timeout to precreatedHttpClient (not tested yet, but I guess it does the trick).

Maybe you could introduce a public static method to prepare a given HttpClient to get "flurl compatible"? (currently just setting Timeout to infinite, would be called from DefaultHttpClientFactory.CreateHttpClient as well)

[Todd Menier]

Maybe, but there are bigger gotchas than this when providing your own client. Things like disabling native CookieContainer support and auto-redirects, necessary for Flurl to take over those things. And they can't be done after a client is already created. So I think this all comes down to better documentation.

[Springy]

Yeah, I already noticed DefaultHttpClientFactory does another things I wanted to adopt, so I'm calling now CreateMessageHandler() but have to cast HttpMessageHandler to HttpClientHandler (somehow unsafe because relying on your internal implementation) in order to change properties introduced there.

So my suggestion how DefaultHttpClientFactory could look like:

	public class DefaultHttpClientFactory : IHttpClientFactory
	{
		public static HttpClientHandler CreateDefaultMessageHandler()
		{
			var httpClientHandler = new HttpClientHandler();

			// flurl has its own mechanisms for managing cookies and redirects

			try { httpClientHandler.UseCookies = false; }
			catch ( PlatformNotSupportedException ) { } // look out for WASM platforms (#543)

			if ( httpClientHandler.SupportsRedirectConfiguration )
				httpClientHandler.AllowAutoRedirect = false;

			if ( httpClientHandler.SupportsAutomaticDecompression )
			{
				// #266
				// deflate not working? see #474
				httpClientHandler.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
			}
			return httpClientHandler;
		}

		public static void PrepareHttpClient(HttpClient client)
		{
			// Timeouts handled per request via FlurlHttpSettings.Timeout
			client.Timeout = System.Threading.Timeout.InfiniteTimeSpan;
		}

		/// <summary>
		/// Override in custom factory to customize the creation of HttpClient used in all Flurl HTTP calls.
		/// In order not to lose Flurl.Http functionality, it is recommended to call base.CreateClient and
		/// customize the result.
		/// </summary>
		public virtual HttpClient CreateHttpClient(HttpMessageHandler handler)
		{
			var client = new HttpClient(handler);
			PrepareHttpClient(client);
			return client;
		}

		/// <summary>
		/// Override in custom factory to customize the creation of HttpClientHandler used in all Flurl HTTP calls.
		/// In order not to lose Flurl.Http functionality, it is recommended to call CreateDefaultMessageHandler and
		/// customize the result.
		/// </summary>
		public virtual HttpMessageHandler CreateMessageHandler() => CreateDefaultMessageHandler();
	}