[Performance] Copy stream to RelayedHttpListenerContext.Response.OutputStream and RelayedHttpListenerContext.Response.CloseAsync() is slow on large data size.
geo-the-dude opened this issue · comments
Actual Behavior
- Copying a stream with 1.16 MB of data takes 3-5 seconds to complete.
- RelayedHttpListenerContext.Response.CloseAsync() also take 3-5 seconds to complete.
Expected Behavior
- Copying to Filestream is taking 40ms to complete. The performance of copying the same data to OutputStream should produce similar results.
- RelayedHttpListenerContext.Response.CloseAsync() should not take seconds to complete.
Versions
- OS platform and version: Windows
- .NET Version: 2.1
- NuGet package version or commit ID: 2.0.1
I'm attaching a test program that returns 1.5MB responses.
Program.zip
Running this program I get output like this:
[12:18:55.910] Opening HybridConnectionListener
[12:18:56.577] Listener is Online.
[12:18:57.115] Listener received request: GET sb://RELAYNS.servicebus.windows.net/HCNAME
[12:18:57.979] Response received after: 1375 ms (1572864 bytes)
[12:18:57.979] Elapsed: 1376 ms
[12:18:58.025] Listener received request: GET sb://RELAYNS.servicebus.windows.net/HCNAME
[12:18:58.695] Response received after: 714 ms (1572864 bytes)
[12:18:58.695] Elapsed: 714 ms
[12:18:58.718] Listener received request: GET sb://RELAYNS.servicebus.windows.net/HCNAME
[12:18:59.403] Response received after: 707 ms (1572864 bytes)
[12:18:59.403] Elapsed: 707 ms
[12:18:59.426] Listener received request: GET sb://RELAYNS.servicebus.windows.net/HCNAME
[12:19:00.090] Response received after: 686 ms (1572864 bytes)
[12:19:00.090] Elapsed: 686 ms
[12:19:00.113] Listener received request: GET sb://RELAYNS.servicebus.windows.net/HCNAME
[12:19:00.794] Response received after: 702 ms (1572864 bytes)
[12:19:00.794] Elapsed: 702 ms
[12:19:00.797] Closing HybridConnectionListener
[12:19:01.278] Listener is Offline.
[12:19:01.287] Listener Closed
If you run this do you see most requests taking 3+ seconds?
Hi @dlstucki, I ran the test program on our machine.
Below are the results:
[10:52:23.763] Opening HybridConnectionListener(TrackingId:089e5546-5c04-41fb-acf4-72ce1e6a4188, Address:sb://relay.servicebus.windows.net/0001)
[10:52:28.716] Listener is Online.
[10:52:29.418] Listener received request: GET sb://relay.servicebus.windows.net/0001
[10:52:30.433] Write to Output Stream elapsed time: 00:00:01.0142859
[10:52:31.234] Response received after: 2500 ms (1572864 bytes)
[10:52:31.235] Elapsed: 2500 ms
[10:52:31.265] Upload to Azure elapsed time: 00:00:00.8315831
[10:52:31.277] Listener received request: GET sb://relay.servicebus.windows.net/0001
[10:52:31.874] Write to Output Stream elapsed time: 00:00:00.5963129
[10:52:32.805] Response received after: 1569 ms (1572864 bytes)
[10:52:32.805] Elapsed: 1569 ms
[10:52:32.860] Upload to Azure elapsed time: 00:00:00.9857836
[10:52:32.864] Listener received request: GET sb://relay.servicebus.windows.net/0001
[10:52:33.477] Write to Output Stream elapsed time: 00:00:00.6135085
[10:52:34.589] Response received after: 1783 ms (1572864 bytes)
[10:52:34.589] Elapsed: 1784 ms
[10:52:34.638] Upload to Azure elapsed time: 00:00:01.1604920
[10:52:34.644] Listener received request: GET sb://relay.servicebus.windows.net/0001
[10:52:36.538] Write to Output Stream elapsed time: 00:00:01.8941643
[10:52:37.542] Response received after: 2952 ms (1572864 bytes)
[10:52:37.542] Elapsed: 2953 ms
[10:52:37.601] Upload to Azure elapsed time: 00:00:01.0627615
[10:52:37.605] Listener received request: GET sb://relay.servicebus.windows.net/0001
[10:52:38.307] Write to Output Stream elapsed time: 00:00:00.7012564
[10:52:39.431] Response received after: 1888 ms (1572864 bytes)
[10:52:39.431] Elapsed: 1888 ms
[10:52:39.433] Closing HybridConnectionListener(TrackingId:089e5546-5c04-41fb-acf4-72ce1e6a4188, Address:sb://relay.servicebus.windows.net/0001)
[10:52:39.488] Upload to Azure elapsed time: 00:00:01.1809807
[10:52:40.124] Listener is Offline.
[10:52:40.127] Listener Closed
I am getting an average of 2 seconds per request. Although, the write to stream seems to be fast, sending data to Azure Relay takes seconds to complete.
Are there settings on the portal that would aid the performance?
The following are the logs from our own relay implementation:
Below we are calling a Rest API on a locally hosted ASP. Net Server, then we are copying the http response stream to the Relay Listener Context stream.
It can be seen below that we are having bottlenecks on copying the stream to the Relay Listener Context and when the Relay Listener Context closes.
[13:14:26.737 INF] Performance: Forward to local server start
{}
[13:14:27.496 INF] Performance: Forward to local server end, elapsed time: 00:00:00.7570901
{}
[13:14:27.509 ERR] Reading from source.
{}
[13:14:27.509 ERR] Read 81920 bytes.
{}
[13:14:27.509 ERR] Writing to dest
{}
[13:14:29.011 ERR] Done writing.
{}
[13:14:29.011 ERR] Reading from source.
{}
[13:14:29.013 ERR] Read 81920 bytes.
{}
[13:14:29.013 ERR] Writing to dest
{}
[13:14:29.226 ERR] Done writing.
{}
[13:14:29.226 ERR] Reading from source.
{}
[13:14:29.227 ERR] Read 81920 bytes.
{}
[13:14:29.228 ERR] Writing to dest
{}
[13:14:29.487 ERR] Done writing.
{}
[13:14:29.487 ERR] Reading from source.
{}
[13:14:29.489 ERR] Read 81920 bytes.
{}
[13:14:29.489 ERR] Writing to dest
{}
[13:14:29.490 ERR] Done writing.
{}
[13:14:29.491 ERR] Reading from source.
{}
[13:14:29.491 ERR] Read 81920 bytes.
{}
[13:14:29.492 ERR] Writing to dest
{}
[13:14:29.493 ERR] Done writing.
{}
[13:14:29.493 ERR] Reading from source.
{}
[13:14:29.494 ERR] Read 81920 bytes.
{}
[13:14:29.494 ERR] Writing to dest
{}
[13:14:29.750 ERR] Done writing.
{}
[13:14:29.750 ERR] Reading from source.
{}
[13:14:29.751 ERR] Read 81920 bytes.
{}
[13:14:29.752 ERR] Writing to dest
{}
[13:14:29.753 ERR] Done writing.
{}
[13:14:29.754 ERR] Reading from source.
{}
[13:14:29.757 ERR] Read 81920 bytes.
{}
[13:14:29.758 ERR] Writing to dest
{}
[13:14:29.759 ERR] Done writing.
{}
[13:14:29.761 ERR] Reading from source.
{}
[13:14:29.762 ERR] Read 81920 bytes.
{}
[13:14:29.763 ERR] Writing to dest
{}
[13:14:29.764 ERR] Done writing.
{}
[13:14:29.765 ERR] Reading from source.
{}
[13:14:29.765 ERR] Read 81920 bytes.
{}
[13:14:29.765 ERR] Writing to dest
{}
[13:14:29.766 ERR] Done writing.
{}
[13:14:29.766 ERR] Reading from source.
{}
[13:14:29.767 ERR] Read 81920 bytes.
{}
[13:14:29.767 ERR] Writing to dest
{}
[13:14:29.771 ERR] Done writing.
{}
[13:14:29.772 ERR] Reading from source.
{}
[13:14:29.773 ERR] Read 81920 bytes.
{}
[13:14:29.773 ERR] Writing to dest
{}
[13:14:29.968 ERR] Done writing.
{}
[13:14:29.969 ERR] Reading from source.
{}
[13:14:29.970 ERR] Read 81920 bytes.
{}
[13:14:29.972 ERR] Writing to dest
{}
[13:14:30.184 ERR] Done writing.
{}
[13:14:30.185 ERR] Reading from source.
{}
[13:14:30.186 ERR] Read 81920 bytes.
{}
[13:14:30.187 ERR] Writing to dest
{}
[13:14:30.391 ERR] Done writing.
{}
[13:14:30.391 ERR] Reading from source.
{}
[13:14:30.395 ERR] Read 68595 bytes.
{}
[13:14:30.397 ERR] Writing to dest
{}
[13:14:30.399 ERR] Done writing.
{}
[13:14:30.399 ERR] Reading from source.
{}
[13:14:30.400 ERR] Done copying.
{}
[13:14:30.400 INF] Performance: Copy stream elapsed time: 00:00:02.8930151
{}
[13:14:33.405 INF] Performance: Send response to relay elapsed time: 00:00:03.0045135
We have discovered that the location of the Relay Namespace plays a big part on the performance.
We have noticed that the output stream have also decreased when we created a Relay Namespace location closer to our network.
Is the output stream a network stream?
Yes, the region the Relay Namespace is in is directly related to the performance you see. The closer the better.
The OutputStream is not literally a NetworkStream, but it does a lot of networking. You can find it in the ResponseStream class. The methods WriteAsync and FlushAsync contain most of the interesting logic surrounding how and when it will write over the network. The first 64KB written to the context.Response.OutputStream (aka ResponseStream) gets buffered in memory but once that threshold is passed each WriteAsync call doesn't return until those bytes have gone over the wire.
Here's a modified version of the demo program which sends sends the exact same number of chunks and total bytes as the trace output you just sent.
ProgramWithTimings.zip
When I run this ProgramWithTimings.zip I get output similar to this:
[10:21:53.511] Listener received request: GET sb://yourrelay.servicebus.windows.net/YOURHC
[10:21:53.659] Writing 81920 to OutputStream completed in 147 ms
[10:21:53.690] Writing 81920 to OutputStream completed in 29 ms
[10:21:53.693] Writing 81920 to OutputStream completed in 2 ms
[10:21:53.693] Writing 81920 to OutputStream completed in 0 ms
[10:21:53.722] Writing 81920 to OutputStream completed in 28 ms
[10:21:53.745] Writing 81920 to OutputStream completed in 23 ms
[10:21:53.746] Writing 81920 to OutputStream completed in 0 ms
[10:21:53.747] Writing 81920 to OutputStream completed in 0 ms
[10:21:53.748] Writing 81920 to OutputStream completed in 0 ms
[10:21:53.755] Writing 81920 to OutputStream completed in 6 ms
[10:21:53.773] Writing 81920 to OutputStream completed in 16 ms
[10:21:53.780] Writing 81920 to OutputStream completed in 6 ms
[10:21:53.796] Writing 81920 to OutputStream completed in 16 ms
[10:21:53.819] Writing 81920 to OutputStream completed in 22 ms
[10:21:53.825] Writing 68595 to OutputStream completed in 5 ms
[10:21:54.109] Response received after: 623 ms (1215475 bytes)
[10:21:54.112] Closing HybridConnectionListener(TrackingId:5dfcf69d-bb47-4198-80cd-111eed2d8125, Address:sb://yourrelay.servicebus.windows.net/YOURHC)
[10:21:54.135] Closing OutputStream completed in 310 ms
[10:21:54.675] Listener is Offline.
[10:21:54.679] Listener Closed
If you could take this demo program and modify it to show the massive delay on OutputStream.CloseAsync that would be very helpful. Also note, that the entire response can be received by the HTTP client before the OutputStream.CloseAsync actually completes (that CloseAsync is also closing a WebSocket connection which the HTTP client doesn't need to wait for).
You can use a tool such as tcping to check the TCP trip time to your Relay Namespace/Region.
tcping YOURRELAY.servicebus.windows.net 443
Probing 23.99.60.253:443/tcp - Port is open - time=21.328ms
Probing 23.99.60.253:443/tcp - Port is open - time=22.254ms
Probing 23.99.60.253:443/tcp - Port is open - time=21.562ms
Probing 23.99.60.253:443/tcp - Port is open - time=25.194msPing statistics for 23.99.60.253:443
4 probes sent.
4 successful, 0 failed.
Approximate trip times in milli-seconds:
Minimum = 21.328ms, Maximum = 25.194ms, Average = 22.584ms
The first 64KB written to the context.Response.OutputStream (aka ResponseStream) gets buffered in memory but once that threshold is passed each WriteAsync call doesn't return until those bytes have gone over the wire.
By over the wire, do you mean that the data is sent to Azure Relay and the context is waiting for an acknowledgement before sending another batch of data? If this is case, is there a way we can increase the network buffer size in order to keep sending data without waiting to get an acknowledgement from Azure?
Furthermore, if we are writing data over the network, why is there some latency on context.Response.CloseAsync()?
WebSocketBase only allows one inflight send at a time. Source code here. It may be possible that exposing HybridConnectionListener.ConnectionBufferSize may have some impact on the performance when sending many large chunks.
In general context.Response.CloseAsync may send data to over the network. That happens in the case where one or more small writes occur and the total size doesn't exceed 64KB. Currently that CloseAsync also waits to close the rendezvouns WebSocket connection if one was needed (typically for requests or responses over 64KB).
Your best bet would be modifying that sample application to demonstrate the issue you're seeing and/or gathering traces. You can gather lower level traces by downloading PerfView then running this command to gather tracing from Microsoft.Azure.Relay.dll while reproducing the issue PerfView.exe /OnlyProviders=*Microsoft-Azure-ServiceBus,*Microsoft-Azure-Relay collect.
Thanks @dlstucki! We are now getting good results on our performance testing. We have point the Relay Namespace in the same region as our local service and we did some optimization efforts on codebase. This was very helpful! I will now close this issue and I will just create another If we need technical assistance.