[SignalR] Performance issue: SendAsync randomly slow

Question

[SignalR] Performance issue: SendAsync randomly slow

stity opened this issue 2 years ago · comments

Valentin Demeusy commented 2 years ago

Is there an existing issue for this?

I have searched the existing issues

Describe the bug

Short version:
Sometimes await Clients.Caller.SendAsync() takes longer whereas the message is the same.

Long version:
I am trying to create a speed test using signalR and need to send the same binary message several times to measure the time it takes for the client to receive here.
But the time taken by the server to call await Clients.Caller.SendAsync() sometimes takes much longer.
Here the time I get when logging the following scenario 4 times :

Generate a random 10KB binary message
Send this message 10 times to the client

Time for sendAsync 206.1098
Time for sendAsync 0.8811000000000035
Time for sendAsync 0.3406999999999982
Time for sendAsync 0.12749999999999773
Time for sendAsync 0.13419999999999277
Time for sendAsync 0.11570000000000391
Time for sendAsync 0.08949999999998681
Time for sendAsync 1.1846000000000174
Time for sendAsync 0.12510000000000332
Time for sendAsync 0.1431999999999789
Time total for sendAsync 209.3248
-----------
Time for sendAsync 0.2945
Time for sendAsync 0.4445
Time for sendAsync 1.689
Time for sendAsync 0.2892000000000001
Time for sendAsync 14.490599999999999
Time for sendAsync 0.7215000000000025
Time for sendAsync 0.3940999999999981
Time for sendAsync 0.25230000000000175
Time for sendAsync 1.2722999999999978
Time for sendAsync 0.339500000000001
Time total for sendAsync 25.4123
-----------
Time for sendAsync 0.0961
Time for sendAsync 0.2046
Time for sendAsync 4.9528
Time for sendAsync 0.18710000000000004
Time for sendAsync 1.5126999999999997
Time for sendAsync 1.594500000000001
Time for sendAsync 0.7821999999999996
Time for sendAsync 1.2670999999999992
Time for sendAsync 1.2669000000000015
Time for sendAsync 1.0816
Time total for sendAsync 14.6355
-----------
Time for sendAsync 1.502
Time for sendAsync 0.5528000000000002
Time for sendAsync 16.1467
Time for sendAsync 1.8293999999999997
Time for sendAsync 12.267300000000002
Time for sendAsync 4.567999999999998
Time for sendAsync 9.352699999999999
Time for sendAsync 0.6216000000000008
Time for sendAsync 5.5057000000000045
Time for sendAsync 6.923199999999994
Time total for sendAsync 62.9212

Expected Behavior

I expect the time to call SendAsync to be more or less reproductible.

Steps To Reproduce

EDIT : remove code and add a link to a git repository instead
https://github.com/stity/signalr-speedtest-issue-demo

Exceptions (if any)

No response

.NET Version

6

Anything else?

IDE : VS 2022
Tested on local windows 10 computer
Browser: Chrome
SignalR protocol: MessagePack
output of dotnet --info:

.NET SDK (reflecting any global.json):
 Version:   6.0.301
 Commit:    43f9b18481

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19044
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.301\

Host (useful for support):
  Version: 6.0.6
  Commit:  7cca709db2

.NET SDKs installed:
  5.0.301 [C:\Program Files\dotnet\sdk]
  6.0.301 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Brennan · Answer 1 · Fri Jun 24 2022 02:43:35 GMT+0800 (China Standard Time)

Are you running the test with the client and server on different machines? If not that can affect your results.

Additionally, you could be hitting backpressure which may affect the speed you are seeing. You can try removing backpressure by setting ApplicationMaxBufferSize and TransportMaxBufferSize to 0 and retrying your test.
https://docs.microsoft.com/aspnet/core/signalr/configuration?view=aspnetcore-6.0&tabs=dotnet#advanced-http-configuration-options

Deleted user · Answer 2 · Fri Jun 24 2022 02:43:45 GMT+0800 (China Standard Time)

Hi @stity. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

David Fowler · Answer 3 · Fri Jun 24 2022 09:45:07 GMT+0800 (China Standard Time)

I think before you start tweaking things we'd like to understand the scenario more. What application logic are you trying to replicate?

Valentin Demeusy · Answer 4 · Fri Jun 24 2022 23:20:55 GMT+0800 (China Standard Time)

@BrennanConroy
I am indeed testing with the client and server on the same machine. How can this affect the result ?
I tried setting ApplicationMaxBufferSize and TransportMaxBufferSize to 0 but this didn't change anything.

@davidfowl
I am trying to implement a speed test to estimate the download speed between server and client (in order to later adapt the frequency of the messages).
So, the idea is to send several messages of a known size and to measure the time it takes for the client to receive them. When the client has received all the messages, he responds with a message that should stop the timer and give the server an estimate of the download speed.
But I get inconsistent results since the time to call SendAsync is often too long and exceed the time it takes for the client to receive a message.
I had a previous implementation using directly the websocket without SignalR and it was working well and I hoped I could migrate to SignalR to benefit other features (such as automatic reconnection, serialization/deserialization of the messages, ...)

David Fowler · Answer 5 · Sat Jun 25 2022 06:35:18 GMT+0800 (China Standard Time)

Can you send us your setup so we can attempt to replicate your results on our infrastructure?

Valentin Demeusy · Answer 6 · Mon Jun 27 2022 23:01:45 GMT+0800 (China Standard Time)

@davidfowl Here is a repository with the minimal code to reproduce the issue.
https://github.com/stity/signalr-speedtest-issue-demo

I successfully reproduced the issue in this repository with the following setup:

VS 2022 (17.2.5)
Windows 10
Google Chrome (also tested with Firefox)
Server and client on the same machine

Also, I noticed that I if I remove the await Task.Delay(5_000) line (line 48) in the SpeedTestHub class, the issue seems to disappear even though it should not impact the time I measure since it it outside of the block I am measuring. So it might be worth looking into that direction.

Thank you in advance for looking into this.

David Fowler · Answer 7 · Tue Jun 28 2022 22:15:53 GMT+0800 (China Standard Time)

Wait, did you originally have a Task.Delay?

Valentin Demeusy · Answer 8 · Tue Jun 28 2022 22:22:53 GMT+0800 (China Standard Time)

@davidfowl I did

David Fowler · Answer 9 · Tue Jun 28 2022 22:29:38 GMT+0800 (China Standard Time)

OK so I'm a little confused. Are you trying to see why there's random slow downs with an explicit Task.Delay? Or was that to simulate a delay? If there's no issue without the delay, then what is the problem?

Valentin Demeusy · Answer 10 · Tue Jun 28 2022 22:52:11 GMT+0800 (China Standard Time)

The slowdown that I see happens once the call to Task.Delay is already finished.
I start measuring the time, only after the Task.Delay:

 public async Task StartSpeedTest()
        {
            var randomBuffer = GetRandomImage(10_000);
            // commenting the following line seems to make the bug disappear or at least very hard to reproduce.
            await Task.Delay(5_000);
            var clock = Stopwatch.StartNew();
            double previousSendTime = 0;
            var count = 10;
            for (var i = 0; i < count; i++)
            {
                await Clients.Caller.SendAsync("Log", randomBuffer);
                var currentTime = clock.Elapsed.TotalMilliseconds;
                Console.WriteLine($"Time for sendAsync {currentTime - previousSendTime}");
                previousSendTime = currentTime;
            }
            var time = clock.Elapsed.TotalMilliseconds;
            Console.WriteLine($"Time total for sendAsync {clock.Elapsed.TotalMilliseconds}");
        }

David Fowler · Answer 11 · Wed Jun 29 2022 20:48:45 GMT+0800 (China Standard Time)

We're unable to reproduce your results (maybe its a hardware difference?)

@stity Can you look at the event counters and see what it looks like while you are reproducing this issue? Maybe something will stand out.

Valentin Demeusy · Answer 12 · Fri Jul 01 2022 21:31:53 GMT+0800 (China Standard Time)

I tried with the event counters but saw no difference between the normal calls and the slowed down ones.
However, I noticed that when executing the project with dotnet run instead of using visual studio the issue disappeared.

Brennan · Answer 13 · Thu Jul 07 2022 23:57:38 GMT+0800 (China Standard Time)

Great, glad you figured it out.

Deleted user · Answer 14 · Sat Jul 09 2022 00:01:18 GMT+0800 (China Standard Time)

This issue has been resolved and has not had any activity for 1 day. It will be closed for housekeeping purposes.

See our Issue Management Policies for more information.