http-max-request-size is not applied to actor response body size

[Darragh Jones]

In what area(s)?

/area runtime

What version of Dapr?

1.13.1

Expected Behavior

An actor should be able to return a response body with a payload up to the size specified by the http-max-request-size argument.

Actual Behavior

An error occurs when an actor response body is > 4 MB irregardless of the value of the http-max-request-size argument.

Note that in my testing, the issue is only with an actor response body....not when sending a large request body.

Steps to Reproduce the Problem

To reproduce the problem, I've cloned the AspireWithDapr project and added an WeatherActor which returns ~ 8 MB of weather forecasts. I've also configured the normal API to also return 8 MB.

I've added the --dapr-http-max-request-size 16 parameter to the dapr sidecar, and I can see it is applied in the sidecar logs,

time="2024-04-03T20:23:06.1058837+01:00" level=info msg="Enabled max body size HTTP middleware with size 16 MB" app_id=web instance=DESKTOP-SE6C36R scope=dapr.runtime.http type=log ver=1.13.1

What I'm seeing is that both the service invocation and actor return an error when the --dapr-http-max-request-size is not specified.

However, when it is set to 16, the service invocation endpoint succeeds, but the actor endpoint returns this error:

time="2024-04-03T20:28:33.8753374+01:00" level=debug msg="{ERR_ACTOR_INVOKE_METHOD error invoke actor method: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (7902254 vs. 4194304)}" app_id=web instance=DESKTOP-SE6C36R scope=dapr.runtime.http type=log ver=1.13.1

I also added a method to the actor to accept an array of weather forecasts, and it seems I can send more than 4 MB request to the actor. So the issue is only when returning a response.

Release Note

RELEASE NOTE:

http-max-request-size argument now applies to actor response body message size.

[Yaron Schneider]

What SDK are you using to invoke the actor?

[Darragh Jones]

I'm using the .NET SDK to both invoke and host the actor. But as noted above, the error is logged in the sidecar, which suggested to me it was not an SDK issue.

[Darragh Jones]

Just to add some more information...I've tested the scenario again, this time just using curl.

When I call the actor via it's sidecar, the response is successful.

C:\dapr>curl -vk http://localhost:57403/v1.0/actors/WeatherActor/1/method/GetWeatherAsync > NUL
*   Trying 127.0.0.1:57403...
* Connected to localhost (127.0.0.1) port 57403 (#0)
> GET /v1.0/actors/WeatherActor/1/method/GetWeatherAsync HTTP/1.1
> Host: localhost:57403
> User-Agent: curl/7.84.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Server: Kestrel
< Traceparent: 00-ce578c6cb3748a2ebddad35ab1169a2b-fe05d3fb9f0dfba9-01
< Date: Wed, 10 Apr 2024 12:04:15 GMT
< Transfer-Encoding: chunked
<
* Connection #0 to host localhost left intact

However, when I call the actor, via the 'client' sidecar, I get the error:

C:\dapr>curl -vk http://localhost:57407/v1.0/actors/WeatherActor/1/method/GetWeatherAsync
*   Trying 127.0.0.1:57407...
* Connected to localhost (127.0.0.1) port 57407 (#0)
> GET /v1.0/actors/WeatherActor/1/method/GetWeatherAsync HTTP/1.1
> Host: localhost:57407
> User-Agent: curl/7.84.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< Content-Type: application/json
< Traceparent: 00-65c671216607b4039e0f213cd8a4d7f3-b731143ebd15e7b9-01
< Date: Wed, 10 Apr 2024 12:04:22 GMT
< Content-Length: 182
<
{"errorCode":"ERR_ACTOR_INVOKE_METHOD","message":"error invoke actor method: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (7902458 vs. 4194304)"}* Connection #0 to host localhost left intact

C:\dapr>

but again, I do see this in the client's sidecar logs:

time="2024-04-10T12:58:56.9051351+01:00" level=info msg="The request body size parameter is: 16" app_id=web instance=DESKTOP-SE6C36R scope=dapr.runtime type=log ver=1.13.1

Hopefully this helps identify the issue

[Yaron Schneider]

Just to add some more information...I've tested the scenario again, this time just using curl.

When I call the actor via it's sidecar, the response is successful.

C:\dapr>curl -vk http://localhost:57403/v1.0/actors/WeatherActor/1/method/GetWeatherAsync > NUL
*   Trying 127.0.0.1:57403...
* Connected to localhost (127.0.0.1) port 57403 (#0)
> GET /v1.0/actors/WeatherActor/1/method/GetWeatherAsync HTTP/1.1
> Host: localhost:57403
> User-Agent: curl/7.84.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Server: Kestrel
< Traceparent: 00-ce578c6cb3748a2ebddad35ab1169a2b-fe05d3fb9f0dfba9-01
< Date: Wed, 10 Apr 2024 12:04:15 GMT
< Transfer-Encoding: chunked
<
* Connection #0 to host localhost left intact

However, when I call the actor, via the 'client' sidecar, I get the error:

C:\dapr>curl -vk http://localhost:57407/v1.0/actors/WeatherActor/1/method/GetWeatherAsync
*   Trying 127.0.0.1:57407...
* Connected to localhost (127.0.0.1) port 57407 (#0)
> GET /v1.0/actors/WeatherActor/1/method/GetWeatherAsync HTTP/1.1
> Host: localhost:57407
> User-Agent: curl/7.84.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< Content-Type: application/json
< Traceparent: 00-65c671216607b4039e0f213cd8a4d7f3-b731143ebd15e7b9-01
< Date: Wed, 10 Apr 2024 12:04:22 GMT
< Content-Length: 182
<
{"errorCode":"ERR_ACTOR_INVOKE_METHOD","message":"error invoke actor method: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (7902458 vs. 4194304)"}* Connection #0 to host localhost left intact

C:\dapr>

but again, I do see this in the client's sidecar logs:

time="2024-04-10T12:58:56.9051351+01:00" level=info msg="The request body size parameter is: 16" app_id=web instance=DESKTOP-SE6C36R scope=dapr.runtime type=log ver=1.13.1

Hopefully this helps identify the issue

That does help a lot, thanks