Actor proxy reuses HTTP request when performing a retry to call an actor

[Simon Headley]

Expected Behavior

When a failure occurs during actor processing and a retry policy is in place, the actor proxy should generate a new HTTP request for each retry.

Actual Behavior

At the moment, the entire HTTP request is reused. This results in an exception with the following stack trace:

---> Dapr.Actors.ActorInvokeException: The request message was already sent. Cannot send the same request message multiple times.
--- End of inner exception stack trace ---
at Dapr.Actors.DaprHttpInteractor.InvokeActorMethodWithRemotingAsync(ActorMessageSerializersManager serializersManager, IActorRequestMessage remotingRequestRequestMessage, CancellationToken cancellationToken)
at Dapr.Actors.Communication.Client.ActorRemotingClient.InvokeAsync(IActorRequestMessage remotingRequestMessage, CancellationToken cancellationToken)
at Dapr.Actors.Client.ActorProxy.InvokeMethodAsync(Int32 interfaceId, Int32 methodId, String methodName, IActorRequestMessageBody requestMsgBodyValue, CancellationToken cancellationToken)

Steps to Reproduce the Problem

1. Build an API controller and bind it to a Pub/Sub topic of your choice that has a try/catch around the controller method (or global exception handling middleware).
2. In your controller, resolve a proxy to an actor of some kind, and pass the contents of the CloudEvent to your actor.
3. Build your actor to call some HTTP endpoint that will return a 400 of any kind.
4. Observe that when the 400 is returned to the actor, the Dapr actor exception swallows the 400 and returns a doubly-wrapped actor exception which does not contain the original error from the HTTP endpoint as the inner-most exception.
5. In your controller/middleware, when the actor exception is thrown, you are unable deduce the original exception due to the nature of Dapr actor exceptions, so you return a 500.
6. Dapr Pub/Sub resiliency will trigger a re-queue/re-process of the event, rather than nack'ing the event off the queue.
7. You will now observe an exception being thrown in the Dapr communications SDK stating that the HTTP request has already been sent as per the stack trace above.

Release Note

RELEASE NOTE:

[Simon Headley]

Bumping thread to keep it alive.