Single fire reminders firing multiple times
aarondandy opened this issue · comments
We register some reminders on actors that are firing multiple times even though we set the period to new TimeSpan(0, 0, 0, 0, -1) . We can reproduce this issue with the code and steps below.
Minimal reproduction: https://github.com/aarondandy/bug-sf-zombies
Specific reproduction steps: https://github.com/aarondandy/bug-sf-zombies/blob/master/readme.md
General reproduction steps:
- Register a one-time reminder for an actor
- Handle the reminder
- Restart or deactivate the primary node for the actor that handled the reminder
- Observe that the reminder is triggered again
Expected: The reminder should not trigger again.
The recommend way to use one-time reminder is to unregister it after it has fired once. Otherwise it will sit there adding to total size of local state of ActorService. If there are many such reminders, it will cause redundant network traffic and I/O when a new replica need to be built.
From functional perspective, this is a bug on our side. We have made a note of it and it will be fixed in future release.
Another scenario I observe in case of TimeSpan(0, 0, 0, 0, -1)
When exception is thrown in the handling of the reminder (regardless of its handling) - reminder will fire again.
That is the expected behavior. Unless the reminder callback completes successfully, ActorRuntime does not consider it as successful and it will fire again on new primary.
We had the same thing and I thought I ought to share our workaround--a small helper library called Prompter that wraps reminders to provide once-off reminder functionality (including removing the reminder from state).
await PromptOnce("wow, what a cool reminder", TimeSpan.FromDays(7));
Source and detailed instructions over 'ere.