OVP 2.22.0: recordingStatusChanged webhook event with status [ready] gets triggerred multiple times intermittently which causes erroneous behaviour in our application
mallikarjungurav opened this issue · comments
Describe the bug
For single vide file [recordingStatusChanged] webhook event with status [ready] gets trigger on provided service endpoint (service API endpoint) multiple times intermittently. We process video file once we receive this webhook. The problem is this webhook gets trigger mulitiple times sometimes twice, 5 times etc.
Expected behavior
We expect for a video file once recording is in ready status, we should receive this webhook only once as that of OVP version 2.17.0
Wrong current behavior
Once recording is ready from OVP, the webhook [recordingStatusChanged] with status [ready] gets trigger mulitiple times. sometimes twice, 5 times etc.
OpenVidu tutorial where to replicate the error
This is an EXTREMELY IMPORTANT STEP. If we are able to replicate the error in any of the official OpenVidu Tutorials or OpenVidu Demos, then we will be able to quickly fix it. If you are getting the error in your own application, please try to add the necessary changes to the most similar tutorial so it fails with the same error (try to keep those changes as contained as possible, so that the original tutorial maintains its integrity). Once you have an application to replicate the error, explain in detail the steps to get it like this:
- Clone repository [...]
- Compile the application like this [...]
- Run OpenVidu Server like this [...]
- Run the application like this [...]
- Join 1 user... Publish a video stream [...]
- See error
OpenVidu deployment info
How is your OpenVidu Server instance deployed when you get the bug. A couple of possible examples are listed below:
- Docker container as explained in OpenVidu tutorials, run with command
docker run ...on macOS Catalina 10.15.1 - AWS deployment as explained in OpenVidu Docs (https://docs.openvidu.io/en/stable/deployment/deploying-aws/)
IMPORTANT NOTE: please, if you think the bug might be related to OpenVidu Server side, specify here if you are also getting the error by using OpenVidu Server Demos instance. This instance is publicly available (use it only for this test, because it is not secure!!!): URL:
https://demos.openvidu.io:4443, SECRET:MY_SECRET
Client device info (if applicable)
Describe the client device(s) or platform(s) where you are able to replicate the error. For example:
- Chrome 78.0.3904.97 (Official Build) (64-bit) on Windows 10 (1903).
- Firefox Mobile 68.2.0 running on OnePlus 6 with Android 9.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
I think the information provided is enough to understand and replicate the issue highlighted.
Hi,
I am afraid we need more details on how to replicate the problem. Without that, we will not be able to fix it. Starting and stopping a recording on OpenVidu > 2.17.0 in a regular deployment only triggers event recordingStatusChanged once. It must be something else regarding your specific setup or usage of the API. Please, consider being more specific on how to replicate the issue. If you are able to get it using the OpenVidu development container, that would be even better.
Hello @mallikarjungurav , I've researched your issue and I think I found a possible reason for your duplicated webhook responses.
The key here is your phrase:
The problem is this webhook gets trigger mulitiple times sometimes twice, 5 times
WebHook retries was a feature added after 2.17.0 to make the WebHookSender more resilient to unexpected situations.
The class which implements the WebHook HTTP sender is this: HttpWebhookSender
The HTTP Header is implemented in such a way that, if an IOException happen, the webhook is resent to the client:
Those retries only happen under an IOException, so this basically means, that the server you have which is receiving those HTTP Webhook events is not responding correctly to OpenVidu, the HTTP Connection is closed, or some exception is happening which makes OpenVidu "believe" that the request is not sent correctly.
Please verify that the server which is receiving those webhooks responds correctly to OpenVidu webhook events.
Once we receive webhook we process it and send 200 ok response, irrespective of whether we get any error while processing at out service end. Even if we get any exception we catch it but we send 200 o response back.
Can you please specify more details as to why only this [ready] event being triger multiple times. We are not getting other webhooks more than once.
See below code:
try
{
openViduConnectionService.handleOpenViduEvent(uriInfo, webhookEvent);
}
catch (Exception e)
{
logger.error("WebhookEventError:: Error in processing webhook event [{}] with exception [{}]", webhookEvent, e);
}
return Response.status(Response.Status.OK).build();
Check logs from OpenVidu, webhooks are resent if a failure happens.
Is the handle made asynchronously? Take for example the call to the method openViduConnectionService.handleOpenViduEvent(uriInfo, webhookEvent); takes too much time. If the method waits to the handling of the event before sending the 200 OK, OpenVidu will think an error occurred because of the time it takes to receive it.
I recommend you to return the 200 OK before processing the event, or execute openViduConnectionService.handleOpenViduEvent(uriInfo, webhookEvent); asynchronously.
Where to check whether it is asynchronous?
It is your code @mallikarjungurav
In general words @mallikarjungurav, If it takes too much time to return the 200 OK, it will fail and OpenVidu will resend the webhook event. You need to send that 200OK before processing the event.
If your code is synchronous and you do not want to change too much of your code base, a way to avoid processing repeated webhooks is to have a Concurrent Set to store there which recordings are you processing and avoid handle it if it is being processed. Something like this:
private Set<String> processingRecordingSet = ConcurrentHashMap.newKeySet();
...
public void handleOpenViduEvent(uriInfo, webhookEvent) {
...
if (webhookEvent.getType().equals("recordingStatusChanged")) {
String recordingId = webhookEvent.getJson().get("id");
if (this.processingRecordingSet.add(recordingId)) {
// Do your processing here
logger.info("Processing recording [{}]", recordingId);
doing.stuff()
more.stuff()
stuff()
this.processingRecordingSet.remove(recordingId);
} else {
logger.info("Repeated webhook event. Recording [{}] is being processed", recordingId)
}
}
...
}This is an example, I don't know how your classes and methods are in your code, but I hope you get the idea.
Did you try our proposed solutions?
I can't use above solution since we have loadbalancer in place so I can't rely on in-memory storage.
Infact I have executed the code which handles the webhook in asynchronous way.
And I think think that solves the problem since I am giving response back to OVP once I receive the webhook. Currently we are in testing phase, I will let you know if I find any issues there. Thanks for your help.