Receiving multiple notifications with WebHooks instead of one when a resource is created.

[Kush Bisen]

Environment

• Server version: 6.1.0
• Node.js version: v20.11.1
• npm version: 10.2.4

Description

As a resource is created once, the expected behaviour is for the HTTP server to receive the notification once, however I am receiving it more than once (for the same resource which is created).
It can be reproduced by sending the following JSON-LD data to http://localhost:3000/.notifications/WebhookChannel2023/

{
  "@context": [ "https://www.w3.org/ns/solid/notification/v1" ],
  "type": "http://www.w3.org/ns/solid/notifications#WebhookChannel2023",
  "topic": "http://localhost:3000/aggregation_pod/",
  "sendTo": "http://localhost:3001/"
}

The response to this was,

{
  "@context": [
    "https://www.w3.org/ns/solid/notification/v1"
  ],
  "id": "http://localhost:3000/.notifications/WebhookChannel2023/cba67232-cfdd-4fc2-a362-5eecb873d07b",
  "type": "http://www.w3.org/ns/solid/notifications#WebhookChannel2023",
  "topic": "http://localhost:3000/aggregation_pod/",
  "sendTo": "http://localhost:3001/",
  "endAt": "2024-03-08T13:08:02.355Z",
  "sender": "http://localhost:3000/.notifications/WebhookChannel2023/webId"
}

Now when creating a text/turtle document to http://localhost:3000/aggregation_pod/ using a POST request,I get the notification twice instead of once.

The requested URL is /
received notification {
  '@context': [
    'https://www.w3.org/ns/activitystreams',
    'https://www.w3.org/ns/solid/notification/v1'
  ],
  id: 'urn:1708693881163:http://localhost:3000/aggregation_pod/',
  type: 'Add',
  object: 'http://localhost:3000/aggregation_pod/b8de7676-001b-4d4e-b7fc-c582b3a3aeae',
  target: 'http://localhost:3000/aggregation_pod/',
  state: '"1708693881000-text/turtle"',
  published: '2024-02-23T13:11:21.163Z'
}
The requested URL is /
received notification {
  '@context': [
    'https://www.w3.org/ns/activitystreams',
    'https://www.w3.org/ns/solid/notification/v1'
  ],
  id: 'urn:1708693881172:http://localhost:3000/aggregation_pod/',
  type: 'Add',
  object: 'http://localhost:3000/aggregation_pod/b8de7676-001b-4d4e-b7fc-c582b3a3aeae',
  target: 'http://localhost:3000/aggregation_pod/',
  state: '"1708693881000-text/turtle"',
  published: '2024-02-23T13:11:21.172Z'
}

with some difference between the publishing time in the field published, which shows that the notification is being generated twice as well as the id field of the notifications are different for the same resource.

My HTTP server to process the notification looks like this, if required.

import * as http from 'http';

const port: number = 3001;

const server: http.Server = http.createServer((req: http.IncomingMessage, res: http.ServerResponse) => {
    console.log(`The requested URL is ${req.url}`); 
    if (req.method === 'POST') {
        let body: string = '';        
        req.on('data', (chunk: Buffer) => {
           body = chunk.toString();
        });
        req.on('end', () => {
            const webhookData: any = JSON.parse(body);
            console.log('received notification', webhookData);
        });
    }
    else {
        res.statusCode = 404;
        res.end('Invalid endpoint');
    }
});

server.listen(port, () => {
    console.log(`Server running at http://localhost:${port}/`);
    
});

[Joachim Van Herwegen]

I seem to be unable to reproduce this. Do you have an easy setup to reproduce?

For reference, I used your server example code (and added code to end the response in the end event), and added the following code to generate the notification:

async function run() {
  const subscription = {
    '@context': [ 'https://www.w3.org/ns/solid/notification/v1' ],
    type: 'http://www.w3.org/ns/solid/notifications#WebhookChannel2023',
    topic: 'http://localhost:3000/',
    sendTo: 'http://localhost:3001/',
  };
  let res = await fetch('http://localhost:3000/.notifications/WebhookChannel2023/', {
    method: 'POST',
    headers: { 'content-type': 'application/ld+json' },
    body: JSON.stringify(subscription),
  });
  console.log('Subscription response', res.status, await res.json());

  res = await fetch('http://localhost:3000/foo', {
    method: 'PUT',
    headers: { 'content-type': 'text/plain' },
    body: 'hello',
  });
  console.log('Create response', res.status);
}
setTimeout(run, 100);

I also tried with POST te create the resource and also only got 1 notification there. I tested with both the default and file-root configs.

[elf Pavlik]

Did you happen to double-check that you only subscribed once? Deleting the .internal/notifications directory could be a way to ensure that there are no webhook channels from some previous subscriptions.

[Joachim Van Herwegen]

Did you happen to double-check that you only subscribed once? Deleting the .internal/notifications directory could be a way to ensure that there are no webhook channels from some previous subscriptions.

If there are multiple subscriptions, those notifications should still have the same id, so something else seems wrong here.

[Kush Bisen]

I kept subscribing multiple times, (till 7) such that I got 7 notifications instead of one. On deleting the .internal/notifications, and resubscribing, I got only one subscription notification on the LDP resource. Thanks! @elf-pavlik
So, not sure if it is a bug anymore as they are expected to provide only one notification with the same id.

[Joachim Van Herwegen]

Maybe I was just wrong about the id's having to be the same 😅. I would have to check but I'll just assume I was wrong.

[Kush Bisen]

@joachimvh , FYI upon subscribing to for notifications twice. I endup with 3 different files in my .internal/notifications and I get two seperate notifications.
Two of the files have the same ID and one has a different ID. The files with the same ID, are different tho. Not sure how they are related but this is what I am getting incase it is something unexpected or expected.

File1

{"id":"http://localhost:3000/.notifications/WebhookChannel2023/151f4fb1-9746-4816-9de1-c90671514d64","type":"http://www.w3.org/ns/solid/notifications#WebhookChannel2023","topic":"http://localhost:3000/aggregation_pod/aggregation/1709893897109/","sendTo":"http://localhost:8085/","endAt":1711457742630}

Fil2:

{"id":"http://localhost:3000/.notifications/WebhookChannel2023/6d99b5bd-bd74-46d5-b2a0-1b38553f4300","type":"http://www.w3.org/ns/solid/notifications#WebhookChannel2023","topic":"http://localhost:3000/aggregation_pod/aggregation/1709893897109/","sendTo":"http://localhost:8085/","endAt":1711457728803}

File3:

["http://localhost:3000/.notifications/WebhookChannel2023/6d99b5bd-bd74-46d5-b2a0-1b38553f4300","http://localhost:3000/.notifications/WebhookChannel2023/151f4fb1-9746-4816-9de1-c90671514d64"]

[Joachim Van Herwegen]

The files are expected. The first two are your two subscriptions, and the last one collects all the subscriptions for a single resource.

[Kush Bisen]

I wonder if the same webook channel can be shared in case the sendTo and the topic field of a new (and further requests) are the same such that only one notification is registered instead of multiple?

[Joachim Van Herwegen]

Not with the current implementation no. Easiest way to get around this would be to probably inject a small cache component somewhere in the notification generation process.

From what I recall I think the current solution was chosen to potentially allow a different notification to be generated based on certain subscription settings should that ever be necessary.

[Kush Bisen]

Makes sense, incase anyone needs this functionality.
I would plug my own software here, https://github.com/SolidLabResearch/solid-stream-notifications-aggregator. It can work if the sendTo of different clients are same or not but the topic should be the same. Multiple clients can request the aggregator for the notification and they are registered to the pod (only once) and thus reduce the load on the pod sending the notifications to multiple clients. In initial experiments I was able to scale this up to 25k clients, with the pod sending the notification just once.

[elf Pavlik]

This is interesting, @argahsuknesib. Do you know if your aggregator also works with protected resources? If it does, I would like to know how it handles access control. I'm also gathering a list of software working with Solid Notifications for https://cxres.inrupt.net/public/SoSy24/RealTimeSolid/index.html. I'm going to include your aggregator as well!

[Kush Bisen]

Hello, @elf-pavlik ,
Not at the moment, I do not support access control for authorization to resources. Currently, I only plan on supporting the aggregator as a service which can be authenticated with the CSS's client credentials for an authenticated read/write session. Great to know! I will be coming to the real-time solid session and would be nice to see you.
You can also add my Solid Stream Aggregator into the list, where I aggregate over real-time sensor data written to the Solid Pod using the Solid Notifications protocol to read the latest events, as well as the historical data from the pod.