emitAndWait() crashes if not "spaced"

Question

emitAndWait() crashes if not "spaced"

itsRems opened this issue 4 years ago · comments

I'm having an issue where if I await a few emitAndWait promises, it will crash. for example if I have three emitAndWait, the first will be fine but the second promise will never reply and the thing will eventually crash, throwing this error:

(node:14664) UnhandledPromiseRejectionWarning: undefined
(node:14664) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:14664) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
2020-05-19T04:02:23.445Z [RABBITMQ_EVENT_MANAGER] error: Unable to delete queue undefined
TypeError: Cannot read property 'deleteQueue' of undefined
    at Object.<anonymous> (D:\Dev\rocketplay\API\core\node_modules\rabbitmq-event-manager\src\adapter\helper\deleteQueue.ts:7:19)
    at Generator.next (<anonymous>)
    at D:\Dev\rocketplay\API\core\node_modules\rabbitmq-event-manager\build\adapter\helper\deleteQueue.js:7:71
    at new Promise (<anonymous>)
    at __awaiter (D:\Dev\rocketplay\API\core\node_modules\rabbitmq-event-manager\build\adapter\helper\deleteQueue.js:3:12)
    at Object.deleteQueue (D:\Dev\rocketplay\API\core\node_modules\rabbitmq-event-manager\build\adapter\helper\deleteQueue.js:14:12)
    at EventManager.<anonymous> (D:\Dev\rocketplay\API\core\node_modules\rabbitmq-event-manager\src\EventManager.class.ts:126:27)
    at Generator.next (<anonymous>)
    at D:\Dev\rocketplay\API\core\node_modules\rabbitmq-event-manager\build\EventManager.class.js:7:71
    at new Promise (<anonymous>)
    at __awaiter (D:\Dev\rocketplay\API\core\node_modules\rabbitmq-event-manager\build\EventManager.class.js:3:12)
    at Immediate._onImmediate (D:\Dev\rocketplay\API\core\node_modules\rabbitmq-event-manager\src\EventManager.class.ts:125:35)
    at processImmediate (internal/timers.js:456:21)
    at process.topLevelDomainCallback (domain.js:137:15)

Looks to me like something is wrong with the message queue or something, but I haven't done much digging.

If you need any more info please let me know
(excuse the lack of details, this is a 6am-this-bug-happened-too-many-times issue)

mimiz · Answer 1 · Tue May 19 2020 23:43:27 GMT+0800 (China Standard Time)

Hi,
I'll look at it as soon as possible, but I'm a bit busy at the moment.
Can you give me more context to reproduce this bug?

Regards

Rémi

Nicolas Theck · Answer 2 · Wed May 20 2020 01:38:19 GMT+0800 (China Standard Time)

Sure !
I've created an example app over here: https://github.com/itsRems/buggyMQ
It contains an API service that calls a dummy worker. One endpoint is perfectly fine, another crashes and the last one has my "fix".

mimiz · Answer 3 · Thu May 21 2020 15:42:47 GMT+0800 (China Standard Time)

Hi,
Here is what I did :

I cloned your repo
I npm i on both api and worker
I changed the default RABBITMQ Url to mine
I ran npm run dev on both projects
I ran curl http://localhost:3500 and then curl http://localhost:3500/crash or curl http://localhost:3500/fix and had the following results :

Last login: Thu May 21 09:23:39 on ttys003
➜  ~ curl http://localhost:3500
The number is: 2. <br> This worked ! Now trying hitting /crash or /fixed%                           
➜  ~ curl http://localhost:3500/crash
The number is: 20%                                                                                  
➜  ~ curl http://localhost:3500/fix
The number is: 45%

Here are the logs of the worker :

9
Mario
8
40
Mario
8
40

And the API logs :

{ _metas:
   { guid: 'a34585bc-524d-4bbb-a6cd-000d2e634f3c',
     name: 'GET_NAME.RESPONSE.63a1e396-3a31-439e-ac9a-408ce95f006b',
     application: 'worker',
     timestamp: 1590046441927,
     responseTo: 'GET_NAME',
     correlationId: '63a1e396-3a31-439e-ac9a-408ce95f006b' },
  name: 'Mario' }
8

I tried on 3 node.js versions : 8.10.0, 10.16.3, and 12.16.3.

And everything seems to work fine.

Can you give me some more details?

Nicolas Theck · Answer 4 · Thu May 21 2020 23:34:34 GMT+0800 (China Standard Time)

Found the issue. I'm running node 13.12.
Switched to 12.16.3 and everything works fine.
Closing for now, thanks for the help :)
PS: May I suggest putting a warning that problems occur above LTS ?

Nicolas Theck · Answer 5 · Fri May 22 2020 00:43:23 GMT+0800 (China Standard Time)

Turns out that after some more testing, the problem came back.
I played a bit with the package and I got everything to run super fine...
By removing this

await adapter.deleteQueue(channel, queueName);
await adapter.deleteExchange(channel, replyTo);

from the "emitAndWait" function. Now obviously this isn't a proper fix (since neither the queue or exchange are deleted) but at least my thing now works.
My guess ? The fact that we're using the same channel to emit/handle new messages as the channel we're currently deleting stuff from is causing issues.
I've got no reproducing method for now but it occurs the same way it did before: the usage of chained emitAndWait() calls makes it that the message is sent but its reply isn't caught.

Nicolas Theck · Answer 6 · Fri May 22 2020 02:40:36 GMT+0800 (China Standard Time)

I fixed the issue by only resolving the emitAndWait Promise after the cleanup and not before like originally implemented.
Please merge this request to fix this bug.

mimiz · Answer 7 · Fri May 22 2020 16:56:54 GMT+0800 (China Standard Time)

Hi,
Thanks for investigating, but I still can not reproduce the problem, I will try to find time to work on it ...
Also, I would like to have a reproducible case so we can create a test and fix it.
I will take a look at your pull request.
Thanks for sharing

Rémi

mimiz · Answer 8 · Fri May 22 2020 18:25:05 GMT+0800 (China Standard Time)

Hi,
After some more tests, I had the same Error you originally had.
It happens on the first "run", I mean the first time the event is emitted or listened.
I wrote this test :

it('Should be able to emit and wait for response two instances', async () => {
    /** given */
    const eventManager = new EventManager({
      url: process.env.AMQP_URL,
      application: 'dummy-test'

    })
    await eventManager.initialize(); // For creating the DLX and Alternate
    eventManager.on('MULTIPLY', (p) => {
      return { result: p.a * p.b };
    })
    /** when */
    const { result } = await eventManager.emitAndWait('MULTIPLY', { a: 4, b: 3 });

    /** then */
    expect(result).to.equal(12);

  })

And (after a while) I got the error mentioned, If I run the same test a second time, it works fine.
I reset all queues and exchanges on RabbitMQ, then I ran the test again, and I got the error, again.
If I "delay" between 'on' and emit It works fine ...

it('Should be able to emit and wait for response two instances', async () => {
    /** given */
    const eventManager = new EventManager({
      url: process.env.AMQP_URL,
      application: 'dummy-test'

    })
    await eventManager.initialize(); // For creating the DLX and Alternate
    eventManager.on('MULTIPLY', (p) => {
      return { result: p.a * p.b };
    })
    await pause(500); // A small delay
    /** when */
    const { result } = await eventManager.emitAndWait('MULTIPLY', { a: 4, b: 3 });

    /** then */
    expect(result).to.equal(12);

  })

If this explanation seems goo to you, I will try to find a solution, for example make eventManager.on a promise that resolves only all queues and exchanges are correctly bound.

Regards

Nicolas Theck · Answer 9 · Sat May 23 2020 02:16:45 GMT+0800 (China Standard Time)

Hi,
I'm still a bit confused on how exactly this problem occurs and how you managed to reproduce, though it's good to see that it does happen to others than me.
In my understanding of your explanation, this is in fact due to RabbitMQ exchanges & queues; what doesn't make sense to me is the fact that you encountered this issue with only one listener and one emitter.
Is there any way you could run this test with #16 and see how that goes ? As I've said before, I stopped encountering the issue after making sure we delete the queue and exchange before resolving the emitAndWait Promise and I'm curious to see if it fixes the problem and/or if it decreases performance by any means.

Many thanks for the help,
Nicolas.

PS: The only reason I'm trying to get you to merge my request is because I'm currently running my own modified version of the package inside my API but would like to switch back to the npm package so that I don't have to worry about updates etc.

mimiz · Answer 10 · Sat May 23 2020 17:20:46 GMT+0800 (China Standard Time)

Hi,
I did find a way to reproduce ... by looping on emitAndWait :

for (let i = 0; i < 100; i++) {
      const { result } = await eventManager.emitAndWait('MULTIPLY', { a: 4, b: i });
      console.log(`i : ${i} : ${result}`); // sometimes stop randomly ... 
      results[i] = result;
    }

Acrually, for a loop of 100; the code always crash, but randomly.
And your fix fixed the problem 😃
As I was working on some other updates (better integration of logs, update documentation, ...) I will add your fix in the update (1.2.0)

Thanks a lot for your help and time ;)

Regards

mimiz · Answer 11 · Sat May 23 2020 18:30:21 GMT+0800 (China Standard Time)

Fixed in version v1.2.0

Nicolas Theck · Answer 12 · Thu May 28 2020 11:15:42 GMT+0800 (China Standard Time)

Well oh well. It's happening again.
This week's bug is brought to you by... more events chaining !
I'm experiencing the same type of issues as we used to while trying to emit a few events, but this time I actually identified the exact problem, but I'll have to explain what I do with your package first.
I use this package to power the API for my website.
The API consists of module, all "connected" together with your rabbitmq event manager. The central module is the actual API; it serves as a web server and handles all of the event requests. When the request body is validated (and other things), it sends emitAndWait() requests to the server and then returns according on the service's response. This used to work just fine with our fix when I was just firing one http requests, however when firing multiple... same thing happens. Here's where I have a problem:

When sending a request to our watch page, the front-end sends three requests.
The first one is the refresh request - it sends the current token & device fingerprint to be handled by the user service to refresh the user's JWT. This request is cached for efficiency and trying to avoid a bug. This request works fine. What does not work is the second request in which we send a request to get the video's infos. This is cached and somewhat works in production (somehow...) but on local, the first request will just void, firing "unhandled promise rejection" errors caused (as identified) by... well I'm not sure. I'm guessing it's again due to the queue but, with my limited experience with RabbitMQ/amqp in general, I have no idea what the problem is. I'm also handling the promise rejection by re-sending the emitAndWait event. Sometimes it works, sometimes it doesn't. It sucks. Note that a third request is sent right after the front-end fetched the video info data, though that usually works just fine. Caching saves me for now but later I'll need for this not to be an issue later on. I'm hoping you can get something out of this example and am also ready to give you partial access to the code that crashes, or try to reproduce something that crashes in a similar way (yet again). Also, if you know any better tech stack, potentially even how to make real microservices under nodejs or simply how I could make my distributed API in a different, more efficient way, I'm all ears; I'm just in quite the love/hate relationship with this package as it's amazing but then... this happens. I'd love to keep using it for my project and hope to hear from you soon.

Thanks already,

Nicolas.

mimiz · Answer 13 · Fri May 29 2020 18:04:34 GMT+0800 (China Standard Time)

Hi,

Thanks again, for letting me know about the issue.
However, I will need a real case to investigate, I will not have much time to do it.
The thing is that I wrote the emitAndWait function to be used when you really don't have other choices, I mean, usage needs to be the exception.
Also, the emitAndWait function was made to create a kind of communication between one producer and only one listener (consumer) not many.
I will add this ionformation to the documentation.
RabbitMQ DOES not implement the emitAndWait when sending a message to RabbitMQ, the only thing you know is that message has been received, you don't even know that the message has been stored in a Queue, and more, you absolutely don't know if there is a consumer that consumes message in the queue.
So I think as you use "a lot" the emitAndWait, for me, it maybe means that RabbitMQ is not a good tool in your case.
If you can work on an example maybe I will try to take a look at your problem.
Regards

Rémi

Nicolas Theck · Answer 14 · Sat May 30 2020 05:26:20 GMT+0800 (China Standard Time)

Hi again.

Following your advice (and even prior to that), I've started looking for an alternative to your package (it's great but as we both saw, it's not meant for what I'm using it for).

I've now moved my entire API to bee-queue (used Redis) and, while the gateway rewrite was quite a lot of work, the workers didn't need much work, all thanks to the handling function syntax you use for this package.

Thanks for all the help with my problems and though I won't be using your package anymore, good luck with its development moving forward !

Regards,
Nicolas.