[RabbitMQ] AmqpConnection.request() unhandled exception when losing connection
cs-salim opened this issue · comments
I have nest microservices making rpc requests using AmqpConnection.request(), which crashes the whole microservice if the connection to RMQ is down at that moment.
Some errors are handled correctly but one is not, and it's just not possible to catch and handle it on my end:
These errors are handled, no problem here:
[my-microservice] [my-microservice] Error 2024-01-26T10:34:18-05:00 [AmqpConnection] Disconnected from RabbitMQ broker (default) - {
[my-microservice] stack: [
[my-microservice] 'Error: Heartbeat timeout\n' +
[my-microservice] ' at Heart.<anonymous> (/Users/me/repos/my-repo/node_modules/amqplib/lib/connection.js:427:19)\n' +
[my-microservice] ' at Heart.emit (node:events:513:28)\n' +
[my-microservice] ' at Heart.runHeartbeat (/Users/me/repos/my-repo/node_modules/amqplib/lib/heartbeat.js:88:17)\n' +
[my-microservice] ' at listOnTimeout (node:internal/timers:559:17)\n' +
[my-microservice] ' at processTimers (node:internal/timers:502:7)'
[my-microservice] ]
[my-microservice] } +12s
[my-microservice] [my-microservice] Error 2024-01-26T10:34:18-05:00 [ExceptionsHandler] channel closed - {
[my-microservice] stack: [
[my-microservice] 'Error: channel closed\n' +
[my-microservice] ' at ConfirmChannel.<anonymous> (/Users/me/repos/my-repo/node_modules/amqplib/lib/channel.js:39:18)\n' +
[my-microservice] ' at ConfirmChannel.emit (node:events:525:35)\n' +
[my-microservice] ' at ConfirmChannel.C.toClosed (/Users/me/repos/my-repo/node_modules/amqplib/lib/channel.js:175:8)\n' +
[my-microservice] ' at Connection.C._closeChannels (/Users/me/repos/my-repo/node_modules/amqplib/lib/connection.js:394:18)\n' +
[my-microservice] ' at Connection.C.toClosed (/Users/me/repos/my-repo/node_modules/amqplib/lib/connection.js:401:8)\n' +
[my-microservice] ' at Heart.<anonymous> (/Users/me/repos/my-repo/node_modules/amqplib/lib/connection.js:430:12)\n' +
[my-microservice] ' at Heart.emit (node:events:513:28)\n' +
[my-microservice] ' at Heart.runHeartbeat (/Users/me/repos/my-repo/node_modules/amqplib/lib/heartbeat.js:88:17)\n' +
[my-microservice] ' at listOnTimeout (node:internal/timers:559:17)\n' +
[my-microservice] ' at processTimers (node:internal/timers:502:7)'
[my-microservice] ]
[my-microservice] } +2ms
I'm then making a request:
[my-microservice] [my-microservice] Error 2024-01-26T10:34:18-05:00 [ExceptionsHandler] AMQP connection is not available - {
[my-microservice] stack: [
[my-microservice] 'Error: AMQP connection is not available\n' +
[my-microservice] ' at AmqpConnection.publish (/Users/me/repos/my-repo/node_modules/@golevelup/nestjs-rabbitmq/src/amqp/connection.ts:598:13)\n' +
[my-microservice] ' at AmqpConnection.request (/Users/me/repos/my-repo/node_modules/@golevelup/nestjs-rabbitmq/src/amqp/connection.ts:382:16)\n' +
[my-microservice] ' at RmqRequestService.request (/Users/me/repos/my-repo/dist/apps/my-microservice/main.js:3343:52)\n' +
[my-microservice] ' at AuthGuardHttp.authenticateAuthorizeToken (/Users/me/repos/my-repo/dist/apps/my-microservice/main.js:3181:49)\n' +
[my-microservice] ' at AuthGuardHttp.canActivate (/Users/me/repos/my-repo/dist/apps/my-microservice/main.js:3085:39)\n' +
[my-microservice] ' at GuardsConsumer.tryActivate (/Users/me/repos/my-repo/node_modules/@nestjs/core/guards/guards-consumer.js:15:34)\n' +
[my-microservice] ' at canActivateFn (/Users/me/repos/my-repo/node_modules/@nestjs/core/router/router-execution-context.js:134:59)\n' +
[my-microservice] ' at /Users/me/repos/my-repo/node_modules/@nestjs/core/router/router-execution-context.js:42:37\n' +
[my-microservice] ' at /Users/me/repos/my-repo/node_modules/@nestjs/core/router/router-proxy.js:9:23\n' +
[my-microservice] ' at Layer.handle [as handle_request] (/Users/me/repos/my-repo/node_modules/express/lib/router/layer.js:95:5)\n' +
[my-microservice] ' at next (/Users/me/repos/my-repo/node_modules/express/lib/router/route.js:144:13)\n' +
[my-microservice] ' at Route.dispatch (/Users/me/repos/my-repo/node_modules/express/lib/router/route.js:114:3)\n' +
[my-microservice] ' at Layer.handle [as handle_request] (/Users/me/repos/my-repo/node_modules/express/lib/router/layer.js:95:5)\n' +
[my-microservice] ' at /Users/me/repos/my-repo/node_modules/express/lib/router/index.js:284:15\n' +
[my-microservice] ' at param (/Users/me/repos/my-repo/node_modules/express/lib/router/index.js:365:14)\n' +
[my-microservice] ' at param (/Users/me/repos/my-repo/node_modules/express/lib/router/index.js:376:14)'
[my-microservice] ]
[my-microservice] } +47ms
Then, 10 seconds later (timeout value), an unhandled error crashes the microservice:
[my-microservice] Error: Failed to receive response within timeout of 10000ms for exchange "{my-exchange}" and routing key "{my-routing-key}"
[my-microservice] at /Users/me/repos/my-repo/node_modules/@golevelup/nestjs-rabbitmq/src/amqp/connection.ts:374:15
[my-microservice] at /Users/me/repos/my-repo/node_modules/rxjs/src/internal/operators/map.ts:58:33
[my-microservice] at OperatorSubscriber._this._next (/Users/me/repos/my-repo/node_modules/rxjs/src/internal/operators/OperatorSubscriber.ts:70:13)
[my-microservice] at OperatorSubscriber.Subscriber.next (/Users/me/repos/my-repo/node_modules/rxjs/src/internal/Subscriber.ts:75:12)
[my-microservice] at /Users/me/repos/my-repo/node_modules/rxjs/src/internal/operators/throwIfEmpty.ts:50:22
[my-microservice] at OperatorSubscriber._this._next (/Users/me/repos/my-repo/node_modules/rxjs/src/internal/operators/OperatorSubscriber.ts:70:13)
[my-microservice] at OperatorSubscriber.Subscriber.next (/Users/me/repos/my-repo/node_modules/rxjs/src/internal/Subscriber.ts:75:12)
[my-microservice] at /Users/me/repos/my-repo/node_modules/rxjs/src/internal/operators/take.ts:60:26
[my-microservice] at OperatorSubscriber._this._next (/Users/me/repos/my-repo/node_modules/rxjs/src/internal/operators/OperatorSubscriber.ts:70:13)
[my-microservice] at OperatorSubscriber.Subscriber.next (/Users/me/repos/my-repo/node_modules/rxjs/src/internal/Subscriber.ts:75:12)
[my-microservice] at AsyncAction.<anonymous> (/Users/me/repos/my-repo/node_modules/rxjs/src/internal/observable/timer.ts:173:20)
[my-microservice] at AsyncAction._execute (/Users/me/repos/my-repo/node_modules/rxjs/src/internal/scheduler/AsyncAction.ts:120:12)
[my-microservice] at AsyncAction.execute (/Users/me/repos/my-repo/node_modules/rxjs/src/internal/scheduler/AsyncAction.ts:95:24)
[my-microservice] at AsyncScheduler.flush (/Users/me/repos/my-repo/node_modules/rxjs/src/internal/scheduler/AsyncScheduler.ts:40:27)
[my-microservice] at listOnTimeout (node:internal/timers:559:17)
[my-microservice] at processTimers (node:internal/timers:502:7)
This last error can't be caught with a try/catch, my (temporary) fix available is to add a listener on process.on('unhandledRejection', ...) which I'd rather not do.
@cs-salim Could you share a reproduction of the setup that you have? Even if it is minimal to try and understand what the fix would look like.
Do you expect that we funnel the exception to the logs but do not throw an exception so it doesn't bubble up?
I would need to understand in order to delegate having a feature flag to make it not throw (because there might be users who rely on that) or simply allow you to pass your own rmq client/callback to deal with the exception/error
@underfisk Here's a repro repo: https://github.com/cs-salim/golevelup-issue686 (just make a GET request to http://localhost:3000 and wait 10 seconds, all relevant code is in src/app.controller.ts)
I found out while reproducing it that there's actually no need to even disconnect from RMQ, the issue seems to happen whenever there is no response after calling AmqpConnection.request()
I think throwing an exception is fine, but as a user I need to be able to catch it to then do whatever I want (log, crash...)