MaxListenersExceededWarning: Possible EventEmitter memory leak detected.

Question

MaxListenersExceededWarning: Possible EventEmitter memory leak detected.

jdgamble555 opened this issue a year ago · comments

Environment:

Node.js Version: 18.13.0
Redis Server Version: 6.26
Node Redis Version: 4.5.1
Platform: Windows 11

import { createClient } from "redis";
import { REDIS_URL } from '$env/static/private';

const client = createClient({ url: REDIS_URL, socket: { keepAlive: false } });
client.setMaxListeners(0);

client.on('error', error => console.error('ERR:REDIS:', error));
client.connect();

export const query = async (q: string) => await client.graph.query('test-graphs', q);

This seems to be a prevalent issue for over 10 years, however, I could not find any bug reports in the Source repository (this repository). As you can see, I even have setMaxListeners set to 0 and keepAlive set to false to no avail. Sure, I would like to ignore the error, but if I could figure out how to FIX the error, that would be nice; I can't seem to do either.

Is there a way to stop listening to the client after every query? This seems to happen after I run 10 or so queries. Sometimes it takes more (not sure why).

J

Leibale Eidelman · Answer 1 · Mon Jan 23 2023 17:10:21 GMT+0800 (China Standard Time)

setMaxListeners is a built-in function on EventEmitter, see here.
You are settings the maximum listeners to 0, then one line after you add a listener for error which will always log this warning, because 1 > 0... why do you even use this function? What are you trying to prevent?
If you want to close the client at some point you can use client.quit()/client.disconnect().

edit: setting keepAlive to false means that if the socket is idle for too long (server-side configuration), the server will close the socket and the client will reconnect.

Jonathan Gamble · Answer 2 · Mon Jan 23 2023 23:13:12 GMT+0800 (China Standard Time)

keepAlive

From version 3.2 onwards, Redis has TCP keepalive (SO_KEEPALIVE socket option) enabled by default and set to about 300 seconds. This option is useful in order to detect dead peers (clients that cannot be reached even if they look connected). Moreover, if there is network equipment between clients and servers that need to see some traffic in order to take the connection open, the option will prevent unexpected connection closed events.

TCP keepalive.

If non-zero, use SO_KEEPALIVE to send TCP ACKs to clients in absence
of communication. This is useful for two reasons:

Detect dead peers.

Force network equipment in the middle to consider the connection to be
alive.

On Linux, the specified value (in seconds) is the period used to send ACKs.
Note that to close the connection the double of the time is needed.
On other kernels the period depends on the kernel configuration.

A reasonable value for this option is 300 seconds, which is the new
Redis default starting with Redis 3.2.1.
tcp-keepalive 300

I don't see any other documentation, but I think the default is false (or is it 300?), as that seems to be the case in other packages. There doesn't seem to be anywhere in node-redis docs that clearly explains what each option does. Does it stay open by default, reconnect? Maybe I'm misunderstanding what the above means? 🤷

setMaxListeners

So, setMaxListeners exists on the redis client as well as in node events:

But I think you're right, I need to import it directly from events instead:

import { setMaxListeners } from 'events';
setMaxListeners(0);

Also, when setMaxListeners is set to 0, it does not mean 0 listeners, it means unlimited listeners.

This seems to solve the the symptom to the problem by not displaying the warning, but not the problem itself. I don't want to create a listener each time I run a query. I want to keep the (one) connection alive so I don't open up a lot of connections if there are multiple requests.

Quit Vs. Disconnect

So apparently I need to use quit instead of disconnect. However, if I use either, I will have to re-open a client connection each time I run a query, which is not what I want.

How do I keep my connection open, and prevent each query from creating a new listener?

This is my real problem. It doesn't seem like there is a way to do this?

J

Leibale Eidelman · Answer 3 · Mon Jan 23 2023 23:28:22 GMT+0800 (China Standard Time)

First of all, setMaxListeners has nothing to do with the socket/connection to Redis, it's the maximum client-side event listeners on the client:

import { EventEmitter } from 'node:events';

const emitter = new EventEmitter();
client.setMaxListeners(1);
client.on('event', () => {});
client.on('event', () => {}); // "MaxListenersExceededWarning: Possible EventEmitter memory leak detected" because the number of listeners is 1, and max listeners is 0

Regarding the question: running a query does not create a new socket/connection/"listener" (I think this is what you mean by "listener", but to be honest I'm not too sure..), it uses the existing socket and executes all the commands on it:

import { createClient } from 'redis';

const client = createClient({ url: '<REDIS_URL>' });

client.on('error', error => console.error('ERR:REDIS:', error));

await client.connect();

// these commands are running on the same socket/connection
await client.ping();
await client.graph.query('a', '*');
await client.graph.query('b', '*');

If at some point you want to gracefully close the application (for , use client.quit()/client.disconnect(), for example:

process.on('SIGINT', () => client.quit());

edit:
regarding the client options docs, see this

Jonathan Gamble · Answer 4 · Tue Jan 24 2023 00:02:39 GMT+0800 (China Standard Time)

So what is creating more than 10 client side event listeners? It seems each query creates one and I get the node error (after I run 10 or so queries) unless I specifically use this code:

import { setMaxListeners } from 'events';
setMaxListeners(0);

which is just ignoring the problem and taking care of the symptom.

Using this:

const client = createClient({ url: REDIS_URL });
client.setMaxListeners(0);

has no effect.

I guess I need to close the client side listeners in order to prevent the error message?

J

Note - That makes sense for gracefully quitting.

Leibale Eidelman · Answer 5 · Tue Jan 24 2023 00:17:24 GMT+0800 (China Standard Time)

Search for "client.on" in your project and make sure you are not creating listeners over time. If you want we can debug it together, I'll wait in this meeting for the next 30 min

Jonathan Gamble · Answer 6 · Tue Jan 24 2023 00:29:06 GMT+0800 (China Standard Time)

I just left my computer until after 5 central, or I would be there! Not sure if this helps, but here is my repo...

https://github.com/jdgamble555/redisgraph-graphql-sveltekit/blob/master/src/lib/server/redis.ts

and where I make the calls

https://github.com/jdgamble555/redisgraph-graphql-sveltekit/blob/master/src/lib/server/db.ts

J

I can't definitely meet this evening or tom if all else fails!

Leibale Eidelman · Answer 7 · Tue Jan 24 2023 01:01:08 GMT+0800 (China Standard Time)

I just ran your project locally, and these are the warnings I got:

(node:45655) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGTERM listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
(node:45655) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added to [process]. Use emitter.setMaxListeners() to increase limit

these warnings are about SIGTERM and SIGINT listeners on global.process, it has nothing to do with the Redis client...

You get these errors because you are creating an ApolloServer instance for every POST request to /graphql instead of reusing the same instance (and ApolloServer is listening to SIGTERM and SIGINT).

moving

const server = new ApolloServer({
    typeDefs,
    resolvers
});

outside the POST function solves the problem.

edit:

I think your project can be a good demonstration of how to use Redis + RedisGraph + GraphQL + SvelteKit. If you can share a little bit more about what you are planning to do with this project that will be very helpful :)

Jonathan Gamble · Answer 8 · Tue Jan 24 2023 04:43:59 GMT+0800 (China Standard Time)

Wow. That fixed the problem! Thank you! Not even sure how you caught that!

Here is the final project if you're curious: https://redisgraph-graphql-sveltekit.vercel.app/

It really was just a test project to get me familiar with putting the technologies together. I am a huge RedisGraph fan, and I'm not even sure why it isn't more popular. I plan on writing articles about some of these concepts in the project on my site Code.Build.

Thanks for taking the time!

J