ECONNREFUSED but the server is running
AdriDevelopsThings opened this issue · comments
I get the following error while connecting with amqplib (rabbit-queue but it depends on amqplib):
api_1 | Error: connect ECONNREFUSED 172.20.0.4:5672
api_1 | at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1247:16)
api_1 | Error: connect ECONNREFUSED 172.20.0.4:5672
api_1 | at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1247:16)
api_1 | Error: connect ECONNREFUSED 172.20.0.4:5672
api_1 | at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1247:16)
api_1 | Error: connect ECONNREFUSED 172.20.0.4:5672
api_1 | at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1247:16)
api_1 | Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
api_1 | at new NodeError (node:internal/errors:387:5)
api_1 | at ServerResponse.setHeader (node:_http_outgoing:603:11)
api_1 | at /app/node_modules/apollo-server-express/dist/ApolloServer.js:85:29
api_1 | at runMicrotasks (<anonymous>)
api_1 | at processTicksAndRejections (node:internal/process/task_queues:96:5)
My rabbit server is running on the latest version 3.
My amqplib is running on version 0.8.0.
The rabbit server and my api server with amqplib running is separated in docker containers.
When I try to connect via netcat in the docker container it works:
/app # nc 172.20.0.4 5672
asdf
AMQP
I don't understand why I get a ECONNREFUSED but I can connect to the server and the host and port ist right.
Can somebody help me?
Thanks
I have the same error with graphql-apollo-server.
Any help ?
There are too many 3rd party layers to be able to help you with this effectively. Can you rule out graphql-apollo-server and rabbit-queue by writing a very simple script that just connects with amqplib and running this in your api_1 docker container please?
Something like...
const amqplib = require('amqplib');
(async () => {
const connection = await amqplib.connect('amqp://guest:guest@172.20.0.4:5672');
console.log('Connected');
})();I looked into rabbit-queue and it doesn't appear to do anything unusual, other than depending on an old version of amqplib.
According to the Node.js error documentation ECONNREFUSED occurs when the target machine actively refuses the connection, usually because nothing is listening on the specified port. i.e. amqplib appears to be attempting to connect as I would expect, but the remote machine (or some intermediary system like a tcp proxy or firewall) is refusing the connection.
Your netcat test indicates the network is fine this though, especially since you ran this from within same docker container that the node app is running in. My guess is then that either there is some strange interaction with another package such as apollo-graphql-server, or something in your networking layer isn't working properly. I've previously experienced issues with docker where the networking layer wasn't fully established until after the main process started, but from memory this impacted DNS resolution. I've also previous had issues with ARP Caches, so maybe it's just possible that the request is being routed to the wrong server, but it's a stretch.
Maybe running something like tcpdump from inside the app container, on the machine that hosts it, and/or on the rabbitmq server will reveal something. It's also worth looking at the RabbitMQ server logs in case it reveals anything. You should see something like
2022-09-04 09:52:04.438427+00:00 [info] <0.945.0> accepting AMQP connection <0.945.0> (172.17.0.1:58486 -> 172.17.0.2:5672)
Sorry I can't be more help
I have been facing this issue and in my case it feels related to the node version.
My app is working with node v16 but crashes with ECONNREFUSED when running on node v17 or v18.
I've been able to reproduce this behaviour on docker with the amqplib async setup.
The docker file is quite simple :
this one works
from node:16-alpine
WORKDIR /usr/src/app
copy . .
run npm install
CMD [ "node", "src/index.js" ]
ECONNREFUSED when running this container
from node:18-alpine
WORKDIR /usr/src/app
copy . .
run npm install
CMD [ "node", "src/index.js" ]
Could it be nodejs/node#40702
Looks more like a DNS / IPv6 issue though. The example above was using IPv4 IP addresses so may not be related
It can be related to this nodejs/node#40702
The connection is successful when using 127.0.0.1 instead of localhost
@NathanGrimaud, I'd like to reproduce as you describe in #704 (comment) but think I'm misunderstanding something. The dockerfiles you show, don't install / start RabbitMQ inside the container, but if you're connecting to localhost / 127.0.0.1 where is rabbit running?
Rabbitmq was running the host and the connection was shared with --network=host
Interestingly I am unable to reproduce. The following script works fine using both node 16 and node 18
const amqplib = require('amqplib');
(async () => {
const queue = 'tasks';
const conn = await amqplib.connect('amqp://localhost');
const ch1 = await conn.createChannel();
await ch1.assertQueue(queue);
ch1.consume(queue, (msg) => {
if (msg !== null) {
console.log('Received:', msg.content.toString());
ch1.ack(msg);
} else {
console.log('Consumer cancelled by server');
}
});
// Sender
const ch2 = await conn.createChannel();
setInterval(() => {
ch2.sendToQueue(queue, Buffer.from(`Something to do: ${process.version}`));
}, 1000);
})();docker run --network host node16
Received: Something to do: v16.17.0
Received: Something to do: v16.17.0
Received: Something to do: v16.17.0
docker run --network host node18
Received: Something to do: v18.8.0
Received: Something to do: v18.8.0
Received: Something to do: v18.8.0
This was with a RabbitMQ container started with the following
docker run --network host rabbitmq:alpine
Just tried it and it works when rabbitmq is started via docker but I get ECONNREFUSED when rabbitmq is running natively
I just spun up a broker using cloudamqp and bot the node18 and node16 connected fine using the hostname. Sadly, I don't think there's anything more I can do to help resolve this issue. The symptoms experienced by @NathanGrimaud are slightly different to those reported by @AdriDevelopsThings, but as far as I can tell neither point to an issue with amqplib.
If you thing there's more I can do to investigate, I'm happy to keep the issue open, but otherwise I propose I close it.
I'll try to get a replicable setup but it feels more related to node than to this lib. In my case a hint in the docs could have helped. it might be the easiest way to avoid this issue.
Thanks @NathanGrimaud, I'll think about the documentation