[INF] Initiating Shutdown...
Giancarlo1974 opened this issue · comments
Observed behavior
I have 2 NATS nodes configured in cluster
In the docker logs I see that every now and then the server does a shutdown
======================== NODE 1 =========================
[1] 2024/04/03 22:07:12.033486 [ERR] Error trying to connect to route (attempt 10): dial tcp 10.1.0.164:4248: connect: connection refused
[1] 2024/04/03 22:07:12.049183 [ERR] Error trying to connect to route (attempt 10): dial tcp 10.1.0.164:4248: connect: connection refused
[1] 2024/04/03 22:07:12.072953 [ERR] Error trying to connect to route (attempt 10): dial tcp 10.1.0.164:4248: connect: connection refused
[1] 2024/04/03 22:07:12.702875 [INF] 10.1.0.164:57960 - rid:73 - Route connection created
[1] 2024/04/03 22:07:12.703046 [INF] 10.1.0.164:57958 - rid:74 - Route connection created
[1] 2024/04/03 22:07:12.707020 [INF] 10.1.0.164:57962 - rid:75 - Route connection created
[1] 2024/04/03 22:07:12.806373 [INF] 10.1.0.164:57970 - rid:76 - Route connection created
[1] 2024/04/03 22:07:13.018694 [INF] 10.1.0.164:4248 - rid:77 - Route connection created
[1] 2024/04/03 22:07:13.019759 [INF] 10.1.0.164:4248 - rid:77 - Router connection closed: Duplicate Route
[1] 2024/04/03 22:07:13.036711 [INF] 10.1.0.164:4248 - rid:78 - Route connection created
[1] 2024/04/03 22:07:13.037526 [INF] 10.1.0.164:4248 - rid:78 - Router connection closed: Duplicate Route
[1] 2024/04/03 22:07:13.052223 [INF] 10.1.0.164:4248 - rid:79 - Route connection created
[1] 2024/04/03 22:07:13.053067 [INF] 10.1.0.164:4248 - rid:79 - Router connection closed: Duplicate Route
[1] 2024/04/03 22:07:13.076057 [INF] 10.1.0.164:4248 - rid:80 - Route connection created
[1] 2024/04/03 22:07:13.076725 [INF] 10.1.0.164:4248 - rid:80 - Router connection closed: Duplicate Route
[1] 2024/04/03 22:19:02.026884 [INF] Initiating Shutdown...
[1] 2024/04/03 22:19:02.027512 [INF] 10.1.0.164:57958 - rid:74 - Router connection closed: Server Shutdown
[1] 2024/04/03 22:19:02.027677 [INF] 10.1.0.164:57960 - rid:73 - Router connection closed: Server Shutdown
[1] 2024/04/03 22:19:02.028677 [INF] 10.1.0.164:57970 - rid:76 - Router connection closed: Server Shutdown
[1] 2024/04/03 22:19:02.028712 [INF] 10.1.0.164:57962 - rid:75 - Router connection closed: Server Shutdown
[1] 2024/04/03 22:19:02.030299 [INF] Server Exiting..
[1] 2024/04/03 22:19:12.456376 [INF] Starting nats-server
[1] 2024/04/03 22:19:12.457011 [INF] Version: 2.10.12
[1] 2024/04/03 22:19:12.457029 [INF] Git: [121169ea]
[1] 2024/04/03 22:19:12.457046 [INF] Cluster: ACSNATS
[1] 2024/04/03 22:19:12.457062 [INF] Name: NDIC56VL6NR6EVLV7UBPNGP5423CBEDOQ5Y56KXDYZLNFN2COPBNS7XR
[1] 2024/04/03 22:19:12.457109 [INF] ID: NDIC56VL6NR6EVLV7UBPNGP5423CBEDOQ5Y56KXDYZLNFN2COPBNS7XR
[1] 2024/04/03 22:19:12.459716 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2024/04/03 22:19:12.460088 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2024/04/03 22:19:12.461610 [INF] Server is ready
[1] 2024/04/03 22:19:12.461711 [INF] Cluster name is ACSNATS
[1] 2024/04/03 22:19:12.461837 [INF] Listening for route connections on 0.0.0.0:4248
[1] 2024/04/03 22:19:12.463426 [ERR] Error trying to connect to route (attempt 1): dial tcp 10.1.0.172:4248: connect: connection refused
[1] 2024/04/03 22:19:12.463570 [ERR] Error trying to connect to route (attempt 1): dial tcp 10.1.0.172:4248: connect: connection refused
[1] 2024/04/03 22:19:12.467452 [INF] 10.1.0.164:4248 - rid:5 - Route connection created
[1] 2024/04/03 22:19:12.468723 [INF] 10.1.0.164:4248 - rid:6 - Route connection created
[1] 2024/04/03 22:19:12.556625 [INF] 10.1.0.164:4248 - rid:7 - Route connection created
[1] 2024/04/03 22:19:12.575444 [INF] 10.1.0.164:4248 - rid:8 - Route connection created
[1] 2024/04/03 22:19:13.084224 [INF] 10.1.0.164:33654 - rid:9 - Route connection created
[1] 2024/04/03 22:19:13.084535 [INF] 10.1.0.164:33652 - rid:10 - Route connection created
[1] 2024/04/03 22:19:13.085419 [INF] 10.1.0.164:33654 - rid:9 - Router connection closed: Client Closed
[1] 2024/04/03 22:19:13.085676 [INF] 10.1.0.164:33652 - rid:10 - Router connection closed: Client Closed
[1] 2024/04/03 22:19:13.087112 [INF] 10.1.0.164:33656 - rid:11 - Route connection created
[1] 2024/04/03 22:19:13.087302 [INF] 10.1.0.164:33658 - rid:12 - Route connection created
[1] 2024/04/03 22:19:13.087917 [INF] 10.1.0.164:33656 - rid:11 - Router connection closed: Client Closed
[1] 2024/04/03 22:19:13.088533 [INF] 10.1.0.164:33658 - rid:12 - Router connection closed: Duplicate Route
[1] 2024/04/03 22:37:01.248201 [INF] 10.1.0.164:4248 - rid:7 - Router connection closed: Client Closed
[1] 2024/04/03 22:37:01.249030 [INF] 10.1.0.164:4248 - rid:5 - Router connection closed: Client Closed
[1] 2024/04/03 22:37:01.249116 [INF] 10.1.0.164:4248 - rid:6 - Router connection closed: Client Closed
[1] 2024/04/03 22:37:01.250857 [INF] 10.1.0.164:4248 - rid:8 - Router connection closed: Client Closed
[1] 2024/04/03 22:37:02.298140 [ERR] Error trying to connect to route (attempt 1): dial tcp 10.1.0.164:4248: connect: connection refused
[1] 2024/04/03 22:37:02.323290 [ERR] Error trying to connect to route (attempt 1): dial tcp 10.1.0.164:4248: connect: connection refused
======================== NODE 2 =========================
[1] 2024/04/03 21:49:12.686105 [INF] 172.16.0.3:4248 - rid:58 - Route connection created
[1] 2024/04/03 21:49:12.686804 [INF] 172.16.0.3:4248 - rid:58 - Router connection closed: Duplicate Route
[1] 2024/04/03 22:07:02.017157 [INF] Initiating Shutdown...
[1] 2024/04/03 22:07:02.017457 [INF] 172.16.0.3:37612 - rid:52 - Router connection closed: Server Shutdown
[1] 2024/04/03 22:07:02.017530 [INF] 172.16.0.3:37616 - rid:51 - Router connection closed: Server Shutdown
[1] 2024/04/03 22:07:02.017547 [INF] 172.16.0.3:37618 - rid:53 - Router connection closed: Server Shutdown
[1] 2024/04/03 22:07:02.017567 [INF] 172.16.0.3:37634 - rid:54 - Router connection closed: Server Shutdown
[1] 2024/04/03 22:07:02.040223 [INF] Server Exiting..
[1] 2024/04/03 22:07:12.738115 [INF] Starting nats-server
[1] 2024/04/03 22:07:12.738378 [INF] Version: 2.10.12
[1] 2024/04/03 22:07:12.738388 [INF] Git: [121169ea]
[1] 2024/04/03 22:07:12.738396 [INF] Cluster: ACSNATS
[1] 2024/04/03 22:07:12.738407 [INF] Name: NDLW5AXE2F5QIUR7PBVHNLFGHBM37M4UVL2VNEBZNNXQJ5XT7SEKVGSR
[1] 2024/04/03 22:07:12.738415 [INF] ID: NDLW5AXE2F5QIUR7PBVHNLFGHBM37M4UVL2VNEBZNNXQJ5XT7SEKVGSR
[1] 2024/04/03 22:07:12.739837 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2024/04/03 22:07:12.740008 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2024/04/03 22:07:12.740723 [INF] Server is ready
[1] 2024/04/03 22:07:12.740801 [INF] Cluster name is ACSNATS
[1] 2024/04/03 22:07:12.740873 [INF] Listening for route connections on 0.0.0.0:4248
[1] 2024/04/03 22:07:12.742482 [ERR] Error trying to connect to route (attempt 1): dial tcp 10.1.0.172:4248: connect: connection refused
[1] 2024/04/03 22:07:12.742572 [ERR] Error trying to connect to route (attempt 1): dial tcp 10.1.0.172:4248: connect: connection refused
[1] 2024/04/03 22:07:12.744767 [INF] 172.16.0.3:4248 - rid:6 - Route connection created
[1] 2024/04/03 22:07:12.744891 [INF] 172.16.0.3:4248 - rid:5 - Route connection created
[1] 2024/04/03 22:07:12.747320 [INF] 172.16.0.3:4248 - rid:7 - Route connection created
[1] 2024/04/03 22:07:12.847253 [INF] 172.16.0.3:4248 - rid:8 - Route connection created
[1] 2024/04/03 22:07:13.060613 [INF] 172.16.0.3:38824 - rid:9 - Route connection created
[1] 2024/04/03 22:07:13.061785 [INF] 172.16.0.3:38824 - rid:9 - Router connection closed: Duplicate Route
[1] 2024/04/03 22:07:13.078488 [INF] 172.16.0.3:38828 - rid:10 - Route connection created
[1] 2024/04/03 22:07:13.079591 [INF] 172.16.0.3:38828 - rid:10 - Router connection closed: Client Closed
[1] 2024/04/03 22:07:13.093958 [INF] 172.16.0.3:38832 - rid:11 - Route connection created
[1] 2024/04/03 22:07:13.094910 [INF] 172.16.0.3:38832 - rid:11 - Router connection closed: Client Closed
[1] 2024/04/03 22:07:13.117794 [INF] 172.16.0.3:38842 - rid:12 - Route connection created
[1] 2024/04/03 22:07:13.118580 [INF] 172.16.0.3:38842 - rid:12 - Router connection closed: Client Closed
[1] 2024/04/03 22:19:02.036164 [INF] 172.16.0.3:4248 - rid:7 - Router connection closed: Client Closed
[1] 2024/04/03 22:19:02.036393 [INF] 172.16.0.3:4248 - rid:8 - Router connection closed: Client Closed
[1] 2024/04/03 22:19:02.036401 [INF] 172.16.0.3:4248 - rid:6 - Router connection closed: Client Closed
[1] 2024/04/03 22:19:02.036805 [INF] 172.16.0.3:4248 - rid:5 - Router connection closed: Client Closed
[1] 2024/04/03 22:19:03.060498 [ERR] Error trying to connect to route (attempt 1): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:03.061615 [ERR] Error trying to connect to route (attempt 1): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:03.067392 [ERR] Error trying to connect to route (attempt 1): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:03.067422 [ERR] Error trying to connect to route (attempt 1): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:04.063186 [ERR] Error trying to connect to route (attempt 2): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:04.063251 [ERR] Error trying to connect to route (attempt 2): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:04.069654 [ERR] Error trying to connect to route (attempt 2): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:04.069697 [ERR] Error trying to connect to route (attempt 2): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:05.065871 [ERR] Error trying to connect to route (attempt 3): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:05.065880 [ERR] Error trying to connect to route (attempt 3): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:05.071486 [ERR] Error trying to connect to route (attempt 3): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:05.071543 [ERR] Error trying to connect to route (attempt 3): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:06.067945 [ERR] Error trying to connect to route (attempt 4): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:06.067945 [ERR] Error trying to connect to route (attempt 4): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:06.073407 [ERR] Error trying to connect to route (attempt 4): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:06.073473 [ERR] Error trying to connect to route (attempt 4): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:07.069945 [ERR] Error trying to connect to route (attempt 5): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:07.069988 [ERR] Error trying to connect to route (attempt 5): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:07.075431 [ERR] Error trying to connect to route (attempt 5): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:07.075462 [ERR] Error trying to connect to route (attempt 5): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:08.072822 [ERR] Error trying to connect to route (attempt 6): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:08.072823 [ERR] Error trying to connect to route (attempt 6): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:08.077272 [ERR] Error trying to connect to route (attempt 6): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:08.077337 [ERR] Error trying to connect to route (attempt 6): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:09.074656 [ERR] Error trying to connect to route (attempt 7): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:09.074730 [ERR] Error trying to connect to route (attempt 7): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:09.079015 [ERR] Error trying to connect to route (attempt 7): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:09.079099 [ERR] Error trying to connect to route (attempt 7): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:10.077422 [ERR] Error trying to connect to route (attempt 8): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:10.077422 [ERR] Error trying to connect to route (attempt 8): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:10.081013 [ERR] Error trying to connect to route (attempt 8): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:10.081273 [ERR] Error trying to connect to route (attempt 8): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:11.079348 [ERR] Error trying to connect to route (attempt 9): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:11.079406 [ERR] Error trying to connect to route (attempt 9): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:11.083136 [ERR] Error trying to connect to route (attempt 9): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:11.083208 [ERR] Error trying to connect to route (attempt 9): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:12.081753 [ERR] Error trying to connect to route (attempt 10): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:12.081887 [ERR] Error trying to connect to route (attempt 10): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:12.085249 [ERR] Error trying to connect to route (attempt 10): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:12.085257 [ERR] Error trying to connect to route (attempt 10): dial tcp 172.16.0.3:4248: connect: connection refused
[1] 2024/04/03 22:19:12.467970 [INF] 172.16.0.3:55772 - rid:62 - Route connection created
[1] 2024/04/03 22:19:12.468164 [INF] 172.16.0.3:55788 - rid:63 - Route connection created
[1] 2024/04/03 22:19:12.557297 [INF] 172.16.0.3:55794 - rid:64 - Route connection created
[1] 2024/04/03 22:19:12.576137 [INF] 172.16.0.3:55810 - rid:65 - Route connection created
[1] 2024/04/03 22:19:13.084075 [INF] 172.16.0.3:4248 - rid:66 - Route connection created
[1] 2024/04/03 22:19:13.084074 [INF] 172.16.0.3:4248 - rid:67 - Route connection created
[1] 2024/04/03 22:19:13.085589 [INF] 172.16.0.3:4248 - rid:67 - Router connection closed: Duplicate Route
[1] 2024/04/03 22:19:13.085758 [INF] 172.16.0.3:4248 - rid:66 - Router connection closed: Duplicate Route
[1] 2024/04/03 22:19:13.087186 [INF] 172.16.0.3:4248 - rid:68 - Route connection created
[1] 2024/04/03 22:19:13.087205 [INF] 172.16.0.3:4248 - rid:69 - Route connection created
[1] 2024/04/03 22:19:13.088156 [INF] 172.16.0.3:4248 - rid:69 - Router connection closed: Duplicate Route
[1] 2024/04/03 22:19:13.088584 [INF] 172.16.0.3:4248 - rid:68 - Router connection closed: Duplicate Route
[1] 2024/04/03 22:37:01.190745 [INF] Initiating Shutdown...
[1] 2024/04/03 22:37:01.191265 [INF] 172.16.0.3:55794 - rid:64 - Router connection closed: Server Shutdown
[1] 2024/04/03 22:37:01.191339 [INF] 172.16.0.3:55788 - rid:63 - Router connection closed: Server Shutdown
[1] 2024/04/03 22:37:01.191400 [INF] 172.16.0.3:55772 - rid:62 - Router connection closed: Server Shutdown
[1] 2024/04/03 22:37:01.191526 [INF] 172.16.0.3:55810 - rid:65 - Router connection closed: Server Shutdown
[1] 2024/04/03 22:37:01.213443 [INF] Server Exiting..
[1] 2024/04/03 22:37:11.876928 [INF] Starting nats-server
[1] 2024/04/03 22:37:11.877187 [INF] Version: 2.10.12
[1] 2024/04/03 22:37:11.877198 [INF] Git: [121169ea]
[1] 2024/04/03 22:37:11.877207 [INF] Cluster: ACSNATS
[1] 2024/04/03 22:37:11.877215 [INF] Name: NDZIXCIPHDBSYNGKBQ4HL5UW4VG7MBZMOW6PN47DUOPDFFUPHHE7IDSS
[1] 2024/04/03 22:37:11.877223 [INF] ID: NDZIXCIPHDBSYNGKBQ4HL5UW4VG7MBZMOW6PN47DUOPDFFUPHHE7IDSS
[1] 2024/04/03 22:37:11.878863 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2024/04/03 22:37:11.879143 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2024/04/03 22:37:11.879930 [INF] Server is ready
Expected behavior
it does not have to shutdown
Server and client version
======================== NODE 1 =========================
cat docker-compose-node01.yml
version: '3.6'
services:
nats1:
image: nats:alpine3.19
network_mode: "host"
command: "--cluster_name ACSNATS --cluster nats://0.0.0.0:4248 -routes nats://10.1.0.172:4248,nats://nats1.mycompany.it:4248,nats://nats2.mycompany.it:4248 --http_port 8222"
restart: unless-stopped
======================== NODE 2 =========================
cat docker-compose-node02.yml
version: '3.6'
services:
nats2:
image: nats:alpine3.19
network_mode: "host"
command: "--cluster_name ACSNATS --cluster nats://0.0.0.0:4248 -routes nats://10.1.0.172:4248,nats://nats1.mycompany.it:4248,nats://nats2.mycompany.it:4248 --http_port 8222"
restart: unless-stopped
Host environment
the third node (10.1.0.172) is offline
I think I can work with just 2 nodes
======================== NODE 1 =========================
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
docker --version
Docker version 24.0.5, build 24.0.5-0ubuntu1~22.04.1
======================== NODE 2 =========================
cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
docker --version
Docker version 18.09.3, build 774a1f4
Steps to reproduce
No response
@Giancarlo1974 this repo is for the NATS .NET client.
I believe you meant to open this against: https://github.com/nats-io/nats-server