Eclipse Ditto on Docker Swarm | nginx: [emerg] host not found in upstream "gateway" in /etc/nginx/nginx.conf:44 | [error] 33#33: *2 connect() failed (111: Connection refused) while connecting to upstream
aissam-en opened this issue · comments
Challenges Running Eclipse Ditto on Docker Swarm
I'm trying to run Eclipse Ditto on Docker Swarm, initially, I was able to successfully run Ditto using the provided docker-compose.yml file in Docker (not Swarm mode) without modifying the nginx.conf file.
I made some adjustments to the docker-compose.yml
file to work in Docker Swarm :
- Commented out
restart: always
; - Commented out
healthcheck
section; - Commented out
volumes
for theswagger-ui
service, since volumes are not be suitable for Swarm, I will configure it later; - Replaced
volumes
for thenginx
service withconfigs
, to mount nginx files;
I'm encountering an issue where the nginx
service is not recongizing gateway
service, here is its logs:
docker service logs -f my_stack_nginx
my_stack_nginx.1.64bhclqugkeq@docker-desktop | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
my_stack_nginx.1.64bhclqugkeq@docker-desktop | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
my_stack_nginx.1.64bhclqugkeq@docker-desktop | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
my_stack_nginx.1.64bhclqugkeq@docker-desktop | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
my_stack_nginx.1.64bhclqugkeq@docker-desktop | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
my_stack_nginx.1.64bhclqugkeq@docker-desktop | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop | /docker-entrypoint.sh: Configuration complete; ready for start up
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | /docker-entrypoint.sh: Configuration complete; ready for start up
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | 2024/06/18 15:44:39 [emerg] 1#1: host not found in upstream "gateway" in /etc/nginx/nginx.conf:44
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | nginx: [emerg] host not found in upstream "gateway" in /etc/nginx/nginx.conf:44
the line 44 from nginx.conf
file :
39 ...
40 # connections api using devops user configured in Ditto
41 location /api/2/connections {
42 include nginx-cors.conf;
43
44 proxy_pass http://gateway:8080;
45 proxy_http_version 1.1;
46 proxy_set_header Host $http_host;
47 proxy_set_header X-Real-IP $remote_addr;
48 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
49 proxy_set_header X-Forwarded-User $remote_user;
50
51 proxy_set_header Connection '';
52 chunked_transfer_encoding off;
53 proxy_buffering off;
54 proxy_cache off;
55 }
56 ...
and when I send request from Postman, ex : GET http://localhost:8080/api/2/things/ with ditto:ditto in Authorization, I get this as output :
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>
The nginx logs :
docker service logs -f my_stack_nginx
my_stack_nginx.1.64bhclqugkeq@docker-desktop | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
my_stack_nginx.1.64bhclqugkeq@docker-desktop | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
my_stack_nginx.1.64bhclqugkeq@docker-desktop | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
my_stack_nginx.1.64bhclqugkeq@docker-desktop | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
my_stack_nginx.1.64bhclqugkeq@docker-desktop | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
my_stack_nginx.1.64bhclqugkeq@docker-desktop | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop | /docker-entrypoint.sh: Configuration complete; ready for start up
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | /docker-entrypoint.sh: Configuration complete; ready for start up
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | 2024/06/18 15:44:39 [emerg] 1#1: host not found in upstream "gateway" in /etc/nginx/nginx.conf:44
my_stack_nginx.1.rtextyzpzpsl@docker-desktop | nginx: [emerg] host not found in upstream "gateway" in /etc/nginx/nginx.conf:44
my_stack_nginx.1.64bhclqugkeq@docker-desktop | 2024/06/18 15:47:55 [error] 33#33: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.2, server: localhost, request: "GET /api/2/things/ HTTP/1.1", upstream: "http://10.0.1.13:8080/api/2/things/", host: "localhost:8080"
my_stack_nginx.1.64bhclqugkeq@docker-desktop | 10.0.0.2 - ditto [18/Jun/2024:15:47:55 +0000] "GET /api/2/things/ HTTP/1.1" 502 150 "-" "PostmanRuntime/7.39.0"
Additional Information:
- Windows 10, 8GB RAM and CPU :
Name : Intel(R) Core(TM) i5-9400H CPU @ 2.50GHz
Number Of Cores : 4
Number Of Logical Processors : 8 - Docker :
Client:
Version: 26.1.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.14.0-desktop.1
Path: C:\Program Files\Docker\cli-plugins\docker-buildx.exe
compose: Docker Compose (Docker Inc.)
Version: v2.27.0-desktop.2
Path: C:\Program Files\Docker\cli-plugins\docker-compose.exe
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.29
Path: C:\Program Files\Docker\cli-plugins\docker-debug.exe
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.2
Path: C:\Program Files\Docker\cli-plugins\docker-dev.exe
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.23
Path: C:\Program Files\Docker\cli-plugins\docker-extension.exe
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.4
Path: C:\Program Files\Docker\cli-plugins\docker-feedback.exe
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.1.0
Path: C:\Program Files\Docker\cli-plugins\docker-init.exe
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: C:\Program Files\Docker\cli-plugins\docker-sbom.exe
scout: Docker Scout (Docker Inc.)
Version: v1.8.0
Path: C:\Program Files\Docker\cli-plugins\docker-scout.exe
Server:
Containers: 34
Running: 9
Paused: 0
Stopped: 25
Images: 53
Server Version: 26.1.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: active
NodeID: mahw8hl8i23zzwm7v2i78vaq7
Is Manager: true
ClusterID: ijzjixaizfhrhv754v8ir4fdy
Managers: 1
Nodes: 1
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 192.168.65.3
Manager Addresses:
192.168.65.3:2377
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e377cd56a71523140ca6ae87e30244719194a521
runc version: v1.1.12-0-g51d5e94
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
Kernel Version: 5.15.153.1-microsoft-standard-WSL2
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 3.721GiB
Name: docker-desktop
ID: ec241856-9f66-4d6c-908f-3ff0184e56b5
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=npipe://\.\pipe\docker_cli
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
The docker-compose.yml
file : https://pastebin.com/nbKcXjDc
The services are runnig :
the services are runnig :
docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
idgug4xs4u6g my_stack_connectivity replicated 1/1 eclipse/ditto-connectivity:latest
16uijjs8vpeb my_stack_ditto-ui replicated 1/1 eclipse/ditto-ui:latest
5pd7diz1uz1i my_stack_gateway replicated 1/1 eclipse/ditto-gateway:latest *:8081->8080/tcp
jkkfj363z20o my_stack_mongodb replicated 1/1 mongo:6.0 *:27017->27017/tcp
mc58254lk2h2 my_stack_nginx replicated 1/1 nginx:1.26-alpine *:8080->80/tcp
kn5evgmkpvdw my_stack_policies replicated 1/1 eclipse/ditto-policies:latest
t4agxrku1991 my_stack_swagger-ui replicated 1/1 swaggerapi/swagger-ui:v5.9.1
v42cz47b90qu my_stack_things replicated 1/1 eclipse/ditto-things:latest
xyoa5siftouk my_stack_things-search replicated 1/1 eclipse/ditto-things-search:latest
Inspect ditto-net-swarm
network :
> docker network ls
NETWORK ID NAME DRIVER SCOPE
e661fefa2fba bridge bridge local
afa0a7369fed docker_default bridge local
636762e55a0e host host local
ipd43285nlar ingress overlay swarm
miy4fnbluu70 my_stack_ditto-net-swarm overlay swarm
4934f4344622 none null local
>docker network inspect my_stack_ditto-net-swarm
[
{
"Name": "my_stack_ditto-net-swarm",
"Id": "miy4fnbluu70bdc30qcqpyqy9",
"Created": "2024-06-18T16:46:24.45599018Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.1.0/24",
"Gateway": "10.0.1.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"22f5721bd207a012953cf4f194abf7ce472ef6b18a6d167df73c4efae26a2ff9": {
"Name": "my_stack_ditto-ui.1.jcnhcxrkxu9kl8v1a7dwjk4e1",
"EndpointID": "008daec7000072bb81e7fef018f022a53a86b27a124aa32392c584b3ab479166",
"MacAddress": "02:42:0a:00:01:06",
"IPv4Address": "10.0.1.6/24",
"IPv6Address": ""
},
"469a013c5ef0ab8c914dbd0aceeecbd522eba314a55648abd0512c0fb844e6c7": {
"Name": "my_stack_things-search.1.x030dszx0c2kdwpptlmfsybbu",
"EndpointID": "eb6fb56aa6d6a5b650e057473cb4576faab98aea3407c53564cae38f3de9b16d",
"MacAddress": "02:42:0a:00:01:0c",
"IPv4Address": "10.0.1.12/24",
"IPv6Address": ""
},
"4bba3cf66c010d629b6fd3ecd7467b9f9a85442363e276d18901e6559247d029": {
"Name": "my_stack_gateway.1.8s3y9sezounm8c0l0sfnp8fcb",
"EndpointID": "62f02710d5f9bf5b861a86c0a348141607167125c10a977d5c1766ca807b8cc7",
"MacAddress": "02:42:0a:00:01:03",
"IPv4Address": "10.0.1.3/24",
"IPv6Address": ""
},
"67b18c17bc245bd547a49c36119f4404d0a15e4d674b681b8bdeb6273aaae364": {
"Name": "my_stack_policies.1.rbgb8unxtoqqg2l0rhw026ppq",
"EndpointID": "96a54ba12d049aebbd4d65caa888676ec009a535fa5bdd713a2d9ab873c9ebd2",
"MacAddress": "02:42:0a:00:01:0f",
"IPv4Address": "10.0.1.15/24",
"IPv6Address": ""
},
"92fe4111b89f0e5a890f877d2d69e33bd0be8ce4765a28a1b8bf7e30811d3c65": {
"Name": "my_stack_swagger-ui.1.xbtbyt3gd63wff5hrph6naj4o",
"EndpointID": "0265305aa5b16b532e304b8f5557e95996e983bfad04bac7e90949c3459c287b",
"MacAddress": "02:42:0a:00:01:0a",
"IPv4Address": "10.0.1.10/24",
"IPv6Address": ""
},
"a46e189a9c4ea4be0bc3cb49213af4d608bdc08995f2f4078d3d287e35333810": {
"Name": "my_stack_mongodb.1.fsatefqvrmwsyo8yw8tp6qgf6",
"EndpointID": "1ca0f13f71fc59027ca925be64c3b11c9a13ce9dc1fca56c77f22ef9f1d4913c",
"MacAddress": "02:42:0a:00:01:13",
"IPv4Address": "10.0.1.19/24",
"IPv6Address": ""
},
"abaf31aa0a26d89630dfc90c0e90fd38fb9ea3dab141855f7820e0d5ed75ad3e": {
"Name": "my_stack_things.1.396rkdghvfvhoxaohkprmyy58",
"EndpointID": "80c8e0ff38da65c9df9db924325743a49311d6e7211d395e0f62b38bd527b7c8",
"MacAddress": "02:42:0a:00:01:15",
"IPv4Address": "10.0.1.21/24",
"IPv6Address": ""
},
"ed73ab9cc3e99314f1e2809fc41eda5c7f700b8f59d129a912cc05659ac73ffa": {
"Name": "my_stack_connectivity.1.njn438mpqzyhgg16r67mrriar",
"EndpointID": "4436d51cf549f07aaf777afa9bf8e310e593ad7cde3df9d4f0395f684a78a1f0",
"MacAddress": "02:42:0a:00:01:11",
"IPv4Address": "10.0.1.17/24",
"IPv6Address": ""
},
"efe84e1eec4843608fb5d950a88b0e63dc81247f2420b75fd4b47293610a74b7": {
"Name": "my_stack_nginx.1.e5c48q82xwb9z8ebapm20pri0",
"EndpointID": "3bf4e453701f08a584d7fed4e24a6953130553cd5741ebd58dac5934d810d71a",
"MacAddress": "02:42:0a:00:01:0d",
"IPv4Address": "10.0.1.13/24",
"IPv6Address": ""
},
"lb-my_stack_ditto-net-swarm": {
"Name": "my_stack_ditto-net-swarm-endpoint",
"EndpointID": "ed13713847ca2649c46e0e4ffe5736485a2d96150c9abf596779a3a053d6ccbe",
"MacAddress": "02:42:0a:00:01:04",
"IPv4Address": "10.0.1.4/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097"
},
"Labels": {
"com.docker.stack.namespace": "my_stack"
},
"Peers": [
{
"Name": "c4f6c67a05a0",
"IP": "192.168.65.3"
}
]
}
]
The logs of my_stack_gateway
: https://pastebin.com/LJrQX5Hw
Hi.
I assume that Ditto will not run on docker swarm out of the box.
I used it 7 years ago or so and back then it required a special DNS based service discovery of the Ditto cluster nodes.
Why would you use docker swarm when just locally starting it on a windows machine?
So, there is no chance to run Ditto on docker swarm.
And regarding why I am running Ditto in Docker Swarm locally :
I ran Ditto locally with regular Docker during development without issues. after finishing development, I tried deploying it on a production server with Docker Swarm, and I faced that issue, so, to troubleshoot it, I set Ditto up locally in docker swarm.
So, there is no chance to run Ditto on docker swarm.
Of course there is. I just can't provide you the exact steps how to do it.
In a former company where I worked, we ran it in Docker swarm.
I see in your docker-compose.yml you have BIND_HOSTNAME=0.0.0.0
You can try to have the specific service name. For example:
BIND_HOSTNAME=policies
This way the pekko-dns discovery method may actually be used correctly:
https://pekko.apache.org/docs/pekko/current/discovery/index.html#discovery-method-dns