Eclipse Ditto on Docker Swarm | nginx: [emerg] host not found in upstream "gateway" in /etc/nginx/nginx.conf:44 | [error] 33#33: *2 connect() failed (111: Connection refused) while connecting to upstream

Question

Eclipse Ditto on Docker Swarm | nginx: [emerg] host not found in upstream "gateway" in /etc/nginx/nginx.conf:44 | [error] 33#33: *2 connect() failed (111: Connection refused) while connecting to upstream

aissam-en opened this issue 2 months ago · comments

Challenges Running Eclipse Ditto on Docker Swarm

I'm trying to run Eclipse Ditto on Docker Swarm, initially, I was able to successfully run Ditto using the provided docker-compose.yml file in Docker (not Swarm mode) without modifying the nginx.conf file.

I made some adjustments to the docker-compose.yml file to work in Docker Swarm :

Commented out restart: always;
Commented out healthcheck section;
Commented out volumes for the swagger-ui service, since volumes are not be suitable for Swarm, I will configure it later;
Replaced volumes for the nginx service with configs, to mount nginx files;

I'm encountering an issue where the nginx service is not recongizing gateway service, here is its logs:

docker service logs -f my_stack_nginx
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Configuration complete; ready for start up
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Configuration complete; ready for start up
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | 2024/06/18 15:44:39 [emerg] 1#1: host not found in upstream "gateway" in /etc/nginx/nginx.conf:44
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | nginx: [emerg] host not found in upstream "gateway" in /etc/nginx/nginx.conf:44

the line 44 from nginx.conf file :

39     ...
40     # connections api using devops user configured in Ditto
41    location /api/2/connections {
42      include nginx-cors.conf;
43
44      proxy_pass                    http://gateway:8080;
45      proxy_http_version            1.1;
46      proxy_set_header              Host                $http_host;
47      proxy_set_header              X-Real-IP           $remote_addr;
48      proxy_set_header              X-Forwarded-For     $proxy_add_x_forwarded_for;
49      proxy_set_header              X-Forwarded-User    $remote_user;
50
51      proxy_set_header Connection  '';
52      chunked_transfer_encoding    off;
53      proxy_buffering              off;
54      proxy_cache                  off;
55    }
56    ...

and when I send request from Postman, ex : GET http://localhost:8080/api/2/things/ with ditto:ditto in Authorization, I get this as output :

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

The nginx logs :

docker service logs -f my_stack_nginx
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Configuration complete; ready for start up
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Configuration complete; ready for start up
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | 2024/06/18 15:44:39 [emerg] 1#1: host not found in upstream "gateway" in /etc/nginx/nginx.conf:44
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | nginx: [emerg] host not found in upstream "gateway" in /etc/nginx/nginx.conf:44
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | 2024/06/18 15:47:55 [error] 33#33: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.2, server: localhost, request: "GET /api/2/things/ HTTP/1.1", upstream: "http://10.0.1.13:8080/api/2/things/", host: "localhost:8080"
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | 10.0.0.2 - ditto [18/Jun/2024:15:47:55 +0000] "GET /api/2/things/ HTTP/1.1" 502 150 "-" "PostmanRuntime/7.39.0"

Additional Information:

Windows 10, 8GB RAM and CPU :
Name : Intel(R) Core(TM) i5-9400H CPU @ 2.50GHz
Number Of Cores : 4
Number Of Logical Processors : 8
Docker :
Client:
Version: 26.1.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.14.0-desktop.1
Path: C:\Program Files\Docker\cli-plugins\docker-buildx.exe
compose: Docker Compose (Docker Inc.)
Version: v2.27.0-desktop.2
Path: C:\Program Files\Docker\cli-plugins\docker-compose.exe
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.29
Path: C:\Program Files\Docker\cli-plugins\docker-debug.exe
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.2
Path: C:\Program Files\Docker\cli-plugins\docker-dev.exe
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.23
Path: C:\Program Files\Docker\cli-plugins\docker-extension.exe
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.4
Path: C:\Program Files\Docker\cli-plugins\docker-feedback.exe
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.1.0
Path: C:\Program Files\Docker\cli-plugins\docker-init.exe
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: C:\Program Files\Docker\cli-plugins\docker-sbom.exe
scout: Docker Scout (Docker Inc.)
Version: v1.8.0
Path: C:\Program Files\Docker\cli-plugins\docker-scout.exe
Server:
Containers: 34
Running: 9
Paused: 0
Stopped: 25
Images: 53
Server Version: 26.1.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: active
NodeID: mahw8hl8i23zzwm7v2i78vaq7
Is Manager: true
ClusterID: ijzjixaizfhrhv754v8ir4fdy
Managers: 1
Nodes: 1
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 192.168.65.3
Manager Addresses:
192.168.65.3:2377
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e377cd56a71523140ca6ae87e30244719194a521
runc version: v1.1.12-0-g51d5e94
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
Kernel Version: 5.15.153.1-microsoft-standard-WSL2
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 3.721GiB
Name: docker-desktop
ID: ec241856-9f66-4d6c-908f-3ff0184e56b5
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=npipe://\.\pipe\docker_cli
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false

The docker-compose.yml file : https://pastebin.com/nbKcXjDc

The services are runnig :

the services are runnig : 
docker service ls
ID             NAME                     MODE         REPLICAS   IMAGE                                PORTS
idgug4xs4u6g   my_stack_connectivity    replicated   1/1        eclipse/ditto-connectivity:latest
16uijjs8vpeb   my_stack_ditto-ui        replicated   1/1        eclipse/ditto-ui:latest
5pd7diz1uz1i   my_stack_gateway         replicated   1/1        eclipse/ditto-gateway:latest         *:8081->8080/tcp
jkkfj363z20o   my_stack_mongodb         replicated   1/1        mongo:6.0                            *:27017->27017/tcp
mc58254lk2h2   my_stack_nginx           replicated   1/1        nginx:1.26-alpine                    *:8080->80/tcp
kn5evgmkpvdw   my_stack_policies        replicated   1/1        eclipse/ditto-policies:latest
t4agxrku1991   my_stack_swagger-ui      replicated   1/1        swaggerapi/swagger-ui:v5.9.1
v42cz47b90qu   my_stack_things          replicated   1/1        eclipse/ditto-things:latest
xyoa5siftouk   my_stack_things-search   replicated   1/1        eclipse/ditto-things-search:latest

Inspect ditto-net-swarm network :

> docker network ls

NETWORK ID     NAME                               DRIVER    SCOPE
e661fefa2fba   bridge                             bridge    local
afa0a7369fed   docker_default                     bridge    local
636762e55a0e   host                               host      local
ipd43285nlar   ingress                            overlay   swarm
miy4fnbluu70   my_stack_ditto-net-swarm           overlay   swarm
4934f4344622   none                               null      local

>docker network inspect my_stack_ditto-net-swarm

[
    {
        "Name": "my_stack_ditto-net-swarm",
        "Id": "miy4fnbluu70bdc30qcqpyqy9",
        "Created": "2024-06-18T16:46:24.45599018Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "22f5721bd207a012953cf4f194abf7ce472ef6b18a6d167df73c4efae26a2ff9": {
                "Name": "my_stack_ditto-ui.1.jcnhcxrkxu9kl8v1a7dwjk4e1",
                "EndpointID": "008daec7000072bb81e7fef018f022a53a86b27a124aa32392c584b3ab479166",
                "MacAddress": "02:42:0a:00:01:06",
                "IPv4Address": "10.0.1.6/24",
                "IPv6Address": ""
            },
            "469a013c5ef0ab8c914dbd0aceeecbd522eba314a55648abd0512c0fb844e6c7": {
                "Name": "my_stack_things-search.1.x030dszx0c2kdwpptlmfsybbu",
                "EndpointID": "eb6fb56aa6d6a5b650e057473cb4576faab98aea3407c53564cae38f3de9b16d",
                "MacAddress": "02:42:0a:00:01:0c",
                "IPv4Address": "10.0.1.12/24",
                "IPv6Address": ""
            },
            "4bba3cf66c010d629b6fd3ecd7467b9f9a85442363e276d18901e6559247d029": {
                "Name": "my_stack_gateway.1.8s3y9sezounm8c0l0sfnp8fcb",
                "EndpointID": "62f02710d5f9bf5b861a86c0a348141607167125c10a977d5c1766ca807b8cc7",
                "MacAddress": "02:42:0a:00:01:03",
                "IPv4Address": "10.0.1.3/24",
                "IPv6Address": ""
            },
            "67b18c17bc245bd547a49c36119f4404d0a15e4d674b681b8bdeb6273aaae364": {
                "Name": "my_stack_policies.1.rbgb8unxtoqqg2l0rhw026ppq",
                "EndpointID": "96a54ba12d049aebbd4d65caa888676ec009a535fa5bdd713a2d9ab873c9ebd2",
                "MacAddress": "02:42:0a:00:01:0f",
                "IPv4Address": "10.0.1.15/24",
                "IPv6Address": ""
            },
            "92fe4111b89f0e5a890f877d2d69e33bd0be8ce4765a28a1b8bf7e30811d3c65": {
                "Name": "my_stack_swagger-ui.1.xbtbyt3gd63wff5hrph6naj4o",
                "EndpointID": "0265305aa5b16b532e304b8f5557e95996e983bfad04bac7e90949c3459c287b",
                "MacAddress": "02:42:0a:00:01:0a",
                "IPv4Address": "10.0.1.10/24",
                "IPv6Address": ""
            },
            "a46e189a9c4ea4be0bc3cb49213af4d608bdc08995f2f4078d3d287e35333810": {
                "Name": "my_stack_mongodb.1.fsatefqvrmwsyo8yw8tp6qgf6",
                "EndpointID": "1ca0f13f71fc59027ca925be64c3b11c9a13ce9dc1fca56c77f22ef9f1d4913c",
                "MacAddress": "02:42:0a:00:01:13",
                "IPv4Address": "10.0.1.19/24",
                "IPv6Address": ""
            },
            "abaf31aa0a26d89630dfc90c0e90fd38fb9ea3dab141855f7820e0d5ed75ad3e": {
                "Name": "my_stack_things.1.396rkdghvfvhoxaohkprmyy58",
                "EndpointID": "80c8e0ff38da65c9df9db924325743a49311d6e7211d395e0f62b38bd527b7c8",
                "MacAddress": "02:42:0a:00:01:15",
                "IPv4Address": "10.0.1.21/24",
                "IPv6Address": ""
            },
            "ed73ab9cc3e99314f1e2809fc41eda5c7f700b8f59d129a912cc05659ac73ffa": {
                "Name": "my_stack_connectivity.1.njn438mpqzyhgg16r67mrriar",
                "EndpointID": "4436d51cf549f07aaf777afa9bf8e310e593ad7cde3df9d4f0395f684a78a1f0",
                "MacAddress": "02:42:0a:00:01:11",
                "IPv4Address": "10.0.1.17/24",
                "IPv6Address": ""
            },
            "efe84e1eec4843608fb5d950a88b0e63dc81247f2420b75fd4b47293610a74b7": {
                "Name": "my_stack_nginx.1.e5c48q82xwb9z8ebapm20pri0",
                "EndpointID": "3bf4e453701f08a584d7fed4e24a6953130553cd5741ebd58dac5934d810d71a",
                "MacAddress": "02:42:0a:00:01:0d",
                "IPv4Address": "10.0.1.13/24",
                "IPv6Address": ""
            },
            "lb-my_stack_ditto-net-swarm": {
                "Name": "my_stack_ditto-net-swarm-endpoint",
                "EndpointID": "ed13713847ca2649c46e0e4ffe5736485a2d96150c9abf596779a3a053d6ccbe",
                "MacAddress": "02:42:0a:00:01:04",
                "IPv4Address": "10.0.1.4/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097"
        },
        "Labels": {
            "com.docker.stack.namespace": "my_stack"
        },
        "Peers": [
            {
                "Name": "c4f6c67a05a0",
                "IP": "192.168.65.3"
            }
        ]
    }
]

The logs of my_stack_gateway : https://pastebin.com/LJrQX5Hw

Thomas Jaeckle · Answer 1 · Thu Jun 20 2024 02:31:44 GMT+0800 (China Standard Time)

Hi.
I assume that Ditto will not run on docker swarm out of the box.
I used it 7 years ago or so and back then it required a special DNS based service discovery of the Ditto cluster nodes.

Why would you use docker swarm when just locally starting it on a windows machine?

Aissam EN-NAHEL · Answer 2 · Thu Jun 20 2024 05:52:56 GMT+0800 (China Standard Time)

So, there is no chance to run Ditto on docker swarm.

And regarding why I am running Ditto in Docker Swarm locally :
I ran Ditto locally with regular Docker during development without issues. after finishing development, I tried deploying it on a production server with Docker Swarm, and I faced that issue, so, to troubleshoot it, I set Ditto up locally in docker swarm.

Thomas Jaeckle · Answer 3 · Thu Jun 20 2024 14:16:08 GMT+0800 (China Standard Time)

So, there is no chance to run Ditto on docker swarm.

Of course there is. I just can't provide you the exact steps how to do it.
In a former company where I worked, we ran it in Docker swarm.

Kalin Kostashki · Answer 4 · Thu Jun 20 2024 15:09:57 GMT+0800 (China Standard Time)

I see in your docker-compose.yml you have BIND_HOSTNAME=0.0.0.0
You can try to have the specific service name. For example:
BIND_HOSTNAME=policies

This way the pekko-dns discovery method may actually be used correctly:

https://pekko.apache.org/docs/pekko/current/discovery/index.html#discovery-method-dns