Can't deploy on Docker Swarm

Question

Can't deploy on Docker Swarm

asijoumi opened this issue 2 months ago · comments

NOTE

If this case is urgent, please subscribe to Subnet so that our 24/7 support team may help you faster.

Hi ! I used the docker-compose.yaml located on the documentation (https://github.com/minio/minio/blob/master/docs/orchestration/docker-compose/docker-compose.yaml) and change it a little bit. But I was unable to used it on docker swarm.

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

docker-compose.yml

version: '3.7'

x-minio-common: &minio-common
  image: quay.io/minio/minio:RELEASE.2024-04-18T19-09-19Z
  command: server --console-address ":9001" http://minio{1...4}/data
  healthcheck:
    test: ["CMD", "mc", "ready", "local"]
    interval: 5s
    timeout: 5s
    retries: 5

services:
  minio1:
    <<: *minio-common
    volumes:
      - minio1:/data
    networks:
      minio:
        aliases:
          - minio1
  minio2:
    <<: *minio-common
    volumes:
      - minio2:/data
    networks:
      minio:
        aliases:
          - minio2
  minio3:
    <<: *minio-common
    volumes:
      - minio3:/data
    networks:
      minio:
        aliases:
          - minio3
  minio4:
    <<: *minio-common
    volumes:
      - minio4:/data
    networks:
      minio:
        aliases:
          - minio4

volumes:
  minio1:
  minio2:
  minio3:
  minio4:

networks:
  minio:
    driver: overlay

To easily test it, you can use https://labs.play-with-docker.com

docker stack deploy -c minio.yaml minio

Context

Regression

Your Environment

Version used (minio --version):
Server setup and configuration:
Operating System and version (uname -a):

jiuker · Answer 1 · Thu Apr 25 2024 14:52:32 GMT+0800 (China Standard Time)

What's the error log ? @asijoumi

Aimen SIJOUMI · Answer 2 · Thu Apr 25 2024 15:38:00 GMT+0800 (China Standard Time)

Hi @jiuker, I have this :

minio_minio1.1.r19hdu824jv3@worker2    | Error: Unable to resolve DNS for http://minio3/data: lookup minio3 on 127.0.0.11:53: no such host (*fmt.wrapError)
minio_minio1.1.r19hdu824jv3@worker2    |        host="minio3", elapsedTime="1 second elapsed"
minio_minio1.1.r19hdu824jv3@worker2    |       10: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
minio_minio1.1.r19hdu824jv3@worker2    |        9: internal/logger/logonce.go:149:logger.LogOnceIf()
minio_minio1.1.r19hdu824jv3@worker2    |        8: cmd/logging.go:82:cmd.bootLogOnceIf()
minio_minio1.1.r19hdu824jv3@worker2    |        7: cmd/endpoint.go:869:cmd.PoolEndpointList.UpdateIsLocal()
minio_minio1.1.r19hdu824jv3@worker2    |        6: cmd/endpoint.go:1007:cmd.CreatePoolEndpoints()
minio_minio1.1.r19hdu824jv3@worker2    |        5: cmd/endpoint-ellipses.go:508:cmd.createServerEndpoints()
minio_minio1.1.r19hdu824jv3@worker2    |        4: cmd/server-main.go:344:cmd.serverHandleCmdArgs()
minio_minio1.1.r19hdu824jv3@worker2    |        3: cmd/server-main.go:687:cmd.serverMain.func2()
minio_minio1.1.r19hdu824jv3@worker2    |        2: cmd/server-main.go:512:cmd.bootstrapTrace()
minio_minio1.1.r19hdu824jv3@worker2    |        1: cmd/server-main.go:683:cmd.serverMain()

jiuker · Answer 3 · Thu Apr 25 2024 19:41:07 GMT+0800 (China Standard Time)

I think that's network issue. Not for minio.

Aimen SIJOUMI · Answer 4 · Thu Apr 25 2024 19:58:08 GMT+0800 (China Standard Time)

It seems strange that the exemple of the documentation won't work with Docker Swarm...

Ok, do you have an exemple of deployment of minio with docker-compose ?

jiuker · Answer 5 · Thu Apr 25 2024 20:18:02 GMT+0800 (China Standard Time)

It seems strange that the exemple of the documentation won't work with Docker Swarm...

Ok, do you have an exemple of deployment of minio with docker-compose ?

You mentioned one could be ok.

Aimen SIJOUMI · Answer 6 · Thu Apr 25 2024 21:42:08 GMT+0800 (China Standard Time)

@jiuker, it is not woking ...

I copied the docker-compose.yml from the documentation :

version: '3.7'

# Settings and configurations that are common for all containers
x-minio-common: &minio-common
  image: quay.io/minio/minio:RELEASE.2024-04-18T19-09-19Z
  command: server --console-address ":9001" http://minio{1...4}/data{1...2}
  expose:
    - "9000"
    - "9001"
  # environment:
    # MINIO_ROOT_USER: minioadmin
    # MINIO_ROOT_PASSWORD: minioadmin
  healthcheck:
    test: ["CMD", "mc", "ready", "local"]
    interval: 5s
    timeout: 5s
    retries: 5

# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
  minio1:
    <<: *minio-common
    hostname: minio1
    volumes:
      - data1-1:/data1
      - data1-2:/data2

  minio2:
    <<: *minio-common
    hostname: minio2
    volumes:
      - data2-1:/data1
      - data2-2:/data2

  minio3:
    <<: *minio-common
    hostname: minio3
    volumes:
      - data3-1:/data1
      - data3-2:/data2

  minio4:
    <<: *minio-common
    hostname: minio4
    volumes:
      - data4-1:/data1
      - data4-2:/data2

  nginx:
    image: nginx:1.19.2-alpine
    hostname: nginx
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "9000:9000"
      - "9001:9001"
    depends_on:
      - minio1
      - minio2
      - minio3
      - minio4

## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration.
volumes:
  data1-1:
  data1-2:
  data2-1:
  data2-2:
  data3-1:
  data3-2:
  data4-1:
  data4-2:

Docker compose and Docker Swarm are not the same. So you do have an example for Docker Swarm ?

jiuker · Answer 7 · Thu Apr 25 2024 21:50:57 GMT+0800 (China Standard Time)

@jiuker, it is not woking ...

I copied the docker-compose.yml from the documentation :

version: '3.7'

# Settings and configurations that are common for all containers
x-minio-common: &minio-common
  image: quay.io/minio/minio:RELEASE.2024-04-18T19-09-19Z
  command: server --console-address ":9001" http://minio{1...4}/data{1...2}
  expose:
    - "9000"
    - "9001"
  # environment:
    # MINIO_ROOT_USER: minioadmin
    # MINIO_ROOT_PASSWORD: minioadmin
  healthcheck:
    test: ["CMD", "mc", "ready", "local"]
    interval: 5s
    timeout: 5s
    retries: 5

# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
  minio1:
    <<: *minio-common
    hostname: minio1
    volumes:
      - data1-1:/data1
      - data1-2:/data2

  minio2:
    <<: *minio-common
    hostname: minio2
    volumes:
      - data2-1:/data1
      - data2-2:/data2

  minio3:
    <<: *minio-common
    hostname: minio3
    volumes:
      - data3-1:/data1
      - data3-2:/data2

  minio4:
    <<: *minio-common
    hostname: minio4
    volumes:
      - data4-1:/data1
      - data4-2:/data2

  nginx:
    image: nginx:1.19.2-alpine
    hostname: nginx
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "9000:9000"
      - "9001:9001"
    depends_on:
      - minio1
      - minio2
      - minio3
      - minio4

## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration.
volumes:
  data1-1:
  data1-2:
  data2-1:
  data2-2:
  data3-1:
  data3-2:
  data4-1:
  data4-2:

Docker compose and Docker Swarm are not the same. So you do have an example for Docker Swarm ?

You need to check your host. Like firework or others.

Klaus Post · Answer 8 · Thu Apr 25 2024 22:02:43 GMT+0800 (China Standard Time)

It seems like something is wrong with your environment, since minio1 hosts cannot be resolved.

Aimen SIJOUMI · Answer 9 · Thu Apr 25 2024 22:23:30 GMT+0800 (China Standard Time)

It's strange ... You can test this example :

version: '3.7'

x-minio-common: &minio-common
  image: quay.io/minio/minio:RELEASE.2024-04-18T19-09-19Z
  command: server --console-address ":9001" http://minio_minio{1...4}/data
  networks:
    - minio
  healthcheck:
    test: ["CMD", "mc", "ready", "local"]
    interval: 5s
    timeout: 5s
    retries: 5

services:
  minio1:
    <<: *minio-common
    volumes:
      - minio1:/data
  minio2:
    <<: *minio-common
    volumes:
      - minio2:/data
  minio3:
    <<: *minio-common
    volumes:
      - minio3:/data
  minio4:
    <<: *minio-common
    volumes:
      - minio4:/data

  nginx:
    image: nginx:1.19.2-alpine
    networks:
      - minio
    depends_on:
      - minio1
      - minio2
      - minio3
      - minio4

volumes:
  minio1:
  minio2:
  minio3:
  minio4:

networks:
  minio:
    driver: overlay
    attachable: true

After that run :

docker run --rm -it --network=minio_minio node:18-alpine sh

Then :

apk add curl

And :

curl http://minio_nginx

We got an answer from nginx .. It's the same docker stack. Docker was able to resolve the nginx domain name but not the minio ?

Harshavardhana · Answer 10 · Thu Apr 25 2024 23:53:07 GMT+0800 (China Standard Time)

The DNS issues are common in docker swarm. Use k8s with a kind cluster much easier and more robust.

Docker is good at containers not at orchestration.