Docker Swarm "Unable to create file "

[Julien Chastang]

Posting on behalf of user. Also ping @isuftin.

---

I have successfully run your THREDDS docker container on my local machine. Everything seems fine. However, when I try to run multiple instances in a docker swarm, I have an issue where the containers close every few minutes and re-launch.

I am following the instructions found here:
https://github.com/Unidata/thredds-docker/blob/master/README_SWARM.md
and I used the docker-compose-swarm.yml file that is referred to on that page.

In addition to those instructions, I have made a number of changes in attempts to get the system working. For example, I have tried removing the drain option in the call to set up the manager. I have also tried modifying docker-compose-swarm.yml in a variety of ways. When I run docker ps on the service, I get the error message:

task: non-zero exit (143): dockerexec: unhealthy container

I have pasted the output from docker service logs at the end of this message. Also, after many attempts, I have modified your suggested docker-compose-swarm.yml as follows:

version: '3.3'

secrets:
  thredds_tomcat_users:
    file: ./tomcat-users.xml

configs:
  thredds_tomcat_users:
    file: ./tomcat-users.xml

networks:
  thredds_network:

volumes:
  thredds_volume:
    labels:
      - "docker.volume.description=THREDDS Docker Volume"
      - "docker.volume.name=volume.thredds"
      - /home/mt/threddstest/:/usr/local/tomcat/logs/
      - /home/mt/threddstest/:/usr/local/tomcat/content/thredds/logs/
      - /home/mt/threddstest/tomcat-users.xml:/usr/local/tomcat/conf/tomcat-users.xml
      - /home/mt/Desktop/v2_results/holder/:/usr/local/tomcat/content/thredds
      - /home/mt/Desktop/v2_results/holder/:/path/to/your/data/directory1

services:
  thredds:
    image: unidata/thredds-docker:latest
    networks:
        thredds_network:
          aliases:
            - thredds
    volumes:
      - "thredds_volume:/usr/local/tomcat/logs/"
      - "thredds_volume:/usr/local/tomcat/content/thredds/logs/"
      - "thredds_volume:/usr/local/tomcat/conf/Catalina/localhost"
      - "thredds_volume:/usr/local/tomcat/work/Catalina/localhost/thredds"
    ports:
      - "80:8080"
      - "443:8443"
      - "8443:8443"
    env_file:
      - "swarm_compose.env"
    secrets:
      - source: thredds_tomcat_users
        target: /usr/local/tomcat/conf/tomcat-users.xml
        uid: '1000'
    healthcheck:
      test: "NONE"

Really, I have two questions:

1. I want to put a THREDDS server on an amazon ec2 instance. How much traffic / use is a single docker container able to handle? In other words, is it necessary for me to set up a docker swarm or can I just use the single docker container version that is already working on my laptop?

2. What am I doing wrong? Please see the logs below. The errors seem to indicate that THREDDS cannot access the file system outside the docker containers. For example:
   ERROR Unable to create file /home/mt/Desktop/v2_results/holder///thredds/logs/threddsServlet.log java.io.IOException: Could not create directory /home/mt/Desktop/v2_results/holder/thredds/logs
   and localhost-startStop-1 ERROR Unable to locate appender "catalogInit" for logger config

I have tried to creating the local folders referenced in the above docker-compose-swarm.yml file on each of the docker containers in the swarm. That does not help either.

I am excited to use the THREDDS server. It is a great piece of software. Any help you can offer would be greatly appreciated.

Here is the output of the docker service logs:

task: non-zero exit (143): dockerexec: unhealthy container

The full log is captured in this gist

[Ivan Suftin]

I haven't tested this locally but a few things jump out at me. First you're mapping /home/mt/threddstest/ twice. This could be an issue? It does mention that /usr/local/tomcat/conf/tomcat-users.xml is on a read-only filesystem which definitely can be an issue because later on I see ERROR FileManager (/home/mt/Desktop/v2_results/holder///thredds/logs/serverStartup.log) java.io.IOException: Could not create directory /home/mt/Desktop/v2_results/holder/thredds/logs java.io.IOException: Could not create directory /home/mt/Desktop/v2_results/holder/thredds/logs

It looks like the rest of the stack trace stems from that issue.

One thing to note is that in swarm, you really shouldn't map to a local volume because of the dynamic nature of the location of the container and the inability to predict on which machine it pops up. I am assuming that you're running into container <-> host permission issues and you might try removing the mappings back to your host for the volumes and see if it works that way.

Also, you don't necessarily have to use Swarm to run this. If not using swarm, you will want to change how files get into the container (and probably then start actual mapping to the host) because the "secrets" section isnt valid outside of swarm.

I can take a look at this later today or tomorrow as time allows.

As far as overhead is concerned, there's no one good answer but a number of articles online do touch on that: https://www.percona.com/blog/2016/02/11/measuring-docker-io-overhead/

[Ivan Suftin]

I've tested the base setup for swarm as it stands in the main repo and it seems to be working as expected. I do think that the main issue is using a docker volume that is backed by the host file system to mount swarm service directories to.

There are workarounds (like using an NFS mount) if something like that is desired, but in the end there is no mandate to use Swarm. If clustering isn't needed, you're better off not using it. If clustering is needed, Swarm is an option but binding services to the local filesystem becomes problematic.

https://docs.docker.com/engine/swarm/services/#bind-mounts

[mtercek]

Thank you for the help. You've answered my main question. I will just run a single docker container on a single EC2 instance and bind to the local ec2 file system. That would be essentially what I have working properly on my local machine. Upon more investigation, it seems that the tomcat server inside the docker container I am using can handle around 200 simultaneous connections, which should be plenty for my use case:

https://stackoverflow.com/questions/18372464/how-many-concurrent-request-can-tomcat-handle-by-default

I don't think it is feasible to put the data served by the THREDDS server inside the docker container because the size is around 3TB and will change routinely.

I will be able to get this running, but it still unclear to me how I would create a swarm of THREDDS docker containers that all looked at an independent file system, whether that be an NFS or locally hosted drive. Were you suggesting that the data read by THREDDS should be in the docker container? If it isn't too much trouble, do you have a working example of a docker-compose-swarm.yml that would do this? Maybe with an NFS mount?

Thanks again.