How to use template stanza

[Martin Buchleitner]

I reconfigured tasks from docker to containerd driver without any big problems - except one: How to configure the mount options when using a template stanza for configuration, e.g.:

task "mosquitto" {
  driver = "docker"
  config {
    image = "docker.io/eclipse-mosquitto:2"
    ports = ["mqtt", "wss"]
    volumes = [
     "local/mosquitto.conf:/mosquitto/config/mosquitto.conf",
      "/srv/nomad/mosquitto:/mosquitto/data",
    ]
  }
  template {
      destination = "local/mosquitto.conf"
      data = <<EOF
bind_address 0.0.0.0
allow_anonymous true
persistence true
persistence_location /mosquitto/data/
log_dest stdout
EOF
  }
}

[Shishir]

@mabunixda
Is this file local/mosquitto.conf created on the host, and you want to mount it into the container root filesystem at /mosquitto/config/mosquitto.conf?

[Martin Buchleitner]

The file local/mosqitto.conf is generated by the template stanza. The docker version of this task mounts the actial file from the working directory of the task. How can I achieve the same behavior wit the containerd implementation?
I also use the template stanza with dynamic data that gets injected into these templated configuration sections, not only static data like this sample.

[Shishir]

@mabunixda
The problem is when you try to mount local/mosquitto.conf (This is not an absolute path, but a relative path), containerd will look for this file relative to it's chroot'ed environment and not relative to Nomad task directory.

Since Nomad is dropping the file in it's sandbox environment and containerd is looking in it's own chroot'ed environment (where the file is not present), the mount fails.

e.g. containerd is looking here (When I tested this in my local vagrant VM)

ERROR: stat /run/containerd/io.containerd.runtime.v2.task/nomad/msq-task-e1dde8d8-8f8f-a3e6-76e8-a820ec776b09/local/mosquitto.conf: no such file or directory

What I can do is add a patch to containerd-driver which will allow you to specify relative paths e.g. local/mosquitto.conf. If the relative path starts with local, containerd-driver will look for the file in nomad sandbox environment instead of containerd chroot'ed environment. It will then be able to locate the file and mount it into the container rootfs.

Currently it only works with absolute paths. If you specify an absolute path, containerd-driver will look for that path on the host.

You can then use a job like this to mount your file:

job "mosquito" {
  datacenters = ["dc1"]

  group "msq-group" {
    task "msq-task" {
      driver = "containerd-driver"

      config {
        image           = "ubuntu:16.04"
        command         = "sleep"
        args            = ["600s"]
        privileged      = true
        mounts = [
           {
                type = "bind"
                target = "/mosquitto/config/mosquitto.conf"
                source = "local/mosquitto.conf"
                options = ["rbind", "rw"]
           }
        ]
      }

      template {
        destination = "local/mosquitto.conf"
        data = <<EOF
bind_address 0.0.0.0
allow_anonymous true
persistence true
persistence_location /mosquitto/data/
log_dest stdout
EOF
      }

      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

The patch will be something like this:

root@vagrant:~/go/src/github.com/Roblox/nomad-driver-containerd# git diff
diff --git a/containerd/containerd.go b/containerd/containerd.go
index 8ee3878..2260f70 100644
--- a/containerd/containerd.go
+++ b/containerd/containerd.go
@@ -250,6 +250,11 @@ func (d *Driver) createContainer(containerConfig *ContainerConfig, config *TaskC
                        return nil, fmt.Errorf("Options cannot be empty for mount type: %s. You need to atleast pass rbind and ro.", mount.Type)
                }

+               if mount.Type == "bind" && strings.HasPrefix(mount.Source, "local") {
+                       mount.Source = containerConfig.TaskDirSrc + mount.Source[5:]
+                       d.logger.Info(mount.Source)
+               }
+
                m := buildMountpoint(mount.Type, mount.Target, mount.Source, mount.Options)
                mounts = append(mounts, m)
        }

From the container:

root@vagrant:~/go/src/github.com/Roblox/nomad-driver-containerd# nomad alloc exec -i -t d4720194 cat /mosquitto/config/mosquitto.conf
bind_address 0.0.0.0
allow_anonymous true
persistence true
persistence_location /mosquitto/data/
log_dest stdout

Let me know if this works for you, and I ll get the patch merged.

[Martin Buchleitner]

That would be awesome!

[lu-zen]

Hey

There's a easier way to use nomad's task folder env var inside the mount config?
Something like:

           {
                type = "bind"
                source = "$NOMAD_TASK_DIR/traefik.toml"
                target = "/etc/traefik/traefik.toml"
                options = ["rbind", "ro"]
           }

[Shishir]

@mabunixda Sorry I was on vacation last 3 weeks, and didn't get a chance to check this out.
Let me work on the PR and will provide an update shortly.

Did you get a chance to try out @lu-zen suggestion?

[Martin Buchleitner]

I tried it now, but this does not work. This results in
/run/containerd/io.containerd.runtime.v2.task/nomad/plugin-949c0e0a-c4c6-385f-2a47-2eac590ca507/$NOMAD_TASK_DIR/traefik.toml is not found - of course ...

[Shishir]

@mabunixda Thanks for confirming. I ll open a PR with the fix I suggested here