`deis logs` fails on minikube
mboersma opened this issue · comments
Kubernetes: 1.3.5
minikube: v0.9.0, xhyve driver
host: macOS Sierra v10.12 beta
Deis Workflow: v2.5.0 Chart
The e2e test specs covering deis logs fail, as it does interactively:
$ deis info
=== joyful-vacation Application
updated: 2016-09-07T20:20:32Z
uuid: 7c350483-e08e-4a53-8075-7523f2d28c0a
created: 2016-09-07T20:16:43Z
url: joyful-vacation.192.168.64.26.nip.io
owner: matt
id: joyful-vacation
=== joyful-vacation Processes
--- web:
joyful-vacation-web-4037832973-5ey1j up (v2)
joyful-vacation-web-4037832973-cnjwy up (v2)
joyful-vacation-web-4037832973-gu2yh up (v2)
joyful-vacation-web-4037832973-ujnxi up (v2)
joyful-vacation-web-4037832973-ukxb5 up (v2)
=== joyful-vacation Domains
joyful-vacation
$ deis logs
Error: There are currently no log messages. Please check the following things:
...If it's possible to accomodate the single-node minikube test k8s environment, we should do so.
From inside the fluentd container, the symlinked log files don't exist:
root@deis-logger-fluentd-nebvp:/# ls -al /var/log/containers/nearby*
lrwxrwxrwx 1 root root 174 Sep 13 17:58 /var/log/containers/nearby-rabbitry-web-1965060662-x2qe6_nearby-rabbitry_POD-ff373bf36f21893880654b9a5bd41040fca54a12c64ad868972cc20fb13e6f7a.log -> /mnt/sda1/var/lib/docker/containers/ff373bf36f21893880654b9a5bd41040fca54a12c64ad868972cc20fb13e6f7a/ff373bf36f21893880654b9a5bd41040fca54a12c64ad868972cc20fb13e6f7a-json.log
lrwxrwxrwx 1 root root 174 Sep 13 17:58 /var/log/containers/nearby-rabbitry-web-1965060662-x2qe6_nearby-rabbitry_nearby-rabbitry-web-8ecd5da18f5b783e27d8b66555481d1065108f7f1bc2e41e4930b2019820c96e.log -> /mnt/sda1/var/lib/docker/containers/8ecd5da18f5b783e27d8b66555481d1065108f7f1bc2e41e4930b2019820c96e/8ecd5da18f5b783e27d8b66555481d1065108f7f1bc2e41e4930b2019820c96e-json.log
root@deis-logger-fluentd-nebvp:/# cat /mnt/sda1/var/lib/docker/containers/ff373bf36f21893880654b9a5bd41040fca54a12c64ad868972cc20fb13e6f7a/ff373bf36f21893880654b9a5bd41040fca54a12c64ad868972cc20fb13e6f7a-json.log
cat: /mnt/sda1/var/lib/docker/containers/8ecd5da18f5b783e27d8b66555481d1065108f7f1bc2e41e4930b2019820c96e/8ecd5da18f5b783e27d8b66555481d1065108f7f1bc2e41e4930b2019820c96e-json.log: No such file or directory
root@deis-logger-fluentd-nebvp:/# Although they do exist on the host:
docker@boot2docker:~$ sudo cat /mnt/sda1/var/lib/docker/containers/8ecd5da18f5b783e27d8b66555481d1065108f7f1bc2e41e
4930b2019820c96e/8ecd5da18f5b783e27d8b66555481d1065108f7f1bc2e41e4930b2019820c96e-json.log
{"log":"[2016-09-13 17:58:10 +0000] [1] [INFO] Starting gunicorn 19.6.0\n","stream":"stderr","time":"2016-09-13T17:58:10.058972481Z"}
{"log":"[2016-09-13 17:58:10 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)\n","stream":"stderr","time":"2016-09-13T17:58:10.059739267Z"}
{"log":"[2016-09-13 17:58:10 +0000] [1] [INFO] Using worker: sync\n","stream":"stderr","time":"2016-09-13T17:58:10.05980172Z"}
{"log":"[2016-09-13 17:58:10 +0000] [29] [INFO] Booting worker with pid: 29\n","stream":"stderr","time":"2016-09-13T17:58:10.06381675Z"}
{"log":"Not Found: /favicon.ico\n","stream":"stderr","time":"2016-09-13T18:02:20.410486541Z"}
Alright so in the chart we mount /var/lib/docker into the same path. But we do not search that path for log files. We should figure out how to better handle this.
Similarly to #50, I worked around this by amending the fluentd daemon mount to /mnt/sda1/var/lib/docker/containers, the host path:
$ cat ~/.helmc/workspace/charts/workflow-dev/manifests/deis-logger-fluentd-daemon.yaml
#helm:generate helm tpl -d $HELM_GENERATE_DIR/tpl/generate_params.toml -o $HELM_GENERATE_DIR/manifests/deis-logger-fluentd-daemon.yaml $HELM_GENERATE_FILE
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: deis-logger-fluentd
namespace: deis
labels:
heritage: deis
annotations:
component.deis.io/version: canary
spec:
selector:
matchLabels:
app: deis-logger-fluentd
heritage: deis
template:
metadata:
name: deis-logger-fluentd
labels:
heritage: deis
app: deis-logger-fluentd
spec:
serviceAccount: deis-logger-fluentd
containers:
- name: deis-logger-fluentd
image: quay.io/deisci/fluentd:canary
imagePullPolicy: Always
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /mnt/sda1/var/lib/docker/containers
readOnly: true
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /mnt/sda1/var/lib/docker/containers$ deis logs
INFO [orange-chinbone]: config orange-chinbone-94b61e9 updated
INFO [orange-chinbone]: mboersma created initial release
INFO [orange-chinbone]: appsettings orange-chinbone-4406a03 updated
INFO [orange-chinbone]: domain orange-chinbone added
INFO [orange-chinbone]: build orange-chinbone-4a5e377 created
INFO [orange-chinbone]: mboersma deployed c1e2e65
INFO [orange-chinbone]: buildpack type detected. Defaulting to $PORT 5000
INFO [orange-chinbone]: adding 30s on to the original 120s timeout to account for the initial delay specified in the liveness / readiness probe
INFO [orange-chinbone]: This deployments overall timeout is 150s - batch timout is 150s and there are 1 batches to deploy with a total of 1 pods
INFO [orange-chinbone]: waited 10s and 1 pods are in service
INFO [orange-chinbone]: waited 20s and 1 pods are in service
INFO [orange-chinbone]: waited 30s and 1 pods are in serviceBehavior is the same with minkube 0.10.0.
Running these commands inside the fluentd container also repairs the symlinking so deis logs works:
$ mkdir -p /mnt/sda1/var/lib/docker
$ ln -s /var/lib/docker/containers /mnt/sda1/var/lib/docker/containersI wonder if I could automate that somehow before fluentd starts, in a backward-compatible way.