Mesos container logs aggregation
frankscholten opened this issue · comments
With "EXECUTOR_FILE_PATH": "/tmp/mesos/slaves/*/frameworks/*/executors/*/runs/latest/stdout, /tmp/mesos/slaves/*/frameworks/*/executors/*/runs/latest/stderr, /var/log/*.INFO, /var/log/*.WARNING" docker inspect of logstash executors is
MacBook-Pro-2:temp vik$ docker inspect dd2e3bd2ae4f
[
{
"Id": "dd2e3bd2ae4f756c17e95e35118deb52254e07a489b58e07e09ad68e55125738",
"Created": "2016-01-20T14:08:30.205507381Z",
"Path": "/tmp/start-executor.sh",
"Args": [
"dummyArgument"
],
"State": {
"Status": "running",
....
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "cd0a3f5d3304308ab0c6677d10f0d1c771840763a31728fdbd044f025f64298a",
"ResolvConfPath": "/mnt/sda1/var/lib/docker/containers/dd2e3bd2ae4f756c17e95e35118deb52254e07a489b58e07e09ad68e55125738/resolv.conf",
"HostnamePath": "/mnt/sda1/var/lib/docker/containers/dd2e3bd2ae4f756c17e95e35118deb52254e07a489b58e07e09ad68e55125738/hostname",
"HostsPath": "/mnt/sda1/var/lib/docker/containers/dd2e3bd2ae4f756c17e95e35118deb52254e07a489b58e07e09ad68e55125738/hosts",
"LogPath": "/mnt/sda1/var/lib/docker/containers/dd2e3bd2ae4f756c17e95e35118deb52254e07a489b58e07e09ad68e55125738/dd2e3bd2ae4f756c17e95e35118deb52254e07a489b58e07e09ad68e55125738-json.log",
"Name": "/mesos-e0baf934-fffe-426b-b576-b450aec02c78-S2.916d49cb-30c8-40ed-9551-5425bd3296b8",
"RestartCount": 0,
....
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/:/logstashpaths:ro",
"/tmp/mesos/slaves/e0baf934-fffe-426b-b576-b450aec02c78-S2/frameworks/e0baf934-fffe-426b-b576-b450aec02c78-0002/executors/executor.0a516626-4495-4f84-a59e-fa7429100fb6/runs/916d49cb-30c8-40ed-9551-5425bd3296b8:/mnt/mesos/sandbox"
],
"ContainerIDFile": "",
....
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"SecurityOpt": null,
"ReadonlyRootfs": false,
"Ulimits": null,
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"CgroupParent": "",
"ConsoleSize": [
0,
0
],
"VolumeDriver": ""
},
"GraphDriver": {
"Name": "aufs",
"Data": null
},
"Mounts": [
{
"Source": "/",
"Destination": "/logstashpaths",
"Mode": "ro",
"RW": false
},
{
"Source": "/tmp/mesos/slaves/e0baf934-fffe-426b-b576-b450aec02c78-S2/frameworks/e0baf934-fffe-426b-b576-b450aec02c78-0002/executors/executor.0a516626-4495-4f84-a59e-fa7429100fb6/runs/916d49cb-30c8-40ed-9551-5425bd3296b8",
"Destination": "/mnt/mesos/sandbox",
"Mode": "",
"RW": true
}
],
"Config": {
"Hostname": "dd2e3bd2ae4f",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": true,
"AttachStderr": true,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"LIBPROCESS_PORT=0",
"MESOS_CHECKPOINT=1",
"MESOS_DIRECTORY=/tmp/mesos/slaves/e0baf934-fffe-426b-b576-b450aec02c78-S2/frameworks/e0baf934-fffe-426b-b576-b450aec02c78-0002/executors/executor.0a516626-4495-4f84-a59e-fa7429100fb6/runs/916d49cb-30c8-40ed-9551-5425bd3296b8",
"MESOS_EXECUTOR_ID=executor.0a516626-4495-4f84-a59e-fa7429100fb6",
"MESOS_FRAMEWORK_ID=e0baf934-fffe-426b-b576-b450aec02c78-0002",
"MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/libmesos-0.25.0.so",
"MESOS_NATIVE_LIBRARY=/usr/lib/libmesos-0.25.0.so",
"MESOS_RECOVERY_TIMEOUT=15mins",
"MESOS_SLAVE_ID=e0baf934-fffe-426b-b576-b450aec02c78-S2",
"MESOS_SLAVE_PID=slave(1)@172.17.0.7:5051",
"JAVA_OPTS=-Xmx128m -Dmesos.logstash.logstash.heap.size=256m",
"MESOS_SANDBOX=/mnt/mesos/sandbox",
"MESOS_CONTAINER_NAME=mesos-e0baf934-fffe-426b-b576-b450aec02c78-S2.916d49cb-30c8-40ed-9551-5425bd3296b8",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"dummyArgument"
],
"Image": "mesos/logstash-executor:latest",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": [
"/tmp/start-executor.sh"
],
"OnBuild": null,
"Labels": {}
},
"NetworkSettings": {
"Bridge": "",
.....
"IPv6Gateway": "",
"MacAddress": "02:42:ac:11:00:0b",
"Networks": {
"bridge": {
"EndpointID": "94d61157a2729b5cf372525a544c47df79069b2c63ead470c30b80bdaedb3de6",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.11",
....
"MacAddress": "02:42:ac:11:00:0b"
}
}
}
}
]
Therefore this logstash configuration will not be working as /logstashpaths/tmp/ is not available in logstash executor container
14:08:33.646 [Thread-3] INFO o.a.m.l.executor.LogstashService - Starting subprocess: LS_HEAP_SIZE=256m HOME=/root /opt/logstash/bin/logstash --log /var/log/logstash.log -e input {
file {
"path" => [ "/logstashpaths/tmp/mesos/slaves/*/frameworks/*/executors/*/runs/latest/stdout", "/logstashpaths/tmp/mesos/slaves/*/frameworks/*/executors/*/runs/latest/stderr", "/logstashpaths/var/log/*.INFO", "/logstashpaths/var/log/*.WARNING" ]
}
}
filter {
mutate {
"add_field" => {
"mesos_slave_id" => "e0baf934-fffe-426b-b576-b450aec02c78-S2"
}
}
}
output {
elasticsearch {
"host" => "172.17.0.6"
"port" => 33000
"protocol" => "http"
}
}
I'd like to review the code to check if paths in /tmp/mesos/slaves/*/frameworks/*/executors/*/runs/latest/stdout, /tmp/mesos/slaves/*/frameworks/*/executors/*/runs/latest/stderr, /var/log/*.INFO, /var/log/*.WARNING can be analysed to extract the most specific non-wild-card path.
Going from
/tmp/mesos/slaves/*/frameworks/*/executors/*/runs/latest/stdout,
/tmp/mesos/slaves/*/frameworks/*/executors/*/runs/latest/stderr,
/var/log/*.INFO,
/var/log/*.WARNING
to
/tmp/mesos/slaves,
/var/log
and mapping those to the container with /logstashpaths prefix
From what I can see --enable.file=true --executor.file-path=/var/log/mesos/mesos-*.INFO is sufficient?
Damn docker-outside-docker of minimesos. In my tests logstash executors run on host, not on mesos agent 😡
will re-structure the tests.... to be continued
Yeah. Hindsight I always found that move a bit a pain in the A…
It's working.
With "EXECUTOR_FILE_PATH": "/Users/vik/projects/cs/devbox/sources/mesos-elk-demo/build/elk.log, /Users/vik/projects/cs/devbox/sources/mesos-elk-demo/build/*/*.log" newly created logs files were picked up by logstash
👍
Just requires testing on real mesos cluster with paths like /tmp/mesos/slaves/*/frameworks/*/executors/*/runs/latest/stdout, /tmp/mesos/slaves/*/frameworks/*/executors/*/runs/latest/stderr
currently blocked by #94 ...
You should be able to continue with MESOS_ROLE=* in your envs
Tested on Mesos Cluster and it works.
Description of the tests is in https://github.com/ContainerSolutions/mesos-elk-demo
👍