Giters

mesos / logstash

Logstash on Mesos

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

logstash-executor not starting

cookandy opened this issue · comments

commented

Hello,

Thanks for the project.

I'm deploying via marathon with the following config:

{
  "id": "logging/logstash",
  "cpus": 1,
  "mem": 1024.0,
  "instances": 1,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "mesos/logstash-scheduler:0.10-RC1",
      "network": "HOST"
    },
    "volumes" : [
         {
             "containerPath": "/etc/hostname.orig",
             "hostPath": "/etc/hostname",
             "mode": "RO"
         },
         {
             "containerPath": "/etc/resolv.conf",
             "hostPath": "/etc/resolv.conf",
             "mode": "RO"
         }
    ]
  },
  "env": {
    "MESOS_ZOOKEEPER_SERVER": "paas-master00.local:2181",
    "MESOS_MASTER": "paas-master00.local:5050",
    "FRAMEWORK_NAME": "logstash",
    "MESOS_ROLE": "logstash",
    "MESOS_USER": "root",
    "LOGSTASH_HEAP_SIZE": "64",
    "LOGSTASH_ELASTICSEARCH_URL": "http://es.service.consul:9200",
    "LOGSTASH_EXECUTOR_IMAGE": "mesos/logstash-executor",
    "LOGSTASH_EXECUTOR_VERSION": "0.10-RC1",
    "EXECUTOR_CPUS": "0.5",
    "EXECUTOR_HEAP_SIZE": "128",
    "ENABLE_COLLECTD": "false",
    "ENABLE_SYSLOG": "false",
    "ENABLE_FILE": "true",
    "ENABLE_DOCKER": "true",
    "EXECUTOR_FILE_PATH": "/tmp/supervisord/*.log"
  }
}

The container starts and the framework gets registered, but the logstash-executor never starts.

I see this in the stderr and stdout

I0203 17:03:16.590608 60267 exec.cpp:161] Version: 1.0.1
I0203 17:03:16.597717 60270 exec.cpp:236] Executor registered on agent 66cda024-c5be-4e60-8f81-8d378b4f231d-S0
I0203 17:03:16.599831 60270 docker.cpp:809] Running docker -H unix:///tmp/docker.sock run --cpu-shares 1024 --memory 1073741824 -e MESOS_ROLE=logstash -e MARATHON_APP_VERSION=2017-02-03T17:03:16.243Z -e LOGSTASH_ELASTICSEARCH_URL=http://es.service.consul:9200 -e HOST=paas-slave00.local -e MARATHON_APP_RESOURCE_CPUS=1.0 -e ENABLE_SYSLOG=false -e EXECUTOR_CPUS=0.5 -e MARATHON_APP_RESOURCE_GPUS=0 -e PORT_10009=31745 -e MARATHON_APP_DOCKER_IMAGE=mesos/logstash-scheduler:0.10-RC1 -e EXECUTOR_FILE_PATH=/tmp/supervisord/*.log -e MESOS_TASK_ID=logging_logstash.a7a4ab9a-ea32-11e6-9f0f-02420ea9906f -e PORT=31745 -e ENABLE_COLLECTD=false -e LOGSTASH_EXECUTOR_VERSION=0.10-RC1 -e MARATHON_APP_RESOURCE_MEM=1024.0 -e EXECUTOR_HEAP_SIZE=128 -e ENABLE_DOCKER=true -e PORTS=31745 -e MESOS_ZOOKEEPER_SERVER=paas-master00.local:2181 -e ENABLE_FILE=true -e FRAMEWORK_NAME=logstash -e MARATHON_APP_RESOURCE_DISK=0.0 -e MARATHON_APP_LABELS= -e MESOS_MASTER=paas-master00.local:5050 -e MARATHON_APP_ID=/logging/logstash -e PORT0=31745 -e LOGSTASH_HEAP_SIZE=64 -e LOGSTASH_EXECUTOR_IMAGE=mesos/logstash-executor -e MESOS_USER=root -e MESOS_SANDBOX=/mnt/mesos/sandbox -e MESOS_CONTAINER_NAME=mesos-66cda024-c5be-4e60-8f81-8d378b4f231d-S0.afef1091-3411-48f3-b990-06640a0c537a -v /etc/hostname:/etc/hostname.orig:ro -v /etc/resolv.conf:/etc/resolv.conf:ro -v /tmp/mesos/slaves/66cda024-c5be-4e60-8f81-8d378b4f231d-S0/frameworks/e415adbe-4e03-4e41-8302-80094b4d611d-0001/executors/logging_logstash.a7a4ab9a-ea32-11e6-9f0f-02420ea9906f/runs/afef1091-3411-48f3-b990-06640a0c537a:/mnt/mesos/sandbox --net host --name mesos-66cda024-c5be-4e60-8f81-8d378b4f231d-S0.afef1091-3411-48f3-b990-06640a0c537a mesos/logstash-scheduler:0.10-RC1
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/logstash-mesos-scheduler.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/logstash-mesos-scheduler.jar!/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2017-02-03 17:03:24,098:1(0x7fc8a2b8b700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2017-02-03 17:03:24,099:1(0x7fc8a2b8b700):ZOO_INFO@log_env@716: Client environment:host.name=paas-slave00
2017-02-03 17:03:24,099:1(0x7fc8a2b8b700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2017-02-03 17:03:24,099:1(0x7fc8a2b8b700):ZOO_INFO@log_env@724: Client environment:os.arch=4.4.0-59-generic
2017-02-03 17:03:24,099:1(0x7fc8a2b8b700):ZOO_INFO@log_env@725: Client environment:os.version=#80-Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017
2017-02-03 17:03:24,099:1(0x7fc8a2b8b700):ZOO_INFO@log_env@733: Client environment:user.name=(null)
2017-02-03 17:03:24,099:1(0x7fc8a2b8b700):ZOO_INFO@log_env@741: Client environment:user.home=/root
2017-02-03 17:03:24,099:1(0x7fc8a2b8b700):ZOO_INFO@log_env@753: Client environment:user.dir=/
2017-02-03 17:03:24,099:1(0x7fc8a2b8b700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=paas-master00.local:2181 sessionTimeout=1000 watcher=0x7fc8c9f7a600 sessionId=0 sessionPasswd=<null> context=0x7fc890000ae0 flags=0
2017-02-03 17:03:24,103:1(0x7fc89f17e700):ZOO_INFO@check_events@1703: initiated connection to server [10.136.17.30:2181]
2017-02-03 17:03:24,111:1(0x7fc89f17e700):ZOO_INFO@check_events@1750: session establishment complete on server [10.136.17.30:2181], sessionId=0x15a04b36574000f, negotiated timeout=4000
I0203 17:03:24.489964    21 sched.cpp:164] Version: 0.25.0
I0203 17:03:24.493461    38 sched.cpp:262] New master detected at master@10.136.17.30:5050
I0203 17:03:24.493846    38 sched.cpp:272] No credentials provided. Attempting to register without authentication
I0203 17:03:24.496641    34 sched.cpp:641] Framework registered with df5c652b-4e6e-4cc3-8db0-41696dd38eb7-0008

--container="mesos-66cda024-c5be-4e60-8f81-8d378b4f231d-S0.afef1091-3411-48f3-b990-06640a0c537a" --docker="docker" --docker_socket="/tmp/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/tmp/mesos/slaves/66cda024-c5be-4e60-8f81-8d378b4f231d-S0/frameworks/e415adbe-4e03-4e41-8302-80094b4d611d-0001/executors/logging_logstash.a7a4ab9a-ea32-11e6-9f0f-02420ea9906f/runs/afef1091-3411-48f3-b990-06640a0c537a" --stop_timeout="50secs"
--container="mesos-66cda024-c5be-4e60-8f81-8d378b4f231d-S0.afef1091-3411-48f3-b990-06640a0c537a" --docker="docker" --docker_socket="/tmp/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/tmp/mesos/slaves/66cda024-c5be-4e60-8f81-8d378b4f231d-S0/frameworks/e415adbe-4e03-4e41-8302-80094b4d611d-0001/executors/logging_logstash.a7a4ab9a-ea32-11e6-9f0f-02420ea9906f/runs/afef1091-3411-48f3-b990-06640a0c537a" --stop_timeout="50secs"
Registered docker executor on paas-slave00.local
Starting task logging_logstash.a7a4ab9a-ea32-11e6-9f0f-02420ea9906f
  |\   /|
  | \ / |
  | / \ |
  |/   \|
 / \   /|
/   \ / | .____                          __                .__
\    |  | |    |    ____   ____  _______/  |______    _____|  |__
 \   |  | |    |   /  _ \ / ___\/  ___/\   __\__  \  /  ___/  |  \
  |  | /  |    |__(  <_> ) /_/  >___ \  |  |  / __ \_\___ \|   Y  \
  |  |/   |_______ \____/\___  /____  > |__| (____  /____  >___|  /
  | /             \/    /_____/     \/            \/     \/     \/
  |/     :: Running Spring Boot 0.1.0 ::
2017-02-03 17:03:19.107  INFO 1 --- [           main] o.a.m.logstash.scheduler.Application     : Starting Application v0.1.0 on paas-slave00 with PID 1 (/tmp/logstash-mesos-scheduler.jar started by root in /)
2017-02-03 17:03:19.118  INFO 1 --- [           main] o.a.m.logstash.scheduler.Application     : No active profile set, falling back to default profiles: default
2017-02-03 17:03:24.497  INFO 1 --- [           main] o.a.m.logstash.scheduler.Application     : Started Application in 6.564 seconds (JVM running for 7.563)

I have even manually pulled the logstash-executor images on the slaves, but they fail to start. Any ideas? Thanks in advance.

commented
commented

Hi @mwl

Thanks for the reply - you were right. In my case, removing MESOS_ROLE wasn't enough, I had to actually set "MESOS_ROLE": "*".

However, it seems I am having another problem with the framework. The scheduler now starts, and tries to kick off the executor, but the executor is unable to access mesos shared library:

Failed to load native Mesos library from /usr/lib/libmesos-1.0.1.so
Exception in thread "main" java.lang.UnsatisfiedLinkError: Can't load library: /usr/lib/libmesos-1.0.1.so
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
	at java.lang.Runtime.load0(Runtime.java:809)
	at java.lang.System.load(System.java:1086)
	at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:159)
	at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:188)
	at org.apache.mesos.MesosExecutorDriver.<clinit>(MesosExecutorDriver.java:52)
	at org.apache.mesos.logstash.executor.Application.run(Application.java:26)
	at org.apache.mesos.logstash.executor.Application.main(Application.java:16)

I am running mesos inside a container, so I'm wondering how I can expose this shared object to the executor.

One idea I had was to map some /tmp volume in my mesos container and then copy the files over, and then start the scheduler with "MESOS_NATIVE_JAVA_LIBRARY": "/tmp/lib/libmesos.so", however that just led me down a rabbit hole of shared object links:

Caused by: java.lang.UnsatisfiedLinkError: /tmp/lib/libmesos-1.0.1.so: libevent_openssl-2.0.so.5: cannot open shared object file: No such file or directory

Can you think of any easy way to use this framework while running mesos/marathon in a container?

commented

Hi. Did you build the scheduler image yourself?

Never the less I'm afraid logstash might have grown incompatible with the latest version of Mesos. I suggest you use the framework from Universe, https://github.com/mesosphere/universe/blob/version-3.x/repo/packages/L/logstash/0/marathon.json.mustache