following the dcos instructions, I just get TASK_FAILED when launching the framework
brentley opened this issue · comments
Following the DCOS instructions in the readme, I launched the framework. I used the basic logstash-options.json provided by the example:
{
"logstash": {
"executor" : {
"volumes" : "/var/log/mesos"
}
}
}
When I run the framework, it deploys, but fails. Here is a full copy of stderr:
mesos-docker-executor: /lib64/libcurl.so.4: no version information available (required by /opt/mesosphere/packages/mesos--8467f0ef9a5aa54a502e0e0ab0f6515d8aecf00f/lib/libmesos-0.24.1.so)
I1214 20:31:52.896688 11492 exec.cpp:133] Version: 0.24.1
I1214 20:31:52.899741 11496 exec.cpp:207] Executor registered on slave 20151214-193035-2030436362-5050-2211-S3
2015-12-14 20:31:58,816:6(0x7f06e2f9a700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2015-12-14 20:31:58,816:6(0x7f06e2f9a700):ZOO_INFO@log_env@716: Client environment:host.name=ip-10-0-1-163.us-west-2.compute.internal
2015-12-14 20:31:58,816:6(0x7f06e2f9a700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2015-12-14 20:31:58,816:6(0x7f06e2f9a700):ZOO_INFO@log_env@724: Client environment:os.arch=4.1.7-coreos
2015-12-14 20:31:58,816:6(0x7f06e2f9a700):ZOO_INFO@log_env@725: Client environment:os.version=#2 SMP Wed Sep 16 22:54:37 UTC 2015
2015-12-14 20:31:58,817:6(0x7f06e2f9a700):ZOO_INFO@log_env@733: Client environment:user.name=(null)
2015-12-14 20:31:58,817:6(0x7f06e2f9a700):ZOO_INFO@log_env@741: Client environment:user.home=/root
2015-12-14 20:31:58,817:6(0x7f06e2f9a700):ZOO_INFO@log_env@753: Client environment:user.dir=/
2015-12-14 20:31:58,817:6(0x7f06e2f9a700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=master.mesos:2181 sessionTimeout=20000 watcher=0x7f07394cfa60 sessionId=0 sessionPasswd=<null> context=0x7f06c0000930 flags=0
2015-12-14 20:31:58,819:6(0x7f06de791700):ZOO_INFO@check_events@1703: initiated connection to server [10.0.6.122:2181]
2015-12-14 20:31:58,822:6(0x7f06de791700):ZOO_INFO@check_events@1750: session establishment complete on server [10.0.6.122:2181], sessionId=0x351a1f8ae2403cb, negotiated timeout=20000
2015-12-14 20:31:59,066:6(0x7f06e2799700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2015-12-14 20:31:59,066:6(0x7f06e2799700):ZOO_INFO@log_env@716: Client environment:host.name=ip-10-0-1-163.us-west-2.compute.internal
2015-12-14 20:31:59,066:6(0x7f06e2799700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2015-12-14 20:31:59,066:6(0x7f06e2799700):ZOO_INFO@log_env@724: Client environment:os.arch=4.1.7-coreos
2015-12-14 20:31:59,066:6(0x7f06e2799700):ZOO_INFO@log_env@725: Client environment:os.version=#2 SMP Wed Sep 16 22:54:37 UTC 2015
I1214 20:31:59.066318 21 sched.cpp:157] Version: 0.22.1
2015-12-14 20:31:59,066:6(0x7f06e2799700):ZOO_INFO@log_env@733: Client environment:user.name=(null)
2015-12-14 20:31:59,066:6(0x7f06e2799700):ZOO_INFO@log_env@741: Client environment:user.home=/root
2015-12-14 20:31:59,066:6(0x7f06e2799700):ZOO_INFO@log_env@753: Client environment:user.dir=/
2015-12-14 20:31:59,066:6(0x7f06e2799700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=master.mesos:2181 sessionTimeout=10000 watcher=0x7f07394cfa60 sessionId=0 sessionPasswd=<null> context=0x7f06cc001720 flags=0
2015-12-14 20:31:59,070:6(0x7f06dd78f700):ZOO_INFO@check_events@1703: initiated connection to server [10.0.6.123:2181]
2015-12-14 20:31:59,073:6(0x7f06dd78f700):ZOO_INFO@check_events@1750: session establishment complete on server [10.0.6.123:2181], sessionId=0x151a1f8c66a03e9, negotiated timeout=10000
I1214 20:31:59.076843 39 group.cpp:313] Group process (group(1)@10.0.1.163:38057) connected to ZooKeeper
I1214 20:31:59.076889 39 group.cpp:790] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I1214 20:31:59.076936 39 group.cpp:385] Trying to create path '/mesos' in ZooKeeper
I1214 20:31:59.103050 39 detector.cpp:138] Detected a new leader: (id='2')
I1214 20:31:59.103186 39 group.cpp:659] Trying to get '/mesos/json.info_0000000002' in ZooKeeper
Failed to detect a master: Failed to parse data of unknown label 'json.info'
and for good measure, stdout:
--container="mesos-20151214-193035-2030436362-5050-2211-S7.34616e7e-c7b5-4ed4-a2cb-b9edc43a4579" --docker="docker" --help="false" --initialize_driver_logging="true" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/var/lib/mesos/slave/slaves/20151214-193035-2030436362-5050-2211-S7/frameworks/20151214-193035-2030436362-5050-2211-0000/executors/logstash.b95cb352-a2a2-11e5-9dd5-0242474a1239/runs/34616e7e-c7b5-4ed4-a2cb-b9edc43a4579" --stop_timeout="0ns"
--container="mesos-20151214-193035-2030436362-5050-2211-S7.34616e7e-c7b5-4ed4-a2cb-b9edc43a4579" --docker="docker" --help="false" --initialize_driver_logging="true" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/var/lib/mesos/slave/slaves/20151214-193035-2030436362-5050-2211-S7/frameworks/20151214-193035-2030436362-5050-2211-0000/executors/logstash.b95cb352-a2a2-11e5-9dd5-0242474a1239/runs/34616e7e-c7b5-4ed4-a2cb-b9edc43a4579" --stop_timeout="0ns"
Registered docker executor on ip-10-0-0-6.us-west-2.compute.internal
Starting task logstash.b95cb352-a2a2-11e5-9dd5-0242474a1239
20:39:09.837 [main] INFO o.a.m.logstash.scheduler.Application - Starting Application on ip-10-0-0-6.us-west-2.compute.internal with PID 6 (/tmp/logstash-scheduler.jar started by root in /)
20:39:09.840 [main] DEBUG o.a.m.logstash.scheduler.Application - Running with Spring Boot v1.2.5.RELEASE, Spring v4.1.7.RELEASE
20:39:09.884 [main] INFO o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext - Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@2de7eb69: startup date [Mon Dec 14 20:39:09 UTC 2015]; root of context hierarchy
20:39:11.224 [main] INFO o.s.b.f.s.DefaultListableBeanFactory - Overriding bean definition for bean 'beanNameViewResolver': replacing [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=org.springframework.boot.autoconfigure.web.ErrorMvcAutoConfiguration$WhitelabelErrorViewConfiguration; factoryMethodName=beanNameViewResolver; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/boot/autoconfigure/web/ErrorMvcAutoConfiguration$WhitelabelErrorViewConfiguration.class]] with [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=org.springframework.boot.autoconfigure.web.WebMvcAutoConfiguration$WebMvcAutoConfigurationAdapter; factoryMethodName=beanNameViewResolver; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/boot/autoconfigure/web/WebMvcAutoConfiguration$WebMvcAutoConfigurationAdapter.class]]
20:39:11.633 [main] INFO o.h.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final
20:39:12.381 [main] INFO o.s.b.c.e.j.JettyEmbeddedServletContainerFactory - Server initialized with port: 9092
20:39:12.386 [main] INFO org.eclipse.jetty.server.Server - jetty-9.2.11.v20150529
20:39:12.620 [main] INFO / - Initializing Spring embedded WebApplicationContext
20:39:12.621 [main] INFO o.s.web.context.ContextLoader - Root WebApplicationContext: initialization completed in 2740 ms
20:39:13.866 [main] INFO o.s.b.c.e.ServletRegistrationBean - Mapping servlet: 'dispatcherServlet' to [/]
20:39:13.870 [main] INFO o.s.b.c.e.FilterRegistrationBean - Mapping filter: 'characterEncodingFilter' to: [/*]
20:39:13.870 [main] INFO o.s.b.c.e.FilterRegistrationBean - Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
20:39:14.298 [main] INFO o.e.j.server.handler.ContextHandler - Started o.s.b.c.e.j.JettyEmbeddedWebAppContext@3648664e{/,null,AVAILABLE}
20:39:14.298 [main] INFO org.eclipse.jetty.server.Server - Started @5782ms
20:39:14.446 [main] INFO o.a.m.logstash.config.ConfigManager - Fetched latest config: null
20:39:14.488 [main] DEBUG o.a.m.l.scheduler.LogstashScheduler - Setting webuiUrl to http:\/\/ip-10-0-0-6.us-west-2.compute.internal:9092
20:39:14.491 [main] DEBUG o.a.m.l.scheduler.LogstashScheduler - Setting webuiUrl to http:\/\/ip-10-0-0-6.us-west-2.compute.internal:9092
20:39:14.493 [main] INFO o.a.m.l.scheduler.LogstashScheduler - Starting Logstash Framework:
user: "root"
name: "logstash"
failover_timeout: 3.14496E7
checkpoint: true
role: "*"
webui_url: "http:\\/\\/ip-10-0-0-6.us-west-2.compute.internal:9092"
What should I do differently to get this going?
From duplicate #62:
The same/similar scheduler stderr:
I1215 16:28:08.349599 37 group.cpp:313] Group process (group(1)@10.114.49.179:56925) connected to ZooKeeper
I1215 16:28:08.349645 37 group.cpp:790] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I1215 16:28:08.349659 37 group.cpp:385] Trying to create path '/mesos' in ZooKeeper
I1215 16:28:08.395557 37 detector.cpp:138] Detected a new leader: (id='25')
I1215 16:28:08.395725 37 group.cpp:659] Trying to get '/mesos/json.info_0000000025' in ZooKeeper
Failed to detect a master: Failed to parse data of unknown label 'json.info'
and @mcuervoe says:
It stops working in mesos 0.24 I think it is related to this:
https://issues.apache.org/jira/browse/MESOS-2340
https://issues.apache.org/jira/browse/MESOS-3135
the elasticsearch project also suffered the same problem:
I'm told the equivalent problem in ES was fixed by this pull: mesos/elasticsearch#386
Logstash has been upgraded to 0.25.0 a while back so this should be fixed. If there is still an issue please reopen.
Below is the json file which I am using
{
"id": "mist-logstash",
"cpus": 1,
"mem": 1024.0,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "mesos/logstash-scheduler:0.0.6",
"network": "HOST"
}
},
"env": {
"JAVA_OPTS": "-Dmesos.logstash.framework.name=logstash -Dmesos.zk=zk://zkhost:2181/mesos"
}
}
still getting below error
2016-01-16 01:12:19,956:5(0x7f80a1ffb700):ZOO_INFO@check_events@1703: initiated connection to server [172.31.4.72:2181]
2016-01-16 01:12:19,959:5(0x7f80a1ffb700):ZOO_INFO@check_events@1750: session establishment complete on server [172.31.4.72:2181], sessionId=0x1516706148feac2, negotiated timeout=10000
I0116 01:12:19.959508 34 group.cpp:313] Group process (group(1)@172.31.4.28:59624) connected to ZooKeeper
I0116 01:12:19.959650 34 group.cpp:790] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0116 01:12:19.959738 34 group.cpp:385] Trying to create path '/mesos' in ZooKeeper
I0116 01:12:19.980105 34 detector.cpp:138] Detected a new leader: (id='85')
I0116 01:12:19.980309 34 group.cpp:659] Trying to get '/mesos/json.info_0000000085' in ZooKeeper
Failed to detect a master: Failed to parse data of unknown label 'json.info'