starting logstash scheduler error

Question

starting logstash scheduler error

AndriiOmelianenko opened this issue 8 years ago · comments

Andrii Omelianenko commented 8 years ago

Trying to use 0.10.0 version (was using 0.8.0 successfully before).
This issue happens when mesos trying to start logstash scheduler

SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:53)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scheduler': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: com.containersolutions.mesos.scheduler.config.MesosConfigProperties com.containersolutions.mesos.scheduler.UniversalScheduler.mesosConfig; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mesosConfig': Could not bind properties to MesosConfigProperties (prefix=mesos, ignoreInvalidFields=false, ignoreUnknownFields=true, ignoreNestedProperties=false); nested exception is org.springframework.validation.BindException: org.springframework.boot.bind.RelaxedDataBinder$RelaxedBeanPropertyBindingResult: 1 errors
Field error in object 'mesos' on field 'resources': rejected value [ports(*):[1025-2180,2182-3887,3889-5049,6000-65535];ports(logstash):[514-514,25826-25826]]; codes [typeMismatch.mesos.resources,typeMismatch.resources,typeMismatch.com.containersolutions.mesos.scheduler.config.ResourcesConfigProperties,typeMismatch]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [mesos.resources,resources]; arguments []; default message [resources]]; default message [Failed to convert property value of type [java.lang.String] to required type [com.containersolutions.mesos.scheduler.config.ResourcesConfigProperties] for property 'resources'; nested exception is java.lang.IllegalStateException: Cannot convert value of type [java.lang.String] to required type [com.containersolutions.mesos.scheduler.config.ResourcesConfigProperties] for property 'resources': no matching editors or conversion strategy found]
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:334)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1214)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:772)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:118)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:766)
    at org.springframework.boot.SpringApplication.createAndRefreshContext(SpringApplication.java:361)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:307)
    at org.apache.mesos.logstash.scheduler.Application.main(Application.java:34)
    ... 6 more
Caused by: org.springframework.beans.factory.BeanCreationException: Could not autowire field: com.containersolutions.mesos.scheduler.config.MesosConfigProperties com.containersolutions.mesos.scheduler.UniversalScheduler.mesosConfig; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mesosConfig': Could not bind properties to MesosConfigProperties (prefix=mesos, ignoreInvalidFields=false, ignoreUnknownFields=true, ignoreNestedProperties=false); nested exception is org.springframework.validation.BindException: org.springframework.boot.bind.RelaxedDataBinder$RelaxedBeanPropertyBindingResult: 1 errors
Field error in object 'mesos' on field 'resources': rejected value [ports(*):[1025-2180,2182-3887,3889-5049,6000-65535];ports(logstash):[514-514,25826-25826]]; codes [typeMismatch.mesos.resources,typeMismatch.resources,typeMismatch.com.containersolutions.mesos.scheduler.config.ResourcesConfigProperties,typeMismatch]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [mesos.resources,resources]; arguments []; default message [resources]]; default message [Failed to convert property value of type [java.lang.String] to required type [com.containersolutions.mesos.scheduler.config.ResourcesConfigProperties] for property 'resources'; nested exception is java.lang.IllegalStateException: Cannot convert value of type [java.lang.String] to required type [com.containersolutions.mesos.scheduler.config.ResourcesConfigProperties] for property 'resources': no matching editors or conversion strategy found]
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:573)
    at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:331)
    ... 21 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mesosConfig': Could not bind properties to MesosConfigProperties (prefix=mesos, ignoreInvalidFields=false, ignoreUnknownFields=true, ignoreNestedProperties=false); nested exception is org.springframework.validation.BindException: org.springframework.boot.bind.RelaxedDataBinder$RelaxedBeanPropertyBindingResult: 1 errors
Field error in object 'mesos' on field 'resources': rejected value [ports(*):[1025-2180,2182-3887,3889-5049,6000-65535];ports(logstash):[514-514,25826-25826]]; codes [typeMismatch.mesos.resources,typeMismatch.resources,typeMismatch.com.containersolutions.mesos.scheduler.config.ResourcesConfigProperties,typeMismatch]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [mesos.resources,resources]; arguments []; default message [resources]]; default message [Failed to convert property value of type [java.lang.String] to required type [com.containersolutions.mesos.scheduler.config.ResourcesConfigProperties] for property 'resources'; nested exception is java.lang.IllegalStateException: Cannot convert value of type [java.lang.String] to required type [com.containersolutions.mesos.scheduler.config.ResourcesConfigProperties] for property 'resources': no matching editors or conversion strategy found]
    at org.springframework.boot.context.properties.ConfigurationPropertiesBindingPostProcessor.postProcessBeforeInitialization(ConfigurationPropertiesBindingPostProcessor.java:339)
    at org.springframework.boot.context.properties.ConfigurationPropertiesBindingPostProcessor.postProcessBeforeInitialization(ConfigurationPropertiesBindingPostProcessor.java:289)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:408)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1570)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1192)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1116)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1014)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:545)
    ... 23 more
Caused by: org.springframework.validation.BindException: org.springframework.boot.bind.RelaxedDataBinder$RelaxedBeanPropertyBindingResult: 1 errors
Field error in object 'mesos' on field 'resources': rejected value [ports(*):[1025-2180,2182-3887,3889-5049,6000-65535];ports(logstash):[514-514,25826-25826]]; codes [typeMismatch.mesos.resources,typeMismatch.resources,typeMismatch.com.containersolutions.mesos.scheduler.config.ResourcesConfigProperties,typeMismatch]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [mesos.resources,resources]; arguments []; default message [resources]]; default message [Failed to convert property value of type [java.lang.String] to required type [com.containersolutions.mesos.scheduler.config.ResourcesConfigProperties] for property 'resources'; nested exception is java.lang.IllegalStateException: Cannot convert value of type [java.lang.String] to required type [com.containersolutions.mesos.scheduler.config.ResourcesConfigProperties] for property 'resources': no matching editors or conversion strategy found]
    at org.springframework.boot.bind.PropertiesConfigurationFactory.validate(PropertiesConfigurationFactory.java:362)
    at org.springframework.boot.bind.PropertiesConfigurationFactory.doBindPropertiesToTarget(PropertiesConfigurationFactory.java:271)
    at org.springframework.boot.bind.PropertiesConfigurationFactory.bindPropertiesToTarget(PropertiesConfigurationFactory.java:241)
    at org.springframework.boot.context.properties.ConfigurationPropertiesBindingPostProcessor.postProcessBeforeInitialization(ConfigurationPropertiesBindingPostProcessor.java:334)
    ... 36 more

my mesos slave config has next configuration line:
MESOS_RESOURCES=ports:[1025-2180,2182-3887,3889-5049,6000-65535];ports(logstash):[514-514,25826-25826]

logstash marathon json:

{
   "id": "/logstash",
   "cpus": 1,
   "mem": 1024.0,
   "instances": 1,
   "uris": [
        "https://github.com/mesos/logstash/releases/download/0.10/logstash-mesos-scheduler-0.1.0.jar"
   ],
   "cmd": "java -jar logstash-mesos-scheduler-0.1.0.jar --server.port=$PORT0",
   "env": {
       "ENABLE_DOCKER": "false",
       "MESOS_ROLE": "logstash",
       "MESOS_ZOOKEEPER_SERVER": "zookeeper.service.consul:2181",
       "MESOS_MASTER=": "master.mesos.service.consul:5050",
       "LOGSTASH_ELASTICSEARCH_HOST": "elastic:9200",
       "LOGSTASH_HEAP_SIZE": "512",
       "EXECUTOR_HEAP_SIZE": "256",
       "ENABLE_SYSLOG": "true",
       "ENABLE_COLLECTD": "true",
       "ENABLE_FILE": "true",
       "EXECUTOR_FILE_PATH": "/var/log/mesos/*.INFO, /var/log/mesos/*.WARNING"
   }
}

Martin Westergaard Lassen · Answer 1 · Thu Apr 14 2016 23:36:22 GMT+0800 (China Standard Time)

Hi @AndriiOmelianenko. Thank you very much for your report.

Unfortunately this is a known issue, ContainerSolutions/mesos-starter#43. The easy fix is to make sure that MESOS_RESOURCES is unset before starting the application.

Andrii Omelianenko · Answer 2 · Fri Apr 15 2016 17:30:02 GMT+0800 (China Standard Time)

@mwl thanks for your response.
I've moved configuration from MESOS_RESOURCES=ports:[1025-2180,2182-3887,3889-5049,6000-65535];ports(logstash):[514-514,25826-25826]
to next option - /usr/sbin/mesos-slave --resources=ports(*):[1025-2180,2182-3887,3889-5049,6000-65535];ports(logstash):[514-514,25826-25826],

submitted logstash json to marathon, it started running (without errors), but nothing happens after that:

stdout:

Registered executor on slave-04.example.com
Starting task logstash.32f2c5b9-02eb-11e6-b828-0242000bf102
sh -c 'java -jar logstash-mesos-scheduler-0.1.0.jar --server.port=$PORT0'
Forked command at 25395
09:19:51.541 [background-preinit] INFO  o.h.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final
  |\   /|
  | \ / |
  | / \ |
  |/   \|
 / \   /|
/   \ / | .____                          __                .__
\    |  | |    |    ____   ____  _______/  |______    _____|  |__
 \   |  | |    |   /  _ \ / ___\/  ___/\   __\__  \  /  ___/  |  \
  |  | /  |    |__(  <_> ) /_/  >___ \  |  |  / __ \_\___ \|   Y  \
  |  |/   |_______ \____/\___  /____  > |__| (____  /____  >___|  /
  | /             \/    /_____/     \/            \/     \/     \/
  |/     :: Running Spring Boot 0.1.0 ::
2016-04-15 09:19:51.645  INFO 25395 --- [           main] o.a.m.logstash.scheduler.Application     : Starting Application v0.1.0 on slave-04.example.com with PID 25395 (/mnt/mesos/slaves/25625492-7806-42fd-af67-90f6b0fef272-S4/frameworks/25625492-7806-42fd-af67-90f6b0fef272-0001/executors/logstash.32f2c5b9-02eb-11e6-b828-0242000bf102/runs/cff5114b-0103-466c-ba97-227ca5b39c83/logstash-mesos-scheduler-0.1.0.jar started by root in /mnt/mesos/slaves/25625492-7806-42fd-af67-90f6b0fef272-S4/frameworks/25625492-7806-42fd-af67-90f6b0fef272-0001/executors/logstash.32f2c5b9-02eb-11e6-b828-0242000bf102/runs/cff5114b-0103-466c-ba97-227ca5b39c83)
2016-04-15 09:19:51.645  INFO 25395 --- [           main] o.a.m.logstash.scheduler.Application     : No active profile set, falling back to default profiles: default
2016-04-15 09:19:53.885  INFO 25395 --- [           main] o.a.m.logstash.scheduler.Application     : Started Application in 2.85 seconds (JVM running for 3.23)

stderr:

I0415 09:19:47.727182 25341 logging.cpp:172] INFO level logging started!
I0415 09:19:47.727496 25341 fetcher.cpp:414] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/25625492-7806-42fd-af67-90f6b0fef272-S4\/root","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"https:\/\/github.com\/mesos\/logstash\/releases\/download\/0.10\/logstash-mesos-scheduler-0.1.0.jar"}}],"sandbox_directory":"\/mnt\/mesos\/slaves\/25625492-7806-42fd-af67-90f6b0fef272-S4\/frameworks\/25625492-7806-42fd-af67-90f6b0fef272-0001\/executors\/logstash.32f2c5b9-02eb-11e6-b828-0242000bf102\/runs\/cff5114b-0103-466c-ba97-227ca5b39c83","user":"root"}
I0415 09:19:47.728615 25341 fetcher.cpp:369] Fetching URI 'https://github.com/mesos/logstash/releases/download/0.10/logstash-mesos-scheduler-0.1.0.jar'
I0415 09:19:47.728631 25341 fetcher.cpp:243] Fetching directly into the sandbox directory
I0415 09:19:47.728643 25341 fetcher.cpp:180] Fetching URI 'https://github.com/mesos/logstash/releases/download/0.10/logstash-mesos-scheduler-0.1.0.jar'
I0415 09:19:47.728667 25341 fetcher.cpp:127] Downloading resource from 'https://github.com/mesos/logstash/releases/download/0.10/logstash-mesos-scheduler-0.1.0.jar' to '/mnt/mesos/slaves/25625492-7806-42fd-af67-90f6b0fef272-S4/frameworks/25625492-7806-42fd-af67-90f6b0fef272-0001/executors/logstash.32f2c5b9-02eb-11e6-b828-0242000bf102/runs/cff5114b-0103-466c-ba97-227ca5b39c83/logstash-mesos-scheduler-0.1.0.jar'
I0415 09:19:50.521008 25341 fetcher.cpp:446] Fetched 'https://github.com/mesos/logstash/releases/download/0.10/logstash-mesos-scheduler-0.1.0.jar' to '/mnt/mesos/slaves/25625492-7806-42fd-af67-90f6b0fef272-S4/frameworks/25625492-7806-42fd-af67-90f6b0fef272-0001/executors/logstash.32f2c5b9-02eb-11e6-b828-0242000bf102/runs/cff5114b-0103-466c-ba97-227ca5b39c83/logstash-mesos-scheduler-0.1.0.jar'
I0415 09:19:50.644446 25338 logging.cpp:172] INFO level logging started!
I0415 09:19:50.645401 25338 exec.cpp:134] Version: 0.25.0
I0415 09:19:50.648433 25391 exec.cpp:208] Executor registered on slave 25625492-7806-42fd-af67-90f6b0fef272-S4
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/mesos/slaves/25625492-7806-42fd-af67-90f6b0fef272-S4/frameworks/25625492-7806-42fd-af67-90f6b0fef272-0001/executors/logstash.32f2c5b9-02eb-11e6-b828-0242000bf102/runs/cff5114b-0103-466c-ba97-227ca5b39c83/logstash-mesos-scheduler-0.1.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mnt/mesos/slaves/25625492-7806-42fd-af67-90f6b0fef272-S4/frameworks/25625492-7806-42fd-af67-90f6b0fef272-0001/executors/logstash.32f2c5b9-02eb-11e6-b828-0242000bf102/runs/cff5114b-0103-466c-ba97-227ca5b39c83/logstash-mesos-scheduler-0.1.0.jar!/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2016-04-15 09:19:53,722:25395(0x7f9daec9b700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2016-04-15 09:19:53,722:25395(0x7f9daec9b700):ZOO_INFO@log_env@716: Client environment:host.name=slave-04.example.com
2016-04-15 09:19:53,722:25395(0x7f9daec9b700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2016-04-15 09:19:53,722:25395(0x7f9daec9b700):ZOO_INFO@log_env@724: Client environment:os.arch=3.10.0-229.20.1.el7.x86_64
2016-04-15 09:19:53,722:25395(0x7f9daec9b700):ZOO_INFO@log_env@725: Client environment:os.version=#1 SMP Tue Nov 3 19:10:07 UTC 2015
2016-04-15 09:19:53,723:25395(0x7f9daec9b700):ZOO_INFO@log_env@733: Client environment:user.name=(null)
2016-04-15 09:19:53,723:25395(0x7f9daec9b700):ZOO_INFO@log_env@741: Client environment:user.home=/root
2016-04-15 09:19:53,723:25395(0x7f9daec9b700):ZOO_INFO@log_env@753: Client environment:user.dir=/mnt/mesos/slaves/25625492-7806-42fd-af67-90f6b0fef272-S4/frameworks/25625492-7806-42fd-af67-90f6b0fef272-0001/executors/logstash.32f2c5b9-02eb-11e6-b828-0242000bf102/runs/cff5114b-0103-466c-ba97-227ca5b39c83
2016-04-15 09:19:53,723:25395(0x7f9daec9b700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=zookeeper.service.consul:2181 sessionTimeout=1000 watcher=0x7f9e1965e700 sessionId=0 sessionPasswd=<null> context=0x7f9d980031d0 flags=0
2016-04-15 09:19:53,727:25395(0x7f9da5ffb700):ZOO_INFO@check_events@1703: initiated connection to server [192.168.0.63:2181]
2016-04-15 09:19:53,731:25395(0x7f9da5ffb700):ZOO_INFO@check_events@1750: session establishment complete on server [192.168.0.63:2181], sessionId=0x154192336800099, negotiated timeout=4000
I0415 09:19:53.872167 25410 logging.cpp:172] INFO level logging started!

is it expected to start executors etc?

Martin Westergaard Lassen · Answer 3 · Fri Apr 15 2016 22:19:16 GMT+0800 (China Standard Time)

The scheduler isn't connecting to Mesos. Probably because you have an equals (=) sign in your MESOS_MASTER= environment variable. Remove it and try again.