VNF Intrusion detection system
danielbehnke opened this issue · comments
Detect all attacks on machine network, created once per machine park
@arocha7 @miguelmesquita Let's use this issue to share and align updates on the development and integration of the IDS into NS2.
Wiki entry for documentation of IDS deployment: https://github.com/sonata-nfv/tng-industrial-pilot/wiki/IDS-deployment
- Add image source and describe involved containers (D, X, P, ...?)
- Update text accordingly
Descriptors for native k8s deployment: https://github.com/sonata-nfv/tng-industrial-pilot/tree/master/k8s
- Remove v1?
- How to allow running privileged containers (with Security Policy)?
The Deployment "ns2-mdc" is invalid: spec.template.spec.containers[1].securityContext.privileged: Forbidden: disallowed by cluster policy
The logstash container is crashing in between:
eu-5gtango-lh-vnf2-0-4-6929a5c3-69b679697-mqd9m 2/2 Running 3 4m40s
After a few initial crashes, it now seems to run quite stable.
Logs give me this error message:
$ kubectl logs eu-5gtango-lh-vnf2-0-4-6929a5c3-69b679697-mqd9m cdu01-6929a5c3-1758-45ac-ae11-7a5401b83c9d
10.104.147.220 10.107.209.98
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2019-09-18T06:46:53,877][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2019-09-18T06:46:53,898][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2019-09-18T06:46:54,863][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.3.1"}
[2019-09-18T06:46:54,929][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"21a4aa16-f747-4f43-8c2d-abc97467855d", :path=>"/usr/share/logstash/data/uuid"}
[2019-09-18T06:46:57,393][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Illegal character in authority at index 7: http://10.104.147.220\n10.107.209.98:9200"}
[2019-09-18T06:46:57,452][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
[2019-09-18T06:47:03,529][INFO ][org.reflections.Reflections] Reflections took 803 ms to scan 1 urls, producing 19 keys and 39 values
[2019-09-18T06:47:10,395][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaNet::URISyntaxException", :message=>"Illegal character in authority at index 7: http://10.104.147.220\n10.107.209.98:9200", :backtrace=>["java.net.URI$Parser.fail(java/net/URI.java:2915)", "java.net.URI$Parser.parseAuthority(java/net/URI.java:3249)", "java.net.URI$Parser.parseHierarchical(java/net/URI.java:3160)", "java.net.URI$Parser.parse(java/net/URI.java:3116)", "java.net.URI.<init>(java/net/URI.java:600)", "jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)", "jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(jdk/internal/reflect/NativeConstructorAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(jdk/internal/reflect/DelegatingConstructorAccessorImpl.java:45)", "java.lang.reflect.Constructor.newInstance(java/lang/reflect/Constructor.java:490)", "org.jruby.javasupport.JavaConstructor.newInstanceDirect(org/jruby/javasupport/JavaConstructor.java:247)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.util.safe_uri.initialize(/usr/share/logstash/logstash-core/lib/logstash/util/safe_uri.rb:21)", "usr.share.logstash.logstash_minus_core.lib.logstash.util.safe_uri.RUBY$method$initialize$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash/util//usr/share/logstash/logstash-core/lib/logstash/util/safe_uri.rb)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.validate_value(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:514)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.RUBY$method$validate_value$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash/config//usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.process_parameter_value(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:329)", "org.jruby.RubyArray.collect(org/jruby/RubyArray.java:2563)", "org.jruby.RubyArray.map(org/jruby/RubyArray.java:2577)", "org.jruby.RubyArray$INVOKER$i$0$0$map19.call(org/jruby/RubyArray$INVOKER$i$0$0$map19.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.process_parameter_value(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:329)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.validate_check_parameter_values(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:352)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1792)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.validate_check_parameter_values(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:346)", "org.jruby.RubyHash$12.visit(org/jruby/RubyHash.java:1438)", "org.jruby.RubyHash$12.visit(org/jruby/RubyHash.java:1435)", "org.jruby.RubyHash.visitLimited(org/jruby/RubyHash.java:690)", "org.jruby.RubyHash.visitAll(org/jruby/RubyHash.java:675)", "org.jruby.RubyHash.iteratorVisitAll(org/jruby/RubyHash.java:1395)", "org.jruby.RubyHash.each_pairCommon(org/jruby/RubyHash.java:1430)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1419)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.validate_check_parameter_values(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:345)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.RUBY$method$validate_check_parameter_values$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash/config//usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.validate(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:235)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.RUBY$method$validate$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash/config//usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.config_init(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:86)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.RUBY$method$config_init$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash/config//usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.outputs.base.initialize(/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:60)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:894)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.RubyClass.finvoke(org/jruby/RubyClass.java:798)", "org.jruby.RubyBasicObject.callMethod(org/jruby/RubyBasicObject.java:363)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.initialize(org/logstash/config/ir/compiler/OutputStrategyExt.java:232)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt$INVOKER$i$1$0$initialize.call(org/logstash/config/ir/compiler/OutputStrategyExt$SimpleAbstractOutputStrategyExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.logstash.config.ir.compiler.OutputDelegatorExt.initialize(org/logstash/config/ir/compiler/OutputDelegatorExt.java:48)", "org.logstash.config.ir.compiler.OutputDelegatorExt.initialize(org/logstash/config/ir/compiler/OutputDelegatorExt.java:30)", "org.logstash.plugins.PluginFactoryExt$Plugins.plugin(org/logstash/plugins/PluginFactoryExt.java:242)", "org.logstash.plugins.PluginFactoryExt$Plugins.buildOutput(org/logstash/plugins/PluginFactoryExt.java:140)", "org.logstash.config.ir.CompiledPipeline.lambda$setupOutputs$0(org/logstash/config/ir/CompiledPipeline.java:115)", "java.util.ArrayList.forEach(java/util/ArrayList.java:1540)", "org.logstash.config.ir.CompiledPipeline.setupOutputs(org/logstash/config/ir/CompiledPipeline.java:112)", "org.logstash.config.ir.CompiledPipeline.<init>(org/logstash/config/ir/CompiledPipeline.java:82)", "org.logstash.execution.JavaBasePipelineExt.initialize(org/logstash/execution/JavaBasePipelineExt.java:50)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(org/logstash/execution/JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.initialize(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.execute(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash/pipeline_action//usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.converge_state(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:834)"]}
[2019-09-18T06:47:10,425][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaNet::URISyntaxException` for `PipelineAction::Create<main>`", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:338:in `block in converge_state'"]}
warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaNet::URISyntaxException` for `PipelineAction::Create<main>`
create at org/logstash/execution/ConvergeResultExt.java:109
add at org/logstash/execution/ConvergeResultExt.java:37
converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:338
[2019-09-18T06:47:10,518][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaNet::URISyntaxException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:338:in `block in converge_state'"]}
[2019-09-18T06:47:10,751][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exitCan't access Kibana:
Kibana pods and service are running fine after instantiation, but I can't access the dashboard. If I try to curl, I get connection refused:
# kubectl get svc
eu-5gtango-k-vnf4-0-4-bcb033cb LoadBalancer 10.102.132.91 10.200.16.61 5601:31552/TCP 6m7s
# trying to access
$ curl 10.200.16.61:5601
curl: (7) Failed to connect to 10.200.16.61 port 5601: Connection refusedIs an issue due to the connection of the CNFs. The wrong env vars, with capital letters, are used. These inculde the IP and port of all running instances, so the CNF can't distinguish which env vars belong to the current instance.
Instead, use the lower case env vars, which are injected by the MANO individually for each deployment. That should resolve the issue with the 2 instances.
Another issue is that the alarm is always triggered - no matter how the env vars for the filter are set.
This is done