Monitoring and Detection of Incidents in Big Data Applications
This project illustrates monitoring, detection and classification of incidents in Big Data applications on a simple example.
Introduction
We introduce a generic architecture of an incident management system that can be adapted to the specific needs of Cloud-based Big Data analytics applications.
Following components are included in the prototype:
- Generic classification database
- Monitoring agent
- Detection and classification rule
Citations
- Hong-Linh Truong, Manfred Halper, Characterizing Incidents in Cloud-based IoT Data Analytics, Proceedings of The 42nd IEEE International Conference on Computers, Software and Applications (COMPSAC 2018), Author version, Pre-print long version, Slides
Installation
Required Software
The simulation of the Big Data application uses the following Software to simulate the incident used to illustrate the functionality:
- Elasticserach
- Logstash
- Kibana
- Docker
- Docker compose
- https://github.com/rdsea/IoTCloudSamples
- Hadoop
- Apache Nifi
- Neo4j
- Python
- Pythonenv
Run the identification and classification rule
A short overview of the scenario.
- Install Docker and Docker Compose
- Load the project from https://github.com/rdsea/IoTCloudSamples https://github.com/rdsea/IoTCloudSamples. This is used to simulate the IoT sensors producing data.
- This scenario starts different Docker containers for further information refer to the product documentation pipenv run python provision.py config.sample.yml
- Install hadoop and start the hdfs
- Start Apache Nifi and configure
- Configure a MQTT processer consuming the MQTT stream from the Dockerised components
- Configure a HDFS processor writing into the local hdfs
- Configure a message queue between the two processors
- Install Elasticsearch
- Install Kibana
- Install Neo4j
- The database has to be initialised with the classification graphdatabase.cypher
- Install logstash on the server running the Apache Nifi installation
- Configure logstash with nifiPipeline.conf
- Install python on the server running the rules
- Execute the nifirule.py
The log data from the test scenario is fetched and fed via logstash into elasticsearch. The rule then identifies, classifies and writes the incident into elasticsearch where it can be visualised with Kibana.
If an incident is added to the prototyp three changes have to be comitted:
- The classification has to be expanded to cover all the specifics of a new incident
- A monitoring agent has to be implemented that delivers the necessary instrumentation data
- A detecion and classification rule needs to be developed.
Contributors
- Daniel Fuvesi
- Manfred Halper
- Hong-Linh Truong