sketch-ws
This project implements data streaming techniques to detect anomalies on streams of data. Given a series of events the system can detect heavy hitters and heavy changers: the former are those events whose frequency of occurrence is higher than a certain threshold value on a given time windows, and the latter are those events showing a difference on their frequency of occurrence on two adjacent time windows that is greater than another (possibly equal) threshold value.
Service configuration file
There are a few changes you may want to make for configuring the service parameters. Relevant configuration sections are listed below. Configuration file: sketch-ws/config.yml.
sketchConfig: this parameters define the shape of the sketches created bySketchManageron each detection epoch. Sub-parameterprimeis used to do some modulo arithmetic and, as it name implies, must be a prime number. Keep in mind thatprime>rows, otherwise the service will fail to start.detectionParameters: here you can change heavy hitters and heavy changers thresholds. ParametersketchRotationIntervaldefine the time window length (in seconds). Finally, the number of sketches stored bySketchHistoryQueueand heavy keys stored byHeavyKeysHistoryQueuecan be adjusted with sub-parametermaxHistoryQueueLength.
How to run the service
The system can by started by using the docker-compose file located on sketch-ws/ci/compose/:
The compose file will build the service Dockerfile image located at sketch-ws/ci/Dockerfile-receiver/. This build process uses two files that must be generated by using the script sketch-ws/ci/Dockerfile-receiver/copy_dockerfile_files.sh (a docker image will be automatically generated with a new push so don't forget to include the .yml and .jar files).
$ cd ci/Dockerfile-receiver/
$ ./copy_dockerfile_files.sh
$ cd ../compose
$ docker-compose up
You will see how the service spins up. Available resources will be shown on the console:
GET /health (com.lpineda.dsketch.resources.Health)
GET /heavykeys (com.lpineda.dsketch.core.HeavyKeysHistoryQueue)
GET /sketchhistory (com.lpineda.dsketch.core.SketchHistoryQueue)
GET /status (com.lpineda.dsketch.resources.Status)
More at: