Flag Alerts Solution Overview

The following were the key design concerns/ decisions for designing the solution:

Batch Processing: As the input events are read from a log file instead of receiving events as a stream, the log file will be batch-processed for flagging alerts
Spring Batch: The solution uses Spring Batch for batch processing, which provides out of the box support for:
- Chunk oriented Batch Processing - Data is read, processed, and then written in configurable chunks which makes it robust and performant for large datasets, and avoids pitfalls like OOM issues from load huge volumes of data to memory and reading/ writing
- Fault Tolerance - Since the primary source of input is from a log file, flag-alerts-batch is made to be fault-tolerant for Input Validation/ Read parsing failures. Skip limit is configurable via property flag-alerts.parser.invalid-entry.skip-limit
- Performance - This solution can parse large log file in an efficient and performant manner. Partitioning logic can also be used for parallel processing - https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#partitioning
The events are flagged for alerts in column LOG_EVENT_ALERT.ALERT along with other event details(See schema-all). Since the events in log file can be unordered, the batch job is made up of two steps :
- Step 1 - Parsing the logfile for Log Events and persist to temporary tables ParseLogEntryStepConfiguration
- Step 2 - Join entries for log entries, and Flag Events and persist Event Alerts into LOG_EVENT_ALERT Table FlagAlertStepConfiguration
Functional/ Integration tests(Method coverage - 90%, Line Coverage -93% via Intellij IDEA code coverage runner) are available in FlagAlertsJobFunctionalTests

Building from Source

Clone the git repository/ download the source using the URL on the Github page: https://github.com/akshaddhoke/flag-alerts-batch

Command Line

Maven is the build tool used for flag-alerts-batch. You can build the project and via the command:

$ ./mvnw clean package

Running tests via maven:

$ ./mvnw clean test

Running the application locally using IDE

Since this is a spring boot application, any IDE can be used to run it locally with: Main class com.test.assignment.cs.flagalerts.FlagAlertsBatchApplication, and configuration parameters/ properties

Program Argument log-events.file="./src/test/resources/logfile-assignment-example.txt" - Pass argument for logfile job parameter(defaults to logfile.txt in working directory)
Spring Property spring.datasource.url=jdbc:hsqldb:file:flag-alerts - By default, hsql file db 'flag-alerts' is created/ used from working directory with the required schema. Use spring.datasource properties to customize the datasource

Running the application using JAR file

Download the built jar from latest build in Maven Workflow, Or once the application is built locally, use the jar from target/flag-alerts-batch-0.0.1-SNAPSHOT.jar to run the application. Example:

$ java -jar flag-alerts-batch-0.0.1-SNAPSHOT.jar log-events.file=logfile-generated.txt

Additional Configuration Properties

#event duration threshold in ms, beyond which the event is flagged for alert
flag-alerts.alerts.event-duration.threshold-ms=4
#Fault tolerance skip limit for invalid entries during log file parsing, before Job Failure
flag-alerts.parser.invalid-entry.skip-limit=10

Reference Documentation

For further reference, please consider the following sections:

Guides

The following guides illustrate how to use some features concretely:

Creating a Batch Service

akshaddhoke / flag-alerts-batch