Anomaly Detection and Explanation

We develop deep learning model that detects and explain anomaly in multivariate time series data.

Our model is based on Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy (ICLR'22). We train and evaluate the model on DBSherlock dataset.

Anomaly Transformer

Anomaly transformer is a transformer-based model that detects anomaly in multivariate time series data. It is based on the assumption that the normal data is highly correlated, while the abnormal data is not. It uses a transformer encoder to learn the correlation between different time steps, and then uses a discriminator to distinguish the normal and abnormal data based on the learned correlation.

An inherent distinguishable criterion as Association Discrepancy for detection.
A new Anomaly-Attention mechanism to compute the association discrepancy.
A minimax strategy to amplify the normal-abnormal distinguishability of the association discrepancy.

For more details, please refer to the paper.

Environment Setup

Start docker container using docker compose, and login to the container

docker compose up -d

Install python packages

pip install -r requirements.txt

Prepare Dataset

Download

Download DBSherlock dataset.

python scripts/dataset/download_datasets.py

Append --download_all argument to download all datasets (i.e., SMD, SMAP, PSM, MSL, and DBSherlock).

python scripts/dataset/download_datasets.py --download_all

Preprocess data

Convert DBSherlock data (.mat file to .json file):

python src/data_factory/dbsherlock/convert.py \
    --input dataset/dbsherlock/tpcc_16w.mat \
    --out_dir dataset/dbsherlock/converted/ \
    --prefix tpcc_16w

python src/data_factory/dbsherlock/convert.py \
    --input dataset/dbsherlock/tpcc_500w.mat \
    --out_dir dataset/dbsherlock/converted/ \
    --prefix tpcc_500w

python src/data_factory/dbsherlock/convert.py \
    --input dataset/dbsherlock/tpce_3000.mat \
    --out_dir dataset/dbsherlock/converted/ \
    --prefix tpce_3000

Convert DBSherlock data into train & validate data for Anomaly Transformer:

python src/data_factory/dbsherlock/process.py \
    --input_path dataset/dbsherlock/converted/tpcc_16w_test.json \
    --output_path dataset/dbsherlock/processed/tpcc_16w/

python src/data_factory/dbsherlock/process.py \
    --input_path dataset/dbsherlock/converted/tpcc_500w_test.json \
    --output_path dataset/dbsherlock/processed/tpcc_500w/

python src/data_factory/dbsherlock/process.py \
    --input_path dataset/dbsherlock/converted/tpce_3000_test.json \
    --output_path dataset/dbsherlock/processed/tpce_3000/

Train and Evaluate

We provide the experiment scripts under the folder ./scripts. You can reproduce the experiment results with the below script:

bash ./scripts/experiment/DBS.sh

or you can run the below commands to train and evaluate the model step by step.

Training

python main.py \
    --anormly_ratio 2 \
    --num_epochs 10  \
    --batch_size 256  \
    --input_c 200 \
    --output_c 200 \
    --win_size 25 \
    --step_size 25 \
    --dataset DBS \
    --data_path dataset/processed_dataset \
    --mode train

Evaluating

python main.py \
    --anormly_ratio 2 \
    --num_epochs 10 \
    --batch_size 256 \
    --input_c 200 \
    --output_c 200 \
    --win_size 25 \
    --step_size 25 \
    --pretrained_model 20 \
    --dataset DBS \
    --data_path dataset/processed_dataset \
    --mode test

Testing

python ...

Reference

This respository is based on Anomaly Transformer.

@inproceedings{
xu2022anomaly,
title={Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy},
author={Jiehui Xu and Haixu Wu and Jianmin Wang and Mingsheng Long},
booktitle={International Conference on Learning Representations},
year={2022},
url={https://openreview.net/forum?id=LzQQ89U1qm_}
}

lykig95 / Anomaly_Explanation