Question about the parameters
KcAcoZhang opened this issue · comments
Hello, can I ask why you set the batch_size=7919, and whether the z_size means latent space or not? Did you use the down-sampling rate mentioned in the article USAD? Thanks for your answer.
Hello! The unusual value for the batch size aimed only to speed up the training process, I left it like that but you should be free to change it affecting only the training time. The z_size is actually the multiplication between the window size = 12 and the latent size = 10, and it indeed corresponds to the actual dimension of the latent space. I did not implement the experiments regarding the down-sampling, sorry.
OK, thanks for your help.
Hello, I have another question. I notice that you use the windows_normal_test+windows_attack to be the test_loader, but in the SWaT dataset, it has Normal/Attack label and you drop it, and not all the windows_attack data is abnormal. Thanks for your answer.
Hello again :)
This is definitely an interesting point. The truth is that I reimplemented the code by myself without a strict supervision of the author of the paper, and since he sent me only a simple scratch of his implementation with dataloaders already implemented, I didn't double check the dataset and the labels. So you can be right, if that is the case in the next days I will correct the mistake. Let me know if you try before and get better results.
Hi, I have just read the paper and studied your code, then I found out the question. Until now I still not try to do it. I'm looking forward to your correction and I will try to do it now.
I also notice this issue. and I try to do it.but the permence of model is very pool.I am confused about it.
Hi @severous what do you mean by permence?
I have the same problem why all attack windows are abnormal?
I have the same problem why all attack windows are abnormal?
This problem has been solved in this implementation.
https://github.com/robustml-eurecom/usad
Thanks
I have the same problem why all attack windows are abnormal?
This problem has been solved in this implementation.
https://github.com/robustml-eurecom/usadThanks
Hello, thanks for your reply! I use the new code to do anomaly detection but I get bad results in f1_score. It' performance is not as good as the original paper.
Hi,
Could you please provide us with some more details regarding your issue? Which dataset?
I have the same problem why all attack windows are abnormal?
This problem has been solved in this implementation.
https://github.com/robustml-eurecom/usad
ThanksHello, thanks for your reply! I use the new code to do anomaly detection but I get bad results in f1_score. It' performance is not as good as the original paper.
Hi,
Could you please provide us with some more details regarding your issue? Which dataset?I have the same problem why all attack windows are abnormal?
This problem has been solved in this implementation.
https://github.com/robustml-eurecom/usad
ThanksHello, thanks for your reply! I use the new code to do anomaly detection but I get bad results in f1_score. It' performance is not as good as the original paper.
I have solved this problem！The original paper selected the best F1_score, I use this way to achieve a similar result in SWaT. Thanks!
@meihuameii Hello, can you explain how to achieve a similar result in SWaT?
Hi,
Could you please provide us with some more details regarding your issue? Which dataset?I have the same problem why all attack windows are abnormal?
This problem has been solved in this implementation.
https://github.com/robustml-eurecom/usad
ThanksHello, thanks for your reply! I use the new code to do anomaly detection but I get bad results in f1_score. It' performance is not as good as the original paper.
I have solved this problem！The original paper selected the best F1_score, I use this way to achieve a similar result in SWaT. Thanks!
Yet I select the best F1_score, it is about 74%, which is still far below that in the paper
@meihuameii Hello, can you explain how to achieve a similar result in SWaT?
Hi,
I've stumbled upon the same problem and found that increasing threshold can achieve similar results. Here are parameters and scores that I used on one of the runs:
window_size = 20
BATCH_SIZE = 1000
N_EPOCHS = 50
hidden_size = 20
Then I increased threshold to 10 and found the predictions with:
threshold = 10
y_pred_ = np.zeros(y_pred.shape[0])
y_pred_[y_pred >= threshold] = 1Classification report:
precision recall f1-score support
0.0 0.95 0.99 0.97 394613
1.0 0.88 0.63 0.74 55286
accuracy 0.94 449899
macro avg 0.92 0.81 0.85 449899
weighted avg 0.94 0.94 0.94 449899
@meihuameii Hello, can you explain how to achieve a similar result in SWaT?
Hi, I've stumbled upon the same problem and found that increasing threshold can achieve similar results. Here are parameters and scores that I used on one of the runs:
window_size = 20 BATCH_SIZE = 1000 N_EPOCHS = 50 hidden_size = 20
Then I increased
thresholdto10and found the predictions with:threshold = 10 y_pred_ = np.zeros(y_pred.shape[0]) y_pred_[y_pred >= threshold] = 1Classification report:
precision recall f1-score support 0.0 0.95 0.99 0.97 394613 1.0 0.88 0.63 0.74 55286 accuracy 0.94 449899 macro avg 0.92 0.81 0.85 449899 weighted avg 0.94 0.94 0.94 449899
Hi,
Thank you for sharing your research. Here is my best score and parameters.
I used StandardScaler for preprocessing and I used y_test y_test=np.concatenate([np.zeros(windows_normal_test.shape[0]), np.ones(windows_attack.shape[0])]) instead of y_test = [1.0 if (np.sum(window) > 0) else 0 for window in windows_labels ]
window_size = 8 BATCH_SIZE = 7919 N_EPOCHS = 200 hidden_size = 20
I checked this score using sklearn.metrics
recall 0.8068351296145237
precision 0.8836299019011222
f1 0.8434881918763455
and did you mean use threshold and prediction in this way?
y_pred=np.concatenate([torch.stack(results[:-1]).flatten().detach().cpu().numpy(), results[-1].flatten().detach().cpu().numpy()]) threshold = 10 y_pred_ = np.zeros(y_pred.shape[0]) y_pred_[y_pred >= threshold] = 1
Im my case, i got recall 0.0852990924871808 precision 1.0 f1 0.1571900190051773
and did you mean use threshold and prediction in this way?
@soemthlng Yes, I increased threshold to trade of recall for f1 (or so I thought, I'm still learning). In my case the score before increasing the threshold was similar to:
precision recall f1-score support
0.0 0.95 0.72 0.82 394901
1.0 0.26 0.72 0.38 55006
accuracy 0.72 449907
macro avg 0.61 0.72 0.60 449907
weighted avg 0.86 0.72 0.76 449907
How did you select threshold?
@finloop
In this case, recall 0.0852990924871808 precision 1.0 f1 0.1571900190051773 I select threshold to 10, but score is not good.
This is my code.
threshold = 10
y_pred_ = np.zeros(y_pred.shape[0])
y_pred_[y_pred >= threshold] = 1
y_test=np.concatenate([np.zeros(windows_normal_test.shape[0]), np.ones(windows_attack.shape[0])])
print("recall", recall_score(y_test,y_pred_))
print("precision", precision_score(y_test, y_pred_))
print("f1", f1_score(y_test, y_pred_))
I have 2 questions.
- Did you modify
y_test? - Did you change the preprocessor function? I used StandardScaler instead of MinMaxScaler.
- I did not modify the y_test (I used the one provided in the repo).
- Yes I used StandardScaler too.
You can check out the whole USAD.ipynb file.
I think I found the issue:
This line in your code could cause the divergence in scores:
y_test=np.concatenate([np.zeros(windows_normal_test.shape[0]), np.ones(windows_attack.shape[0])])This line creates a long array of zeros and ones like [0,0,0,0 ... 1,1,1,1], it doesn't take into account that not all windows in windows_attack are 1.
This is what y_test looks like for me:
Those are my predictions:
This is what I think your y_test looks like:
!
I cannot recreate it because windows_normal_test(I assumed it's small) is not present in the original notebook. This would check out with your results:
#threshold=10
y_test=np.concatenate([np.ones(windows_attack.shape[0])])
plt.plot(y_test)
plt.ylim([0,1.5])
print(sklearn.metrics.classification_report(y_test, y_pred_)) precision recall f1-score support
0.0 0.00 0.00 0.00 0
1.0 1.00 0.09 0.16 449899
accuracy 0.09 449899
macro avg 0.50 0.04 0.08 449899
weighted avg 1.00 0.09 0.16 449899
Could you show me your notebook?
I run this code in 'Ununtu 16.04 Server, so I do not have notebook.
This is my code of y_test and y_pred
y_pred=np.concatenate([torch.stack(results[:-1]).flatten().detach().cpu().numpy(), results[-1].flatten().detach().cpu().numpy()])
y_test=np.concatenate([np.zeros(windows_normal_test.shape[0]), np.ones(windows_attack.shape[0])])
and this is my y_test figure
Did not modify the y_test mean that you use this?
y_test = [1.0 if (np.sum(window) > 0) else 0 for window in windows_labels ]
Could you share your full notebook?
I’ll use it only for study.
Did not modify the y_test mean that you use this?
y_test = [1.0 if (np.sum(window) > 0) else 0 for window in windows_labels ]
Yes :)
Could you share your full notebook?
Sure. Here you go: Link to notebook. https://github.com/finloop/usad/blob/dev/USAD.ipynb
@finloop
After test your code, I am confused because of the metrics.
this is classification_report's result
precision recall f1-score support
0.0 0.95 0.99 0.97 394613
1.0 0.91 0.63 0.75 55286
accuracy 0.95 449899
macro avg 0.93 0.81 0.86 449899
weighted avg 0.95 0.95 0.94 449899
I think 1.0 class's f1-score is what we want.
Are you using 0.0 class as true value?
Thanks.
I think 1.0 class's f1-score is what we want.
Yes It is what we want.
Are you using 0.0 class as true value?
What do you mean by true value?.
Class 0.0 is what we consider as normal data.
Class 1.0 - anomalies.
I mean true value is the value that I want to find.
Previously, you said that you got results similar to the results presented in the paper.
Did you mean that the value was based on class 0.0?
Did you mean that the value was based on class 0.0?
No. The f1, recall etc. was based on class 1.0. Also my results were slightly worse than the ones in the paper (check it out if you want, it's in README).
Hi, Could you please provide us with some more details regarding your issue? Which dataset?
I have the same problem why all attack windows are abnormal?
This problem has been solved in this implementation.
https://github.com/robustml-eurecom/usad
ThanksHello, thanks for your reply! I use the new code to do anomaly detection but I get bad results in f1_score. It' performance is not as good as the original paper.
Hello, have you reproduced the results from the original paper (F1 result without point-adjust on the SWAT dataset is 0.7917). I've been very distressed recently that I can't reproduce the results from my original paper. In my code, BATCH_ SIZE = 1024, N_ EPOCHS = 100, hidden_ Size = 20, window_ Size=10. Standard Scaler () or MinMaxScaler () were used for data preprocessing, and downsampling was used in the code, with a downsampling rate of 5. The best results from these methods were only about 0.74. Do you have any skills to make the results similar to those of the original paper? I'd love to wait for your reply! Thank you very much.
Hello,
Has anyone managed to get the results of the paper? if so how?
Thanks!
y_test = [1.0 if (np.sum(window) > 0) else 0 for window in windows_labels ]
Hello, sorry to bother you, may I ask why you set the threshold to 0, after I set the result 1.0 acc value is 0
Hello, sorry to bother you, may I ask why you set the threshold to 0, after I set the result 1.0 acc value is 0
@hisong182
A window is considered anomalous if it contains at least one anomaly. Checkout my notebook: https://github.com/finloop/usad/blob/dev/USAD.ipynb
A window is considered anomalous if it contains at least one anomaly. Checkout my notebook:
Thank you for your answer, before I set the wrong data normalization to MinMaxScaler, after the change the accuracy increased a lot
Thanks. How to get the best result on wadi dataset?
Hi, Could you please provide us with some more details regarding your issue? Which dataset?
I have the same problem why all attack windows are abnormal?
This problem has been solved in this implementation.
https://github.com/robustml-eurecom/usad
ThanksHello, thanks for your reply! I use the new code to do anomaly detection but I get bad results in f1_score. It' performance is not as good as the original paper.
Hello, have you reproduced the results from the original paper (F1 result without point-adjust on the SWAT dataset is 0.7917). I've been very distressed recently that I can't reproduce the results from my original paper. In my code, BATCH_ SIZE = 1024, N_ EPOCHS = 100, hidden_ Size = 20, window_ Size=10. Standard Scaler () or MinMaxScaler () were used for data preprocessing, and downsampling was used in the code, with a downsampling rate of 5. The best results from these methods were only about 0.74. Do you have any skills to make the results similar to those of the original paper? I'd love to wait for your reply! Thank you very much.
I have the same questions too. I will check out the new codes which was uploaded by @finloop https://github.com/finloop/usad/blob/dev/USAD.ipynb