Problems with the MNIST experiment
A1exy opened this issue · comments
Dear Sir:
I am very interested in your paper. But some problems occurred when I repeated MNIST experiment. I wrote the test code and adopted the same LOSS function as the training code. Visualize the LOSS value of G and D, but the normal samples and abnormal samples may not be distinguished.
For example, In the training,the number 1 is treated as a normal sample; in the test, the number 0,1,2 and 3 are visualized; the number 1 is red, the number 0 is blue, the number 2 is green, and the number 3 is yellow. The parameters are all default values. The results look good.
But if the number 2 is treated as a normal sample; in the test, the number 0,1,2 and 3 are visualized; the number 2 is red, the number 0 is blue, the number 1 is green, and the number 3 is yellow. The parameters are all default values. The results turned out badly.
I tried adding noise to the test sample and adjusted r_alpha, but none of the results improved significantly.
This question has been bothering me for a few days and How should the LOSS of the test sample be selected，Could you give me some advice. Thank you!
Best wishes!
I also had the same issue. It takes longer than several days to distinguish outliers with normal case '2'.
Now I, however, agree with this paper after tuning up the hyper-parameters. Target class number 2 on MNIST can also be detected with proposed structure, with delicate controls.
As you see, I finally detect outliers with target class both 1 and 2. Both tries watched MNIST training sets for 1K epochs and tested with MNIST test sets.
There are histograms of Detector's outputs for whole test sets.
ALOCC was a very interesting and useful idea! I test it from scratch and I found that finding the threshold could be another new idea which could be useful for extending their project. The ALOCC idea in the pure structure was very nice! I try to use it with adversarial training, I hope it could work in the refining of adversarial examples.