Screw class seems fail to reach the reported detetcion AUROC
HuWeiYu opened this issue · comments
There is no denying that promptAD is a very impressive work and thanks for the open resoursing for the knowledge broadcast in the field of AD.
But for those who is familiar with the mvtec dataset, there is no dout that the screw class is the most challenging one because of the none-allignment(basicly the rotation). In your paper , you report that in one-shot the auroc is blow 70,and then in two-shot setting the metric reach 98,and in four shot the metric just rocket to 100? Bro, u got be kidding me, since u even miswrite the carpet class with Garpet in your supplymentary table. I don't want to be mean, but you know, the data i see is just unrealist to me. And I run your code, and in each shot ,the detection auroc of screw is around 70.
And I think maybe the blame is not on you, so I check your baseline code the repo(WinClip) that be reimplemented by Mr.caoyunkang. In Cao's reimplentation, WinCLIP cannaot reach the reported detection auroc of amazon reported in class screw too,with a large margin of 20. Which makes me concused aboat which data should i really quote, the ones written in the paper, or the one i saw with my own eyes.
Thanks for the correction, we will further check the paper for similar errors. I re-checked the training log and found that the results recorded in the supplementary material for 2-shot and 4-shot Screw and Toothbrush seemed to be reversed. The following screenshots are the training logs for a certain random seed under 2-shot and 4-shot, respectively