What is the predefined threshold for the likelihood of test samples?
L-KID opened this issue · comments
Hi,
I'm trying to repeat your experiments on MNIST. But to get the F1-score, may I know how you set the likelihood threshold to distinguish the inliers and outliers of the test samples?
Thanks!
I think their idea was pretty nice! It was very interesting for my latest project in our company. But I think your request need further research to achieve the best likelihood threshold. Because most AI researches (also top companies like Google, Microsoft, etc.) need some sampling situation in their models or some specific configuration when they want to report or deploy into any devices, But in a specific situation which I tasted it last week, they set this sensitive hyper-parameter by using some intuition and experience as they mentioned in their paper. I think the authors of this paper was in a busy situation and they couldn't check their profile and maybe if we compose mail to them, they may pay more attention to answer or close the issues. In the end, I'm very excited about their work (ALOCC). I hope you could find it in your project by test it with some specific range as they mentioned in their paper (I start from their hyper-parameter which they mentioned in their code and then find a proper threshold and it was OK). Therefore, If you have any further intuition about that, maybe I could help you.
I think their idea was pretty nice! It was very interesting for my latest project in our company. But I think your request need further research to achieve the best likelihood threshold. Because most AI researches (also top companies like Google, Microsoft, etc.) need some sampling situation in their models or some specific configuration when they want to report or deploy into any devices, But in a specific situation which I tasted it last week, they set this sensitive hyper-parameter by using some intuition and experience as they mentioned in their paper. I think the authors of this paper was in a busy situation and they couldn't check their profile and maybe if we compose mail to them, they may pay more attention to answer or close the issues. In the end, I'm very excited about their work (ALOCC). I hope you could find it in your project by test it with some specific range as they mentioned in their paper (I start from their hyper-parameter which they mentioned in their code and then find a proper threshold and it was OK). Therefore, If you have any further intuition about that, maybe I could help you.
Hi,
Thank you for your detailed reply!
In my opinion, the easiest way is to check the distribution of the likelihood, and then set a threshold with some intuition and experience. BTW, could you please point out where the hyper-parameter of the threshold is in their code? I don't think they gave it out.
Mmm, last month, I test an idea in one application and I configure the threshold via some experiment to detect specific concept. I think as you mentioned, it depends on our empirical data distribution in most of the applications when you want to deploy and launch it (as I have some experience in this part).
Dear @L-KID and @cod3r0k
I hope this work could be covered with your applications. As @cod3r0k and @L-KID said, it depends on each application which anyone wants to use. We could config this before we want to embed it into any applications like any configs. Also, as @L-KID , we could set the initial value with analyzing the likelihood ratio.