khundman / telemanom

A framework for using LSTMs to detect anomalies in multivariate time series data. Includes spacecraft anomaly data and experiments from the Mars Science Laboratory and SMAP missions.

Home Page:https://arxiv.org/abs/1802.04431

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Table 2 F_Beta Scores don't add up

pshagnea opened this issue · comments

Looking at table two, there's something amiss. Under the first heading, "Non-Parametric w/ Pruning (p = 0.13)", the Precision and Recall scores are equal but the F_0.5 score is 0.71. If the precision and recall are equal the score should also be the same, no matter what the Beta is.

Thresholding Approach Precision Recall F0.5 score
Non-Parametric w/ Pruning (p = 0.13)
MSL 92.6% 69.4% 0.69
SMAP 85.5% 85.5% 0.71
Total 87.5% 80.0% 0.71

I calculated the F_0.5 scores for a few other rows in the table based off of the precision and recall and was unable to get the same resulting score as the paper. Am I interpreting this table correctly?

Good catch, others have found this as well - see answer in #36

Thanks for addressing it!
Re-calculating the F_beta score with the precision and recall, the performance is higher than reported in the paper.

               Precision Recall F_0.5_original F_0.5_New
Non-Parametric w/ Pruning (p = 0.13)	
        MSL     0.926 	0.694	0.69	0.87
	SMAP	0.855	0.855	0.71	0.86
	Total	0.875	0.8	0.71	0.86
Non-Parametric w/out Pruning (p = 0)	
        MSL     0.758	0.694	0.61	0.74
	SMAP	0.43     0.928	0.44	0.48
	Total	0.489	0.848	0.47	0.53
Gaussian Tail (ϵnorm = 0.0001)	
        MSL     0.842	0.444	0.54	0.71
	SMAP	0.885	0.783	0.71	0.86
	Total	0.875	0.667	0.66	0.82
Gaussian Tail (ϵnorm = 0.01)	
        MSL     0.613	0.528	0.48	0.59
	SMAP	0.824	0.812	0.68	0.82
	Total	0.758	0.714	0.62	0.75
Gaussian Tail w/ Pruning (ϵnorm = 0.01,p = 0.13)	
        MSL     0.882	0.417	0.54	0.72
	SMAP	0.927	0.739	0.71	0.88
	Total	0.917	0.629	0.66	0.84