Results replication mentioned in the paper
aks1207 opened this issue · comments
aks1207 commented
Hi,
I read the paper and it was a great read. I have a couple of questions, namely
- I tried running the experiment on GasSensorData using the below mentioned parameters
- batch_size=200
- latent_size=128
- learning_rate=0.001
- net_type='mlp'
- num_epochs=100
- num_folds=5
- num_layers=3
- oodclass_idx=0
- reg_lambda=1.0
and got the results as
== Final results ==
The best ID accuracy on "data_gas" test samples : 96.14 ( 1.163)
The best OOD accuracy on "class_0" test samples :
TNR85 TNR95 AUROC DTACC AUIN AUOUT
49.22 ( 4.726) 30.70 ( 6.331) 71.50 ( 1.772) 67.74 ( 2.368) 54.77 ( 1.536) 81.01 ( 1.656).
But the paper mentions better results. Can you kindly help me if I have missed something in the experimentation ?
- In the paper one of the baselines is Mahalnobis Distance utilisation. Can you eloborate on how it is being used ?
- Is it being used for distance calculation instead of equation 4 mentioned in the paper ?
- If yes, Is the mahalnobis distance covariance matrix is calculated using all data points representation that belong to a particular class?
- What is the covariance matrix during testing phase as we donot know what class each point belongs to? Do we still use the one built from using training samples ?
- Does the score function utilise this Mahalnobis distance and is it then followed by push and pull loss ?
- If not, what is the setup for Mahalnobis distance based baseline ?
3.In section 3.1.1 it was mentioned that BCD (Block coordinate distance is used for optimisation), but I could see mini batch gradient descent with Adam optimiser in the code. Is my interpretation wrong? If so, kindly correct me.