Wrong output for regression

[AmosJoseph]

Hi, it's exciting to find this nice python library for Bayesian deep neural network!

I convert an existing deterministic deep neural network (dnn, with good accuracy) model to Bayesian deep neural network (bnn) model. Below is the code:

from sklearn.metrics import mean_squared_error, r2_score
import torch
from bayesian_torch.models.dnn_to_bnn import dnn_to_bnn, get_kl_loss

const_bnn_prior_parameters = {
"prior_mu": 0.0,
"prior_sigma": 1.0,
"posterior_mu_init": 0.0,
"posterior_rho_init": -3.0,
"type": "Reparameterization", # Flipout or Reparameterization
"moped_enable": False, # True to initialize mu/sigma from the pretrained dnn weights
"moped_delta": 0.5,
}

model = Net()
dnn_to_bnn(model, const_bnn_prior_parameters)

criterion = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), 0.0001)

output = model(X_train)
kl = get_kl_loss(model)
ce_loss = criterion(output, y_train)
loss = ce_loss + kl / 20

loss.backward()
optimizer.step()

predictions_mc = []

for _ in range(10000):
output = model(X_test)
predictions_mc.append(output)

predictions_ = torch.stack(predictions_mc)
predictive_mean = torch.mean(predictions_, dim=0)
predictive_std = torch.std(predictions_, dim=0)

print("Mean squared error: %.2f" % mean_squared_error(y_test, predictive_mean.detach().numpy()))
print("R2 score : %.2f" % r2_score(y_test, predictive_mean.detach().numpy()))

from bayesian_torch.utils.util import predictive_entropy, mutual_information

predictive_uncertainty = predictive_entropy(predictions_.data.cpu().numpy())
model_uncertainty = mutual_information(predictions_.data.cpu().numpy())

Running output as below:
Mean squared error: 6.66
R2 score : -1.72
D:\anaconda3\envs\xxx\lib\site-packages\bayesian_torch\utils\util.py:42: RuntimeWarning: invalid value encountered in log
return -1 * np.sum(prob * np.log(prob + 1e-15), axis=-1)

But what's wrong? Could you please give some advice?

Best regards!

[Ranganath Krishnan]

@AmosJoseph The predictive_entropy and mutual_information are uncertainty metrics for classification task (requires a list of probabilities that sum to 1), and are not not applicable for regression model. You may consider using variance/std to estimate uncertainty i.e. np.var(predictions_.data.cpu().numpy()).
I hope this helps.

[AmosJoseph]

Many thanks for your help! There is something wrong, R2 score : -1.72. Please see the code:
bnnCode.txt

Could you give an example for regression?

And what's the recommended number of mc_run, at least 1e5?

Best!

[Ranganath Krishnan]

@AmosJoseph Based on the code snippet that you've shared, it is not clear how many epochs you are training the model and if the training has converged. Can you check the training convergence with MSE loss vs epochs?

MC samples depends on your dataset and model. Typically you can start between 10-100 and see if that helps.