Accessing probabilities of bagged models

[Aman Malali]

Hi, I was wondering if there's any way of accessing the probabilities of each of the bagged models which are averaged to get the output of predict_proba() for the L1 models? This would be helpful to be able to calculate uncertainties for each of the models as well as uncertainty for the entire weighted ensemble

Thanks!

[Nick Erickson]

Hi @amanmalali, this is a good feature request, and I've been playing around with ideas on how to add this in a nice way (want to also make the fold models viewable in .leaderboard() for example). This is on our radar to add in an upcoming release.

[Aman Malali]

Hi @Innixma, any way I could help? I've been modifying the repo to access this, so if there's a branch it's being worked on I could try to help!

[Nick Erickson]

@amanmalali Thanks for your interest! if you have a branch with the changes, I could take a look as a reference. For this one it is a bit in-depth for the official implementation, since I want the fold models to also be viewable in predictor.leaderboard(..., show_folds=True).

For the implementation I would be basing things off of BaggedEnsembleModel.predict_proba_children to fetch the fold predictions. If you have an implementation for calculating uncertainty of a bagged model based on the child predictions, that could be valuable to share as a reference code implementation. We could add it as a new method to BaggedEnsembleModel, such as BaggedEnsembleModel.compute_uncertainty(X_holdout)