As it has been discussed, Python's SciKit, while it contains a great functionality for computing evaluation metrics of estimators (using cross_val_score), it seems to fail when it comes to computing multiple metrics for the same classifier without trainning it again.
The problem arises because of the scoring parameter of the function which accepts only a single metric name or a single callable.
Module multiscorer of this repo, is a workaround for using any number of metrics in cross_val_score.
To "install" the module simply download the source code and place it in your project's directory.
(Alternativelly, download and add to your project just multiscorer.py file).
From a Python script, you can write:
from multiscorer import MultiScorer
from sklearn.metrics import accuracy_score, precision_score # Scikit's libraries for demonstration
from sklearn.model_selection import cross_val_score
from numpy import average
scorer = MultiScorer({ # Create a MultiScorer instance
'accuracy': (accuracy_score, {}),
'precision': (precision_score, {'average': 'macro'}) # Param 'average' will be passed to precision_score as kwarg
})
...
cross_val_score(clf, X, target, scoring=scorer, cv=10) # Use the function with our socrer. Ignore its result
results = scorer.get_results() # Get a dict of lists containing the scores for each metric
for metric in results.keys(): # Iterate and use the results
print("%s: %.3f" % (metric, average(results[metric])))- You can get results only from specific metrics and/or specific folds (see examples ).
- You can also use your own custom metric functions (see how).
- For a full documentation see Wiki.
- Original documentation of SciKit: http://scikit-learn.org/stable/
This module was something I had the need for while working with Scikit's libraries and I just thought it might help somebody.
For questions, bugs, suggestions etc, feel free to contact me or submit a Pull Request.