Add model validation threshold for acceptance
srowen opened this issue · comments
Sean Owen commented
Right now, when a model is built, several variations of the model can be built in order to select the hyperparameters that result in the best model according to some evaluation metric. However it might also be useful to further require that the metric meet some threshold before it's published. That way, if the 'best' model were quite bad, it would be rejected.
Neeraj commented
so this is another parameter that can be set from oryx.conf or is there more to it? Like the model coming up with a reasonable threshold that evolves with more training data.