How to Monitor NLP Models?

[Riley Hun]

Hi Jeremy,

I'm following your template for a POC, and it's been very helpful. I'm creating a REST API for an NLP model (Multinomial Naive Bayes) and I'm not sure how to monitor this particular model when the predictions are classes instead of float values like the wine quality prediction model. How would the prometheus instrumentation be used to capture metrics for classification models?

Thanks,

Riley

[Jeremy Jordan]

Hi @rileyhun! There's a few things you can do here:

• log the probability for each class (you can use labels on the metric as long as it's sufficiently low cardinality)
• log the word count for your input text
• log the character count for your input text
• log the number of unique characters in your input text

I'm sure there's more, these are just the first couple ideas off the top of my head.

[Riley Hun]

This is helpful! Thanks @jeremyjordan. I'll give it a try

Hi @rileyhun! There's a few things you can do here:

• log the probability for each class (you can use labels on the metric as long as it's sufficiently low cardinality)
• log the word count for your input text
• log the character count for your input text
• log the number of unique characters in your input text

I'm sure there's more, these are just the first couple ideas off the top of my head.

This is helpful! Thanks @jeremyjordan. I'll give it a try

[Sujan Shirol]

Hi @jeremyjordan,

Thank you for listing out what to log. I'm working on a similar project. What do you think is the right tool to log and monitor. I'm looking into Prometheus and Grafana. What are your thoughts?

[Rohit Rangarajan]

@rileyhun Hi! I am working on a text classification problem, and would like to use prometheus and grafana to monitor. Would you be able to share your code for me to understand how it works?