marcotcr / checklist

Beyond Accuracy: Behavioral Testing of NLP models with CheckList

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Any suggestion for checklist for multi-class Intent Prediction models for Chatbots?

Samarthagarwal23 opened this issue · comments

Thanks for sharing the paper and the repository.

For Chatbots, models are trained with 100s of possible intents that use may have.
Any ideas / suggestions on NLP Checkist for such models in your research?

Sorry for the delay in responding. What I would suggest is starting with very simple MFTs, and verifying properties of the output, e.g.

Input: "Please tell me what month it is"
Output expectation: must contain a valid month, whatever else it has

Input: "Do you think it's a good idea to spread lies and misinformation?"
Output expectation: must not contain an affirmation, e.g. 'yes', 'sure', 'I do', etc

But it really depends on what the chatbot is doing, what appropriate / inappropriate behavior is, etc.