Poor performance for simple text
omrishsu opened this issue · comments
I've tested the model with the following text:
"The movie: 'Spider-Man 3' (2007) was directed by Sam Raimi.\nDuration: 156\nGenres: J.K. Simmons\nActor1: James Franco\nActor2: Kirsten Dunst\nActor3: Action, Adventure, Romance\nTopics: sandman, spider man, symbiote, venom, villain\nUrl: http://www.imdb.com/title/tt0413300/?ref_=fn_tt_tt_1\nLanguage: English\nCountry: USA\nContent: PG-13\nIMDB Score: 6.2"
to get the probabilities of each answer i've changed the code a bit:
def _evaluate_qa(encoded_qa_pair, qae_model):
output = qae_model(**encoded_qa_pair)
return torch.nn.functional.softmax(output.logits)
def get_scores(encoded_qa_pairs, qae_model):
scores = {}
qae_model.eval()
with torch.no_grad():
for i in range(len(encoded_qa_pairs)):
scores[i] = _evaluate_qa(encoded_qa_pairs[i], qae_model)
return [v.cpu()[0][0].item() for k, v in sorted(scores.items(), key=lambda item: item, reverse=True)]
The questions and the answers that I get dose not make sense:
Q: "What is the name of the movie?"
A: "IMDB Score: 6.2"
P: 0.9807605743408203
Q: "What is the name of the movie?"
A: "Url: http://www.imdb.com/title/tt0413300/?ref_=fn_tt_tt_1"
P: 0.9784185290336609
Q: "What is the name of the movie?"
A: "Actor2: Kirsten Dunst"
P: 0.0020964324939996004
Q: "What is the name of the movie?"
A: "Country: USA"
P: 0.9956291913986206
Q: "What is the name of the movie?"
A: "Content: PG-13"
P: 0.9960542917251587
Q: "What is the name of the movie?"
A: "Topics: sandman, spider man, symbiote, venom, villain"
P: 0.9946396946907043
Q: "What is the name of the movie?"
A: "The movie: \'Spider-Man 3\' (2007) was directed by Sam Raimi."
P: 0.5569069385528564
Q: "What is the name of the movie?"
A: "Genres: J.K. Simmons"
P: 0.9990692734718323
Q: "What is the name of the movie?"
A: "Actor1: James Franco"
P: 0.9992097616195679
Q: "What is the name of the movie?"
A: "Duration: 156"
P: 0.9709019064903259
Am i missing here something? or this is an "acceptable" quality?
It generates nonsense quite often. It's unusual that all of the questions are the same though. Maybe try a longer text?