Language Test: Scenarios with BrainResponder
BramKraai opened this issue · comments
Hey,
I manually tested the language test scenarios in a live application: worked through them all and looked if they were working. All entries with a tick box are working! For every entry that is not working 100%, the log is provided. Hope this is useful for debugging :)
These tests were done on the latest Develop branch!
Here we go:
- you are a robot
- are you a robot
- who is a robot
- who is the robot (Google picked this up, resulting in no answer, which might be okay, actually)
10/16/19 13:25:16 INFO pepper.Chat (Bram) 006 Bram: "who is the robot"
10/16/19 13:25:16 DEBUG pepper.pepper.language.analyzer initial RDF: {'predicate': u'is', 'object': u'the-robot', 'subject': ''}
10/16/19 13:25:16 DEBUG pepper.pepper.language.analyzer after VP: {'predicate': u'be', 'object': u'the-robot', 'subject': ''}
10/16/19 13:25:16 DEBUG pepper.pepper.language.analyzer after NP: {'predicate': u'be', 'object': u'the-robot', 'subject': ''}
10/16/19 13:25:16 DEBUG pepper.pepper.language.analyzer final RDF: {'predicate': {'text': u'be', 'type': u'verb.stative'}, 'object': {'text': u'the-robot', 'type': [u'noun.artifact']}, 'subject': {'text': '', 'type': []}} {'neg': False, 'wh_word': {u'response': u'person'}}
10/16/19 13:25:16 INFO pepper.pepper.language.analyzer RDF subject: {"text": "", "type": []}
10/16/19 13:25:16 INFO pepper.pepper.language.analyzer RDF predicate: {"text": "be", "type": "verb.stative"}
10/16/19 13:25:16 INFO pepper.pepper.language.analyzer RDF object: {"text": "the-robot", "type": ["noun.artifact"]}
10/16/19 13:25:16 DEBUG pepper.RdfBuilder Booted
10/16/19 13:25:17 DEBUG pepper.BrainResponder TRIPLE: ?_be_the-robot [_->_artifact or object])
10/16/19 13:25:17 INFO pepper.LongTermMemory Triple: ?_be_the-robot [_->_artifact or object])
10/16/19 13:25:17 DEBUG pepper.LongTermMemory Posting query
10/16/19 13:25:17 DEBUG pepper.BrainResponder REPLY: I know things usually be artifact, but I do not know this case
10/16/19 13:25:17 INFO pepper.ResponsePicker BrainResponder (0.18s)
10/16/19 13:25:17 INFO pepper.Chat (Bram) 007 Leolani: "I know things usually be artifact, but I do not know this case"
- you live in Amsterdam
works, but does not form a reply, resulting in another responder to be called.
10/16/19 13:26:06 INFO pepper.Chat (Bram) 014 Bram: "you live in Amsterdam"
10/16/19 13:26:06 DEBUG pepper.pepper.language.analyzer initial RDF: {'predicate': u'live', 'object': u'in-Amsterdam', 'subject': u'you'}
10/16/19 13:26:06 DEBUG pepper.pepper.language.analyzer after VP: {'predicate': u'live', 'object': u'in-Amsterdam', 'subject': u'you'}
10/16/19 13:26:06 DEBUG pepper.pepper.language.analyzer after NP: {'predicate': u'live', 'object': u'in-Amsterdam', 'subject': u'leolani'}
10/16/19 13:26:06 DEBUG pepper.pepper.language.analyzer final RDF: {'predicate': {'text': u'live-in', 'type': [u'verb.stative', 'preposition']}, 'object': {'text': u'Amsterdam', 'type': u'noun.location'}, 'subject': {'text': u'leolani', 'type': ['robot']}} {'neg': False}
10/16/19 13:26:06 INFO pepper.pepper.language.analyzer RDF subject: {"text": "leolani", "type": ["robot"]}
10/16/19 13:26:06 INFO pepper.pepper.language.analyzer RDF predicate: {"text": "live-in", "type": ["verb.stative", "preposition"]}
10/16/19 13:26:06 INFO pepper.pepper.language.analyzer RDF object: {"text": "Amsterdam", "type": "noun.location"}
10/16/19 13:26:06 DEBUG pepper.RdfBuilder Booted
10/16/19 13:26:06 DEBUG pepper.BrainResponder TRIPLE: leolani_live-in_amsterdam [robot_->_location])
10/16/19 13:26:06 DEBUG pepper.LocationReasoner Posting query
10/16/19 13:26:06 INFO pepper.LongTermMemory Triple: leolani_live-in_amsterdam [robot_->_location])
10/16/19 13:26:06 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:26:06 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:26:06 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:26:06 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:26:06 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:26:07 INFO pepper.LongTermMemory Posting triples
10/16/19 13:26:07 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:26:07 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:26:07 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:26:07 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:26:07 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:26:07 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:26:07 DEBUG pepper.LongTermMemory Posting query
10/16/19 13:26:07 DEBUG pepper.BrainResponder REPLY:
10/16/19 13:26:07 INFO pepper.ResponsePicker GreetingResponder (0.00s)
10/16/19 13:26:07 INFO pepper.Chat (Bram) 015 Leolani: "What's new?, Bram!"
-
who lives in Amsterdam
-
do you live in Amsterdam
-
I am a man
again: works, but does not form a reply, resulting in another responder to be called.
10/16/19 13:27:01 INFO pepper.Chat (Bram) 024 Bram: "I am a man"
10/16/19 13:27:01 DEBUG pepper.pepper.language.analyzer initial RDF: {'predicate': u'am', 'object': u'a-man', 'subject': u'I'}
10/16/19 13:27:01 DEBUG pepper.pepper.language.analyzer after VP: {'predicate': u'be', 'object': u'a-man', 'subject': u'I'}
10/16/19 13:27:01 DEBUG pepper.pepper.language.analyzer after NP: {'predicate': u'be', 'object': u'a-man', 'subject': 'Bram'}
10/16/19 13:27:01 DEBUG pepper.pepper.language.analyzer final RDF: {'predicate': {'text': u'be', 'type': u'verb.stative'}, 'object': {'text': u'a-man', 'type': [u'noun.person']}, 'subject': {'text': 'Bram', 'type': ['person']}} {'neg': False}
10/16/19 13:27:01 INFO pepper.pepper.language.analyzer RDF subject: {"text": "Bram", "type": ["person"]}
10/16/19 13:27:01 INFO pepper.pepper.language.analyzer RDF predicate: {"text": "be", "type": "verb.stative"}
10/16/19 13:27:01 INFO pepper.pepper.language.analyzer RDF object: {"text": "a-man", "type": ["noun.person"]}
10/16/19 13:27:01 DEBUG pepper.RdfBuilder Booted
10/16/19 13:27:01 DEBUG pepper.BrainResponder TRIPLE: bram_be_a-man [person_->_person])
10/16/19 13:27:01 DEBUG pepper.LocationReasoner Posting query
10/16/19 13:27:01 INFO pepper.LongTermMemory Triple: bram_be_a-man [person_->_person])
10/16/19 13:27:01 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:27:01 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:27:01 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:27:01 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:27:01 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:27:02 INFO pepper.LongTermMemory Posting triples
10/16/19 13:27:02 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:27:02 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:27:02 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:27:02 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:27:02 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:27:02 DEBUG pepper.LongTermMemory Posting query
10/16/19 13:27:02 DEBUG pepper.BrainResponder REPLY:
10/16/19 13:27:02 INFO pepper.ResponsePicker UnknownResponder (0.00s)
10/16/19 13:27:02 INFO pepper.Chat (Bram) 025 Leolani: "Why?"
-
who is a man
-
I like baking cakes
-
what do I like
-
do I like baking cakes
-
who likes baking cakes - you told me you like baking cakes
-
Birds can sing
The <object> part of the triple is empty? Resulting in strange reply!
10/16/19 13:28:09 INFO pepper.Chat (Bram) 036 Bram: "birds can sing"
10/16/19 13:28:09 DEBUG pepper.pepper.language.analyzer initial RDF: {'predicate': u'can', 'object': u'sing', 'subject': u'birds'}
10/16/19 13:28:09 DEBUG pepper.pepper.language.analyzer after VP: {'predicate': u'can', 'object': u'sing', 'subject': u'birds'}
10/16/19 13:28:09 DEBUG pepper.pepper.language.analyzer after NP: {'predicate': u'can', 'object': u'sing', 'subject': u'birds'}
10/16/19 13:28:09 DEBUG pepper.pepper.language.analyzer final RDF: {'predicate': {'text': u'can-sing', 'type': [u'verb.creation']}, 'object': {'text': '', 'type': []}, 'subject': {'text': u'birds', 'type': u'noun.animal'}} {'neg': False}
10/16/19 13:28:09 INFO pepper.pepper.language.analyzer RDF subject: {"text": "birds", "type": "noun.animal"}
10/16/19 13:28:09 INFO pepper.pepper.language.analyzer RDF predicate: {"text": "can-sing", "type": ["verb.creation"]}
10/16/19 13:28:09 INFO pepper.pepper.language.analyzer RDF object: {"text": "", "type": []}
10/16/19 13:28:09 DEBUG pepper.RdfBuilder Booted
10/16/19 13:28:09 DEBUG pepper.BrainResponder TRIPLE: birds_can-sing_? [animal_->_])
10/16/19 13:28:09 DEBUG pepper.TypeReasoner Posting query
10/16/19 13:28:09 DEBUG pepper.TypeReasoner Posting query
10/16/19 13:28:12 DEBUG pepper.TypeReasoner Posting query
10/16/19 13:28:12 DEBUG pepper.TypeReasoner Posting query
10/16/19 13:28:12 DEBUG pepper.TypeReasoner Posting query
10/16/19 13:28:12 DEBUG pepper.TypeReasoner Posting query
10/16/19 13:28:12 DEBUG pepper.LocationReasoner Posting query
10/16/19 13:28:12 INFO pepper.LongTermMemory Triple: birds_can-sing_? [animal_->_])
10/16/19 13:28:12 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:28:12 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:28:12 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:28:12 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:28:12 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:28:13 INFO pepper.LongTermMemory Posting triples
10/16/19 13:28:13 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:28:13 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:28:13 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:28:13 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:28:13 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:28:13 DEBUG pepper.LongTermMemory Posting query
10/16/19 13:28:13 DEBUG pepper.BrainResponder REPLY: Exciting news! I had never heard about before!
10/16/19 13:28:13 INFO pepper.ResponsePicker BrainResponder (4.45s)
10/16/19 13:28:13 INFO pepper.Chat (Bram) 037 Leolani: "Exciting news! I had never heard about before!"
- can birds sing
- what can birds do
no answer from brain
10/16/19 13:28:23 INFO pepper.Chat (Bram) 038 Bram: "what can birds do"
10/16/19 13:28:23 DEBUG pepper.pepper.language.analyzer initial RDF: {'predicate': u'can-do', 'object': '', 'subject': u'birds'}
10/16/19 13:28:23 DEBUG pepper.pepper.language.analyzer after VP: {'predicate': u'can', 'object': '', 'subject': u'birds'}
10/16/19 13:28:23 DEBUG pepper.pepper.language.analyzer after NP: {'predicate': u'can', 'object': '', 'subject': u'birds'}
10/16/19 13:28:23 DEBUG pepper.pepper.language.analyzer final RDF: {'predicate': {'text': u'can', 'type': None}, 'object': {'text': '', 'type': []}, 'subject': {'text': u'birds', 'type': u'noun.animal'}} {'neg': False, 'aux': {u'person': [u'first', u'second', u'third'], u'tense': u'present', u'number': [u'singular', u'plural']}, 'wh_word': {u'response': u'object'}}
10/16/19 13:28:23 INFO pepper.pepper.language.analyzer RDF subject: {"text": "birds", "type": "noun.animal"}
10/16/19 13:28:23 INFO pepper.pepper.language.analyzer RDF predicate: {"text": "can", "type": null}
10/16/19 13:28:23 INFO pepper.pepper.language.analyzer RDF object: {"text": "", "type": []}
10/16/19 13:28:23 DEBUG pepper.RdfBuilder Booted
10/16/19 13:28:23 DEBUG pepper.BrainResponder TRIPLE: birds_can_? [animal_->_])
10/16/19 13:28:23 INFO pepper.LongTermMemory Triple: birds_can_? [animal_->_])
10/16/19 13:28:23 DEBUG pepper.LongTermMemory Posting query
10/16/19 13:28:23 DEBUG pepper.BrainResponder REPLY: I know animal usually can things, but I do not know this case
10/16/19 13:28:23 INFO pepper.ResponsePicker BrainResponder (0.13s)
10/16/19 13:28:23 INFO pepper.Chat (Bram) 039 Leolani: "I know animal usually can things, but I do not know this case"
-
who can sing
-
Robots can't eat food
Think all goes well here, because {'neg': True}
But again, no reply is formed, resulting in another responder to respond
10/16/19 13:29:01 INFO pepper.Chat (Bram) 044 Bram: "robots can't eat food"
10/16/19 13:29:01 DEBUG pepper.pepper.language.analyzer initial RDF: {'predicate': u'can-not-eat', 'object': u'food', 'subject': u'robots'}
10/16/19 13:29:01 DEBUG pepper.pepper.language.analyzer after VP: {'predicate': u'can-eat', 'object': u'food', 'subject': u'robots'}
10/16/19 13:29:01 DEBUG pepper.pepper.language.analyzer after NP: {'predicate': u'can-eat', 'object': u'food', 'subject': u'robots'}
10/16/19 13:29:01 DEBUG pepper.pepper.language.analyzer final RDF: {'predicate': {'text': u'can-eat', 'type': [u'verb.consumption']}, 'object': {'text': u'food', 'type': u'noun.Tops'}, 'subject': {'text': u'robots', 'type': u'noun.artifact'}} {'neg': True}
10/16/19 13:29:01 INFO pepper.pepper.language.analyzer RDF subject: {"text": "robots", "type": "noun.artifact"}
10/16/19 13:29:01 INFO pepper.pepper.language.analyzer RDF predicate: {"text": "can-eat", "type": ["verb.consumption"]}
10/16/19 13:29:01 INFO pepper.pepper.language.analyzer RDF object: {"text": "food", "type": "noun.Tops"}
10/16/19 13:29:01 DEBUG pepper.RdfBuilder Booted
10/16/19 13:29:01 DEBUG pepper.BrainResponder TRIPLE: robots_can-eat_food [artifact or object_->_Tops])
10/16/19 13:29:01 DEBUG pepper.LocationReasoner Posting query
10/16/19 13:29:01 INFO pepper.LongTermMemory Triple: robots_can-eat_food [artifact or object_->_tops])
10/16/19 13:29:01 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:29:01 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:29:01 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:29:01 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:29:01 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:29:01 INFO pepper.LongTermMemory Posting triples
10/16/19 13:29:02 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:29:02 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:29:02 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:29:02 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:29:02 DEBUG pepper.ThoughtGenerator Posting query
10/16/19 13:29:02 DEBUG pepper.LongTermMemory Posting query
10/16/19 13:29:02 DEBUG pepper.BrainResponder REPLY:
10/16/19 13:29:02 INFO pepper.ResponsePicker UnknownResponder (0.00s)
10/16/19 13:29:02 INFO pepper.Chat (Bram) 045 Leolani: "I'm guessing you told me: robots can't eat food, but I have no clue!"
- can robots eat food
Pepper thinks robot CAN eat food, while I just told her robots cannot.
Observe that {'neg': False}
10/16/19 13:29:21 INFO pepper.Chat (Bram) 046 Bram: "can robots eat food"
10/16/19 13:29:21 DEBUG pepper.pepper.language.analyzer initial RDF: {'predicate': u'can-eat', 'object': u'food', 'subject': u'robots'}
10/16/19 13:29:21 DEBUG pepper.pepper.language.analyzer after VP: {'predicate': u'can-eat', 'object': u'food', 'subject': u'robots'}
10/16/19 13:29:21 DEBUG pepper.pepper.language.analyzer after NP: {'predicate': u'can-eat', 'object': u'food', 'subject': u'robots'}
10/16/19 13:29:21 DEBUG pepper.pepper.language.analyzer final RDF: {'predicate': {'text': u'can-eat', 'type': [u'verb.consumption']}, 'object': {'text': u'food', 'type': u'noun.Tops'}, 'subject': {'text': u'robots', 'type': u'noun.artifact'}} {'neg': False}
10/16/19 13:29:21 INFO pepper.pepper.language.analyzer RDF subject: {"text": "robots", "type": "noun.artifact"}
10/16/19 13:29:21 INFO pepper.pepper.language.analyzer RDF predicate: {"text": "can-eat", "type": ["verb.consumption"]}
10/16/19 13:29:21 INFO pepper.pepper.language.analyzer RDF object: {"text": "food", "type": "noun.Tops"}
10/16/19 13:29:21 DEBUG pepper.RdfBuilder Booted
10/16/19 13:29:21 DEBUG pepper.BrainResponder TRIPLE: robots_can-eat_food [artifact or object_->_Tops])
10/16/19 13:29:21 INFO pepper.LongTermMemory Triple: robots_can-eat_food [artifact or object_->_Tops])
10/16/19 13:29:21 DEBUG pepper.LongTermMemory Posting query
10/16/19 13:29:21 DEBUG pepper.BrainResponder REPLY: you told me Robots can eat food
10/16/19 13:29:21 INFO pepper.ResponsePicker BrainResponder (0.15s)
10/16/19 13:29:21 INFO pepper.Chat (Bram) 047 Leolani: "you told me Robots can eat food"
- who can eat food
Same problem as above, the negation doesn't come through
10/16/19 13:29:29 INFO pepper.Chat (Bram) 048 Bram: "who can eat food"
10/16/19 13:29:29 DEBUG pepper.pepper.language.analyzer initial RDF: {'predicate': u'can-eat', 'object': u'food', 'subject': ''}
10/16/19 13:29:29 DEBUG pepper.pepper.language.analyzer after VP: {'predicate': u'can-eat', 'object': u'food', 'subject': ''}
10/16/19 13:29:29 DEBUG pepper.pepper.language.analyzer after NP: {'predicate': u'can-eat', 'object': u'food', 'subject': ''}
10/16/19 13:29:29 DEBUG pepper.pepper.language.analyzer final RDF: {'predicate': {'text': u'can-eat', 'type': [u'verb.consumption']}, 'object': {'text': u'food', 'type': u'noun.Tops'}, 'subject': {'text': '', 'type': []}} {'neg': False, 'wh_word': {u'response': u'person'}}
10/16/19 13:29:29 INFO pepper.pepper.language.analyzer RDF subject: {"text": "", "type": []}
10/16/19 13:29:29 INFO pepper.pepper.language.analyzer RDF predicate: {"text": "can-eat", "type": ["verb.consumption"]}
10/16/19 13:29:29 INFO pepper.pepper.language.analyzer RDF object: {"text": "food", "type": "noun.Tops"}
10/16/19 13:29:29 DEBUG pepper.RdfBuilder Booted
10/16/19 13:29:29 DEBUG pepper.BrainResponder TRIPLE: ?_can-eat_food [_->_Tops])
10/16/19 13:29:29 INFO pepper.LongTermMemory Triple: ?_can-eat_food [_->_Tops])
10/16/19 13:29:29 DEBUG pepper.LongTermMemory Posting query
10/16/19 13:29:29 DEBUG pepper.BrainResponder REPLY: you told me Robots can eat food
10/16/19 13:29:29 INFO pepper.ResponsePicker BrainResponder (0.14s)
10/16/19 13:29:29 INFO pepper.Chat (Bram) 049 Leolani: "you told me Robots can eat food"
- Elena can see a laptop
- what can Selene see
- who can see a laptop
These come from test/language/data/statements.txt
and they all work!
- lana can read a book: lana can-read a-book
- lana must read: lana must-read
- john will come to Amsterdam: john will-come-to Amsterdam
- you have been in paris: leolani have-been-in paris
- animals must drink water: animals must-drink water
- food should be healthy: food should-be healthy
- lana is reading a book: lana read a-book
- you are going to England: leolani go-to england
- you like talking to people: leolani like talking-to-people
- john enjoys watching movies: john enjoy watching-movies
- lana read a book: lana read a-book
- I live in the Netherlands: lenka live-in the-netherlands
- you live in this office: leolani live-in this-office
- you are a robot: leolani be a-robot
- I am a person: lenka be a-person
- humans can eat food: humans can-eat food
- swimming is a sport: swimming be a-sport
- bram loves sport: bram love sport
Google just won't accept bram loves sport and makes Bram love Sport from it. Annoying...
- you come from france: leolani come-from france
NOTE: In more than 50% of the cases there is no reply, resulting in the UnknownResponder to be triggered (since there is no other responder that can deal with these sentences). So she will say something in the vein of: "I have no clue what you're talking about", while the sentence was accurately parsed, the triple nicely made and the brain correctly updated.