There are three ways to preprocess the question, what is the difference ?
gaopeng-eugene opened this issue · comments
if nlp == 'nltk':
ex['question_words'] = word_tokenize(str(s).lower())
elif nlp == 'mcb':
ex['question_words'] = tokenize_mcb(s)
else:
ex['question_words'] = tokenize(s)
Sorry for the late answer @gaopeng-eugene
mcb (what we are using by default)
(https://github.com/Cadene/vqa.pytorch/blob/master/vqa/datasets/vqa_processed.py#L45)
import re
def tokenize_mcb(s):
t_str = s.lower()
for i in [r'\?',r'\!',r'\'',r'\"',r'\$',r'\:',r'\@',r'\(',r'\)',r'\,',r'\.',r'\;']:
t_str = re.sub( i, '', t_str)
print(t_str)
for i in [r'\-',r'\/']:
t_str = re.sub( i, ' ', t_str)
print(t_str)
q_list = re.sub(r'\?','',t_str.lower()).split(' ')
print(q_list)
q_list = list(filter(lambda x: len(x) > 0, q_list))
print(q_list)
return q_list
tokenize_mcb("Hello I'm a co_ol ro-b-ot, wO0O0T ? !")
> hello i'm a co_ol ro-b-ot, wo0o0t !
hello i'm a co_ol ro-b-ot, wo0o0t
hello im a co_ol ro-b-ot, wo0o0t
hello im a co_ol ro-b-ot, wo0o0t
hello im a co_ol ro-b-ot, wo0o0t
hello im a co_ol ro-b-ot, wo0o0t
hello im a co_ol ro-b-ot, wo0o0t
hello im a co_ol ro-b-ot, wo0o0t
hello im a co_ol ro-b-ot, wo0o0t
hello im a co_ol ro-b-ot wo0o0t
hello im a co_ol ro-b-ot wo0o0t
hello im a co_ol ro-b-ot wo0o0t
hello im a co_ol ro b ot wo0o0t
hello im a co_ol ro b ot wo0o0t
['hello', 'im', 'a', 'co_ol', 'ro', 'b', 'ot', 'wo0o0t', '', '']
['hello', 'im', 'a', 'co_ol', 'ro', 'b', 'ot', 'wo0o0t']
Else (https://github.com/Cadene/vqa.pytorch/blob/master/vqa/datasets/vqa_processed.py#L42)
[-."',:? !$#@~()*&^%;[]/\+<>\n=]
Easy!
I should have documented that, sorry haha... anyway it returns this:
> ['Hello', 'I', "'", 'm', 'a', 'co_ol', 'ro', '-', 'b', '-', 'ot', ',', 'wO0O0T', '?', '!']
nltk
convert to lowercase and tokenize the string to split off punctuation other than periods
source: http://www.nltk.org/api/nltk.tokenize.html