Question in gramcount.pl

Question

Question in gramcount.pl

utuzenkyo opened this issue 5 years ago · comments

Hi Li, I'm trying to run PSDVec using Japanese Wikipedia text.

How did you deal with the stopwords in top1gram-wiki.txt? I ask you this because top2gram-wiki.txt is not uploaded on vectors&corpus Dropbox. And why there are no stopwords in top1gram-reuters.txt? I'd like to know when to delete the stopwords?

Sorry if it sound a stupid question.
Thank you!

askerlee · Answer 1 · Tue Jul 16 2019 11:14:20 GMT+0800 (China Standard Time)

Sorry for coming back to your question so late. Very busy these days... If memory serves me, I kept all stopwords because they are actually important for word embedding. Even "the" could indicate the following word might be a noun.

askerlee · Answer 2 · Tue Jul 16 2019 11:16:02 GMT+0800 (China Standard Time)

I remember top2gram-wiki.txt is a huge file (more than 10 GB?) so it's not uploaded.

askerlee · Answer 3 · Tue Jul 16 2019 11:21:48 GMT+0800 (China Standard Time)

I checked and indeed stop words in top1grams-reuters.txt are removed. I forgot why I did that... Maybe in the classification task, I removed stop words as a preprocessing step? But I'm sure when generating word embeddings for Reuters, stop words were kept.

utuzenkyo · Answer 4 · Tue Jul 16 2019 14:26:52 GMT+0800 (China Standard Time)

I checked and indeed stop words in top1grams-reuters.txt are removed. I forgot why I did that... Maybe in the classification task, I removed stop words as a preprocessing step? But I'm sure when generating word embeddings for Reuters, stop words were kept.

Thank you so much for your clear explanations! Have a nice day!