rm / word2vec

Automatically exported from code.google.com/p/word2vec

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

The definition of sentences?

GoogleCodeExporter opened this issue · comments

Hi, I am new to word2vec. I am preparing corpus in sentences using wikipedia 
dump. However the dump is pre-splitted in paragraphs which seems need to 
further be processed into sentences. 

My question is 

is it possible to directly train paragraphs instead of sentences? Or it is a 
must that word2vec (the SkipGram model) has to work with sentences. 

Since the algorithm trains the data by a context window, I didn't see much 
difference by add the extra window across sentences within the same paragraph. 

Original issue reported on code.google.com by yel...@gmail.com on 24 Feb 2015 at 9:35

You can train with paragraphs instead of sentences, it should be OK.

Original comment by tmiko...@gmail.com on 25 Feb 2015 at 8:24