kpu / kenlm

KenLM: Faster and Smaller Language Model Queries

Home Page:http://kheafield.com/code/kenlm/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Detailed Usage of lm.State

xiabingquan opened this issue · comments

in python/example.py

#Stateful query
state = kenlm.State()
state2 = kenlm.State()
#Use <s> as context.  If you don't want <s>, use model.NullContextWrite(state).
model.BeginSentenceWrite(state)
accum = 0.0
accum += model.BaseScore(state, "a", state2)
accum += model.BaseScore(state2, "sentence", state)

I tried using the same State variable along the whole process. That's to say:

accum = 0.0
accum += model.BaseScore(state, "a", state)
accum += model.BaseScore(state, "sentence", state)

It turns out that this alternative doesn't work. I'm not quite familiar with C++ so I didn't read the source code. Could anybody provide some hints or link to related docs on how we should use State? Any help is appreciated.Thanks!