Transient NullPointerException in sentence detection

Question

Transient NullPointerException in sentence detection

jbnunn opened this issue 11 years ago · comments

When trying sentenceDetector.sentDetect(someText), I sometimes get:

NullPointerException: null (GISModel.java:127)
[error] chalk.learn.model.IndexHashTable.get(IndexHashTable.java:118)
[error] chalk.learn.maxent.GISModel.eval(GISModel.java:127)
[error] chalk.learn.maxent.GISModel.eval(GISModel.java:107)
[error] chalk.learn.maxent.GISModel.eval(GISModel.java:99)
[error] chalk.tools.sentdetect.SentenceDetectorME.sentPosDetect(SentenceDetectorME.java:185)
[error] chalk.tools.sentdetect.SentenceDetectorME.sentDetect(SentenceDetectorME.java:134)
...

If I rerun the test, this error usually immediately goes away. Anyone else experienced this?

Jason Baldridge commented 11 years ago

Great!

Jason Baldridge · Answer 1 · Tue Jan 01 2013 02:52:41 GMT+0800 (China Standard Time)

I'm not sure what is going on -- everything works fine for me. Can you do the following?

  $ cd /tmp
  $ wget http://opennlp.sourceforge.net/models-1.5/en-sent.bin
  $ cd $CHALK_DIR
  $ ./build
  > console
  scala> import java.io.FileInputStream
  import java.io.FileInputStream

  scala> import chalk.tools.sentdetect._
  import chalk.tools.sentdetect._

  scala> val sdetector = new SentenceDetectorME(new SentenceModel(new FileInputStream("/tmp/en-sent.bin")))
  sdetector: chalk.tools.sentdetect.SentenceDetectorME = chalk.tools.sentdetect.SentenceDetectorME@74dd590f

  scala> val sentences = sdetector.sentDetect("Here is a sentence. Here is another with Mr. Brown in it. Hurrah.")
  sentences: Array[java.lang.String] = Array(Here is a sentence., Here is another with Mr. Brown in it., Hurrah.)

  scala> sentences.foreach(println)
  Here is a sentence.
  Here is another with Mr. Brown in it.
  Hurrah.

Jeff Nunn · Answer 2 · Tue Jan 01 2013 03:06:43 GMT+0800 (China Standard Time)

Thanks Jason--works like a champ, each time.

So, my guess is I didn't build right the first time--but now after a build, I can't reproduce the error, which is good--sorry for the false alarm :)

Ben Burton · Answer 3 · Thu Mar 27 2014 21:58:43 GMT+0800 (China Standard Time)

I have found myself running into this error occasionally. I will see if I can get a consistent set of steps to reproduce.

fatefree · Answer 4 · Mon Oct 06 2014 22:45:38 GMT+0800 (China Standard Time)

This happens when you use SentenceDetectorME in a threaded environment. It is not a thread-safe class.

Deleted user · Answer 5 · Wed Jan 14 2015 23:42:11 GMT+0800 (China Standard Time)

@fatefree Can only confirm this. I've been running in this issue occasionally and it is a threading problem. Each thread needs its own SentenceDetector!