Reduce size of LINNAEUS automaton
rockt opened this issue · comments
Check whether CustomRunAutomaton can create a transition table!
Update: "@param tableize if true, a transition table is created which makes the run method faster in return of a higher memory usage" -> turn tableize off
Does that change make any significant difference? I think the way LINNAEUS is encoding each dictionary entry may be intrinsically quite memory inefficient; I believe it is essentially creating a state machine that matches n * (entry + delimiter +id)
As the ids are all unique it is not possible for the states that recognise the different entries to be shared so the state machine scales linearly with the number of entries in the input rather than sublinearly.
If you can do without the IDs, have you considered using the underlying libraries function:
http://www.brics.dk/automaton/doc/dk/brics/automaton/BasicAutomata.html#makeStringUnion%28java.lang.CharSequence...%29
(once created you probably would use the Automaton to create a RunAutomaton which is faster/more memory efficient and serialise that)
Due to not capturing the ids the resultant automaton will be substantially smaller. By taking advantage of the fact that the input is only strings (not regexes) the time to create the automaton is also vastly reduced. I assume building the automaton with LINNAEUS took hours?
Many thanks Daniel! You were absolutely right. I am now using brics directly and it reduced the main-memory consumption to 25% compared to LINNAEUS. Also, building this automaton now takes 2 days maximum (compared to 2 weeks). I agree... normalization should be a subsequent step and not included in the automaton.