Huge memory allocations from `predict.sbo_preds`
vgherard opened this issue · comments
Valerio Gherardi commented
The current (C++) implementation of the predict.sbo_preds()
method has two big issues:
- Every call to
predict()
makes a copy of the entire k-gram prediction tables. This is memory expensive and slow ifpredict()
is called in a non-vectorized way (as would happen e.g. in interactive text prediction). - The look-up method in prediction tables is very slow, and causes huge memory allocations/deallocations for large vector input, which slow down a lot model evaluations in
eval_sbo_preds()
. Maybe #8 could partially fix this?
Valerio Gherardi commented