new multiprocessing parameter for Sequence and TrainingConfig

Question

new multiprocessing parameter for Sequence and TrainingConfig

bfreuden opened this issue 4 years ago · comments

There are some reports of multiprocessing causing locking problems (with Keras hanging forever):
keras-team/keras#3181
keras-team/keras#10340
keras-team/keras#9964
We encountered this problem when training sequence labelling without ELMo (where multiprocessing is disabled by Delft). To be honest it might be caused by some gloomy Python process fork, but I'm not able to find out why we're having this problem.
Having the possibility to disable multiprocessing would be great.

I will submit a PR today.

Luca Foppiano commented 4 years ago

👍

Patrice Lopez · Answer 1 · Sun Dec 15 2019 17:53:04 GMT+0800 (China Standard Time)

Thank you @bfreuden, it's nice to have such a parameter in general (on my side, I've not seen this problem without ELMo).

bfreuden · Answer 2 · Sun Dec 15 2019 18:43:55 GMT+0800 (China Standard Time)

You're very welcome. We don't have the problem neither when running Delft in command-line. However the problem arises when running Delft in a Python web framework (gunicorn).

Luca Foppiano · Answer 3 · Mon Dec 23 2019 23:24:45 GMT+0800 (China Standard Time)

Is this solved with PR #72 ?

bfreuden · Answer 4 · Tue Dec 24 2019 17:56:38 GMT+0800 (China Standard Time)

@lfoppiano yes, that's it