Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/queues.py", line 266, in _feed send(obj) IOError: bad message length

Question

Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/queues.py", line 266, in _feed send(obj) IOError: bad message length

mlukasik opened this issue 7 years ago · comments

Hi!

When I run training on 10M examples (each described by a small subset of 100K features), it breaks with the error:

....
Splitting 2033
Training classifier
Splitting 1201
Training classifier
Splitting 1323
Training classifier
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/queues.py", line 266, in _feed
send(obj)
IOError: bad message length

Do you know what is the reason and how it could be fixed?

I tried smaller datasets (100K, 1M examples) and the training worked for them.

Cheers,
Michal

Andrew Stanton · Answer 1 · Fri Jan 26 2018 00:11:38 GMT+0800 (China Standard Time)

Curious. I wonder if we're running into the max size of pickle for serialization. Does the problem appear if you increase the max leaf size to say 100? Can you also check to see if you received an out of memory error around that time?

…

On Jan 25, 2018 1:41 AM, "Michal Lukasik" ***@***.***> wrote: Hi! When I run training on 10M examples (each described by a small subset of 100K features), it breaks with the error: .... Splitting 2033 Training classifier Splitting 1201 Training classifier Splitting 1323 Training classifier Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/queues.py", line 266, in _feed send(obj) IOError: bad message length Do you know what is the reason and how it could be fixed? I tried smaller datasets (100K, 1M examples) and the training worked for them. Cheers, Michal — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#9>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABdKFajP8MgToRsYjtyZbFhYJp4iVfVZks5tOExegaJpZM4Rskp6> .

Andrew Stanton · Answer 2 · Sun Jan 28 2018 04:27:23 GMT+0800 (China Standard Time)

Any updates?

Michal Lukasik · Answer 3 · Mon Jan 29 2018 04:05:57 GMT+0800 (China Standard Time)

Thanks for the reply! I am rerunning with 100 max_leaf_size to see if it will pass, however I think it might hurt classification accuracy. I didn't see any out of memory error around that time.

Andrew Stanton · Answer 4 · Mon Jan 29 2018 04:16:44 GMT+0800 (China Standard Time)

It certainly possible it will; this is intended to test whether the tree is too large to serialize correctly. How many labels are you predicting?

Michal Lukasik · Answer 5 · Mon Jan 29 2018 04:41:06 GMT+0800 (China Standard Time)

I got 100K labels (and 100K features).

Andrew Stanton · Answer 6 · Sun Feb 04 2018 03:23:28 GMT+0800 (China Standard Time)

Any updates?

Michal Lukasik · Answer 7 · Sun Feb 04 2018 04:00:55 GMT+0800 (China Standard Time)

Thanks for following up. I am trying to run the training with --max_leaf_size 100 and --threads 5, but it seems to be training forever...

Andrew Stanton · Answer 8 · Sun Feb 04 2018 04:09:53 GMT+0800 (China Standard Time)

--threads 5 is going to hurt if you're using the default set of trees, which is 50. You might ramp that down to 5 trees for debugging purposes for the time being.

Michal Lukasik · Answer 9 · Sun Feb 04 2018 04:10:43 GMT+0800 (China Standard Time)

sounds good, i'll do that!

Michal Lukasik · Answer 10 · Sun Feb 04 2018 22:03:10 GMT+0800 (China Standard Time)

When running with 5 threads and 5 trees, I got this error message:

9790000 docs encoded
9800000 docs encoded
Traceback (most recent call last):
File "/usr/local/bin/fxml.py", line 4, in
import('pkg_resources').run_script('fastxml==2.0.0', 'fxml.py')
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 750, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 1534, in run_script
exec(script_code, namespace, namespace)
File "/usr/local/lib/python2.7/dist-packages/fastxml-2.0.0-py2.7-linux-x86_64.egg/EGG-INFO/scripts/fxml.py", line 646, in

File "/usr/local/lib/python2.7/dist-packages/fastxml-2.0.0-py2.7-linux-x86_64.egg/EGG-INFO/scripts/fxml.py", line 453, in train

File "build/bdist.linux-x86_64/egg/fastxml/trainer.py", line 468, in fit
File "build/bdist.linux-x86_64/egg/fastxml/trainer.py", line 410, in _build_roots
File "build/bdist.linux-x86_64/egg/fastxml/proc.py", line 50, in f2
File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
self._popen = Popen(self)
File "/usr/lib/python2.7/multiprocessing/forking.py", line 121, in init
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

Andrew Stanton · Answer 11 · Tue Feb 06 2018 03:53:03 GMT+0800 (China Standard Time)

There we have it. How much memory does the machine have?

You'll want to try increasing the regularization coefficient to increase sparsity of the linear classifiers. You can also use the --subset flag to send only a subset of the data to each tree (ala random forests).

Michal Lukasik · Answer 12 · Thu Feb 15 2018 03:54:22 GMT+0800 (China Standard Time)

My machine has actually quite a lot of memory:
mlukasik@mlukasik:~/workspace/fastxml_py$ cat /proc/meminfo
MemTotal: 65865896 kB

Is it because we try to load all data at once?