Data marshaling problem
johanmazel opened this issue · comments
I am currently getting some errors "Fatal error: exception Failure("output_value: object too big")".
I am not absolutely sure about this, but I strongly suspect the Marshal.to_string call in the function marshal at line 114 of the parmap.ml file to be the cause of this.
Do you have any idea how to fix this ?
Can bin_prot/biniou help in this case ? If yes, it would be nice to be able to use these alternative marshaling methods.
I personally have no idea, not even a workaround.
Are you really running parmap on an array of more than 4 gigabytes? That's huge! :-)
But it's true that with modern machines this can happen. There is no way I can see of bypassing this limitation currently (biniou will not help here), so the (unsatisfactory, but easy) workaround is to just split your array/list in several chunks, run parmap on each of them, and then join the results.
The solution proposed by Roberto is good but reduces the parallel section of the program.
Another way is to design your program so that you don't serialize
back to the parent process (if that's doable): use a pariter instead of a fold or map.