JSON.dump for saving nnet.tmp

Question

JSON.dump for saving nnet.tmp

tofigh- opened this issue 10 years ago · comments

Hi yasjie,
I'm wondering if there is a particular reason that you used json format to dump the nnet dictionary. As the size of the network increases, it because a huge burden since a network with four parallel conv layers may need more than 1.5G space to be saved and you also can imagine how much dump it may get to load this back to the memory.

Deleted user · Answer 1 · Thu Feb 05 2015 05:07:24 GMT+0800 (China Standard Time)

My initial implementation used JSON (for reasons I don't remember now...). Later versions just inherited it. I am aware of this problem, but haven't figured out a better workaround. Any suggestions?

Tofigh Naghibi · Answer 2 · Sat Feb 07 2015 02:12:37 GMT+0800 (China Standard Time)

One solution could be writing each layer in a separate file and reading them layer by layer for initialization. But to be honest, I also don't know the optimal solution. I have to do some test with cPickle first.

Tofigh Naghibi · Answer 3 · Wed Feb 18 2015 00:18:14 GMT+0800 (China Standard Time)

Hi Yasjie,
Besides layer-wise saving which also gives more freedom later when we only want to initialize a subset of layers, replacing json with pickle.dump also seems to help a lot to speed up the loading process.