Out of Memory when converting sparse matrix to dense matrix

[Jinan Zhou]

Hello @Garrafao! Thank you for your work!

When I run run_tr_ppmi.sh with a medium size corpus (30M after compression), I encounter the following error

Traceback (most recent call last):
File "space_creation/tr2bin.py", line 71, in
main()
File "space_creation/tr2bin.py", line 56, in main
new_space = Space(DenseMatrix(new_matrix), list(targets), id2column)
File "./modules/composes/matrix/dense_matrix.py", line 35, in init
self.mat = data.todense()
File "/home/jinan/.conda/envs/TR/lib/python2.7/site-packages/scipy/sparse/base.py", line 849, in todense
return np.asmatrix(self.toarray(order=order, out=out))
File "/home/jinan/.conda/envs/TR/lib/python2.7/site-packages/scipy/sparse/compressed.py", line 962, in toarray
out = self._process_toarray_args(order, out)
File "/home/jinan/.conda/envs/TR/lib/python2.7/site-packages/scipy/sparse/base.py", line 1187, in _process_toarray_args
return np.zeros(self.shape, dtype=self.dtype, order=order)
MemoryError

According to my observation, it happens when trying to convert a scipy sparse matrix to a dense numpy matrix. Anyone has any suggestions to deal with the error?

[Garrafao]

Hi Jinan, thanks for pointing this out. There was an unnecessary cast to dense matrix, which I now removed. Could you please check whether the script works now with your corpus?

[Jinan Zhou]

Yes, the problem is fixed. Thank you!