How to train large quantities of data

Question

How to train large quantities of data

qiansi opened this issue 7 years ago · comments

hi，I would like to train more named entities, via more data, about 10 types of entities, 50M of data, but will crash whether through GPU or CPU memory. Is there any solution?

Guillaume Lample · Answer 1 · Mon Nov 27 2017 18:57:29 GMT+0800 (China Standard Time)

Hi,

50M of data is small, this should not crash. What is the problem you get?

qiansi · Answer 2 · Mon Nov 27 2017 19:21:25 GMT+0800 (China Standard Time)

@glample Thank you for your reply. You are right. I was having a problem with the sentence. I treated all the input as an entire sentence. Now that I have solved the problem, I have been training normally and I will report the results of the training later.