writeDataTable to enable chunks (low memory consumption)
surpavan opened this issue · comments
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
When writing a file as big as 200MB, I need about 32GB ram PC to save the file.
Describe the solution you'd like
To Reduce the Ram consumption, can we enable writing data in chunks? so that data when writen to excel is done in small parts rather than loading the entire think in RAM. Although I dont know how internally xlsx works.
Only difference I see is writing header and appending rows - chunks vs full file at sametime.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Just to reduce ram consumption when saving files.
I've just written out a file with ~ 6.2 million rows (split into 7 worksheets) and 82 columns.
It took 45 minutes to write and used 89 GB RAM. Final file is 2.2 GB.
I think something is really going wrong here.