hisabimbola / slack-history-export

A NPM module that allows slack users export their history

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Handle large history export

hisabimbola opened this issue · comments

If the history being exported is very large, memory consumption might be very large because we storing all history in memory and then writing it to a file after all histories have been fetched. This can be very costly and if an error occurs when fetching messages, all messages will be lost.

Also, another risk for this is when #5 is implemented, memory usage can be very high.

A simple solution is to write the history to file in batch, once gotten from slack API and append subsequent messages to end of the file.

Another elegant solution is to open a NodeJS writable stream and close when done. Writing to file will be faster.