Ways of Increment the velocity of the crawling?
iBocchese opened this issue · comments
Has anyone tried to implement some modifications to the repository to crawl faster?
For example, I crawled Rihanna' s profile, but it took days, precisely 5.
Also, when it requires too much time download info, the crawl generates automatically an error of download and starts downloading the next element...
it depends on your use case, everything could be configured:
Don't crawl posts or just X posts
Don't crawl like
Don't crawl follower
Don't crawl comments
I added a new function that's used by default that captures number of likes, number of views, number of comments, caption, location, and a few other pieces of data which makes crawling significantly faster.
It doesn't capture comments, likers, commenters, and other pieces of data you'd need by visiting each post & scrolling through the DOM.