eklem / browsercrawler

Crawling content from a site within the browser. A basis for i.e. a search solution for static sites.

Home Page:https://eklem.github.io/browsercrawler/doc/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Do all crawls (fetch) and cheerio-stuff before adding to search index

eklem opened this issue · comments

https://daveceddia.com/waiting-for-promises-in-a-loop/

Use the second example, then it's easier to add a play-nice timer between each fetch() request:

var chain = $q.when();
for(var i = 0; i < 5; i++) {
	chain = chain.then(function() {
		return $http.get('/data' + i);
	});
}

After the foor-loop a search-index adder can be called.

Actually, no. Do one and one, and let search-index-housekeeper keep track of whats indexed and not. We don't know when the user will click to a new page.

Check promise.all