spatie / crawler

An easy to use, powerful crawler implemented in PHP. Can execute Javascript.

Home Page:https://freek.dev/308-building-a-crawler-in-php

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

setConcurrency not working if using browsershoot

lenusch opened this issue · comments

Hi, i can't explain so good, but is it possible to speed up the crawl?
We wait hours to finish big Sites. We need browsershoot to run, because we want to crawl .js files.
Can't we set something which is running concurrently instead of one by one.
Good tool but in the End really slow, with our Usecase.
Problem is we need browsershoot as JS is often loaded a little bit later.

BTW: you can set any number you want with setConcurrency
if will only start the browser once. I wont start several Instance. We also see no corresponding code for that.