[FEATURE] add option for concurrency

[Johannes Siipola]

Is your feature request related to a problem?

It looks like the concurrency is hard-coded to 5 at the moment.

Describe the solution you'd like

I would like to have a command line flag or an ENV variable that can be used to change this.

Describe alternatives you've considered

I could always create my own crawler class but that would be a lot of duplicate code.

Additional context

I was thinking I could put together a PR for this but I don't know what would be the best way to implement this due to the modular structure of the library.

[Elias Häußler]

Hi @joppuyo, thanks for your feature request. That sounds like a very good idea. I will have a look at it in the next days.

[Elias Häußler]

I released a new version 0.8.0 with this feature. You can now pass a set of crawler options to crawlers implementing ConfigurableCrawlerInterface (the default crawlers already implement this interface; take a look at the crawler classes to get an overview about possible crawler options).

This can be done either on CLI:

$ ./vendor/bin/cache-warmup "https://example.com/sitemap.xml" --crawler-options='{"concurrency": 3}'

Or with the PHP API:

$crawler = new \EliasHaeussler\CacheWarmup\Crawler\ConcurrentCrawler();
$crawler->setOptions(['concurrency' => 3]);

$cacheWarmer = new \EliasHaeussler\CacheWarmup\CacheWarmer('https://example.com/sitemap.xml');
$cacheWarmer->run($crawler);

[Johannes Siipola]

Thank you so much for implementing this feature @eliashaeussler 🙂

I just tested the latest version of the application and the concurrency option working perfectly for my needs (my only challenge was to find out how to correctly escape JSON, inside bash, inside YAML so it actually works 😉 )