spatie / crawler

An easy to use, powerful crawler implemented in PHP. Can execute Javascript.

Home Page:https://freek.dev/308-building-a-crawler-in-php

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Redirect URLs including info similar to cURL

RobertEcker opened this issue · comments

Hi (it's a question not issue ;-) ),

is there a way, starting from a given URL, ...

... to see all redirected URLs (similar to cURL)?
... with info such as HTTP code, total time etc. (similar to cURL, see https://www.php.net/manual/en/function.curl-getinfo.php)?

hint: javascript is needed because some pages work only with it

$url = 'http://example.com';
Crawler::create()
->setCrawlObserver(new PageCrawlObserver)
->setUserAgent('My browser')
->setConcurrency(1)
->setMaximumCrawlCount(99)
->setMaximumDepth(99)
->executeJavaScript()
->startCrawling($url);

Thank you,

Robert