How to implement the decision maker function
kunal0810 opened this issue · comments
kunal0810 commented
Steven commented
Apologies, the doc snippet was using a no longer supported method. Switch it to a property instead as the example below. I also updated the readme doc.
crawler.ShouldCrawlPageDecisionMaker = (pageToCrawl, crawlContext) =>
{
var decision = new CrawlDecision{ Allow = true };
if(pageToCrawl.Uri.Authority == "google.com")
return new CrawlDecision{ Allow = false, Reason = "Dont want to crawl google pages" };
return decision;
};