sjdirect / abot

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

How to implement the decision maker function

kunal0810 opened this issue · comments

I am working on project where before crawling a page i need to check whether should a crawl this page or not. i have tried using the shouldcrawlpagedecisionmaker and does not get the right way to implement that.

can someone please help to get it correct.
Displaying image.png
image

Apologies, the doc snippet was using a no longer supported method. Switch it to a property instead as the example below. I also updated the readme doc.

crawler.ShouldCrawlPageDecisionMaker = (pageToCrawl, crawlContext) => 
{
	var decision = new CrawlDecision{ Allow = true };
	if(pageToCrawl.Uri.Authority == "google.com")
		return new CrawlDecision{ Allow = false, Reason = "Dont want to crawl google pages" };
	
	return decision;
};