sjdirect / abot

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Cannot crawl on specific site

gbarnis opened this issue · comments

I've been using ABOT on multiple sites successfully, but for some reason it doesn't find any links to crawl on when dealing with:
https://www.accessdata.fda.gov/scripts/cder/ob/search_patent.cfm?listed=new

Is it related to the TLS 1.2 issue?

Thanks,
Guy

After further investigation, it looks like the reason the crawler does not proceed further is due to the use of relative addresses

results_product.cfm?Appl_Type=N&Appl_No=022156
instead of
https://www.accessdata.fda.gov/scripts/cder/ob/results_product.cfm?Appl_Type=N&Appl_No=022156

Suggestions for a solution are welcome.