Cannot crawl on specific site
gbarnis opened this issue · comments
gbarnis commented
I've been using ABOT on multiple sites successfully, but for some reason it doesn't find any links to crawl on when dealing with:
https://www.accessdata.fda.gov/scripts/cder/ob/search_patent.cfm?listed=new
Is it related to the TLS 1.2 issue?
Thanks,
Guy
gbarnis commented
After further investigation, it looks like the reason the crawler does not proceed further is due to the use of relative addresses
results_product.cfm?Appl_Type=N&Appl_No=022156
instead of
https://www.accessdata.fda.gov/scripts/cder/ob/results_product.cfm?Appl_Type=N&Appl_No=022156
Suggestions for a solution are welcome.