No scraping allowed here... where to output that?
maelle opened this issue · comments
I've tested the bow and scrape functions on an URL I knew didn't allow webscraping (I knew it because I had used robotstxt on it two days ago) and I was wondering whether it'd make sense for bow to already output a message or warning when scraping is not allowed, maybe with a verbose argument? I'm wondering about it because I expected bow to tell me "Go away!". 😹
notok <- polite::bow("https://www.biodiversitylibrary.org/pageimage")
#> No encoding supplied: defaulting to UTF-8.
polite::scrape(notok)
#> No scraping allowed here!
#> NULLCreated on 2018-07-28 by the reprex package (v0.2.0).
Great idea, Maelle! Check out the updated version where I warn from bow and nod (provided verbose=TRUE and finally throw an error from scrape.
And thank you for taking package for a spin!