Giters

vifreefly / kimuraframework

Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

How to parse pages with HTTP errors (403, 404)

n-studio opened this issue · comments

commented

I'm using mechanize.

Some pages with errors (403, 404...) still display valuable information, but the scraper will just retry or skip the page. Is there a way to treat error pages just like 200 pages?

commented

I fixed my issue by configuring Capybara.

Capybara.configure do |config|
  config.raise_server_errors = false
end