Giters

vifreefly / kimuraframework

Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Create directories before saving item

Tails opened this issue · comments

commented

It seems currently no mkdir -p is run when items are saved for the first time.

Errno::ENOENT: No such file or directory @ rb_sysopen - ./results/spidername/.categories.json
  /usr/local/bundle/gems/kimurai-1.2.0/lib/kimurai/base/saver.rb:64:in `initialize'
  /usr/local/bundle/gems/kimurai-1.2.0/lib/kimurai/base/saver.rb:64:in `open'
  /usr/local/bundle/gems/kimurai-1.2.0/lib/kimurai/base/saver.rb:64:in `save_to_pretty_json'
  /usr/local/bundle/gems/kimurai-1.2.0/lib/kimurai/base/saver.rb:31:in `block in save'
  /usr/local/bundle/gems/kimurai-1.2.0/lib/kimurai/base/saver.rb:23:in `synchronize'
  /usr/local/bundle/gems/kimurai-1.2.0/lib/kimurai/base/saver.rb:23:in `save'
  /usr/local/bundle/gems/kimurai-1.2.0/lib/kimurai/base.rb:236:in `save_to'
  /app/spiders/application_spider.rb:66:in `create_category'
commented

@Tails Yes, save_to does not check if directory created or not. I think it is user responsibility to create the required directory.