sparklemotion / mechanize

Mechanize is a ruby library that makes automated web interaction easy.

Home Page:https://www.rubydoc.info/gems/mechanize/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Files only partially downloading

activestylus opened this issue · comments

I'm iterating over a list of links on a page. Mechanize clicks each link and then find the download URL on the resulting page in a transact block. My script runs correctly but all my files are truncated at 24K filesize

transact do
  dlpage = click link
  downloading = false
  if dlbutton
    path = dlpage.links_with(:id => /dlbutton/).attribute(:href)
    filename = URI.decode_www_form_component path.split('/').last
    file = get("http://www.domain.com#{path}")
    file.save("/downloads/#{filename}")
  else
    puts "DEAD LINK"
  end
end

Another thought: are there any sort of callbacks which can determine whether a file has completed downloading? Ideally I'd like to set up a download queue so I don't hammer the server with requests. I suppose a simple sleep() might also work but am open to suggestions

Does this still happen? I have a similar script and cannot reproduce. Closing for now, happy to re-open an investigate (sorry for the delay)