Giters

documentcloud / cloud-crowd

Parallel Processing for the Rest of Us

Home Page:https://github.com/documentcloud/cloud-crowd/wiki

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

S3 - Connection Reset By Peer - Over Https

forgotpw1 opened this issue · comments

commented

This is a fantastic tool and exactly what I am looking for.

In my testing, I hit a Connection Reset By Peer when my input urls are for resources on s3 and I request them over https.

Here is the error reported in the node console

Connection reset by peer
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/timeout.rb:62:in `timeout'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/timeout.rb:93:in `timeout'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2009:in `read_new'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:1050:in `__request__'/Users/user/.gem/ruby/1.8/gems/rest-client-1.6.1/lib/restclient/net_http_ext.rb:17:in `request'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:948:in `request_get'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:380:in `get_response'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:543:in `start'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:379:in `get_response'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/action.rb:49:in `download'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/action.rb:48:in `open'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/action.rb:48:in `download'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/action.rb:123:in `download_input'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/action.rb:120:in `chdir'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/action.rb:120:in `download_input'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/action.rb:35:in `initialize'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/worker.rb:77:in `new'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/worker.rb:77:in `run_work_unit'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/worker.rb:102:in `run'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/node.rb:60:in `POST /work'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/node.rb:60:in `fork'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/node.rb:60:in `POST /work'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:809:in `call'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:809:in `route'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:488:in `instance_eval'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:488:in `route_eval'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:477:in `route!'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:474:in `catch'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:474:in `route!'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:453:in `each'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:453:in `route!'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:569:in `dispatch!'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:388:in `call!'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:536:in `instance_eval'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:536:in `invoke'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:536:in `catch'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:536:in `invoke'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:388:in `call!'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:377:in `call'/Users/user/.gem/ruby/1.8/gems/thin-1.2.11/lib/thin/connection.rb:84:in `pre_process'/Users/user/.gem/ruby/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `catch'/Users/user/.gem/ruby/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `pre_process'/Users/user/.gem/ruby/1.8/gems/thin-1.2.11/lib/thin/connection.rb:57:in `process'/Users/user/.gem/ruby/1.8/gems/thin-1.2.11/lib/thin/connection.rb:42:in `receive_data'/Users/user/.gem/ruby/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'/Users/user/.gem/ruby/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'/Users/user/.gem/ruby/1.8/gems/thin-1.2.11/lib/thin/backends/base.rb:61:in `start'/Users/user/.gem/ruby/1.8/gems/thin-1.2.11/lib/thin/server.rb:159:in `start'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/node.rb:93:in `start'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/node.rb:93:in `initialize'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/node.rb:93:in `new'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/node.rb:93:in `start'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/node.rb:79:in `initialize'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:923:in `new'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:923:in `new'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/bin/../lib/cloud_crowd/command_line.rb:118:in `start_node'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/bin/../lib/cloud_crowd/command_line.rb:107:in `run_node'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/bin/../lib/cloud_crowd/command_line.rb:43:in `initialize'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/bin/crowd:5:in `new'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/bin/crowd:5/Users/user/.gem/ruby/1.8/bin/crowd:19:in `load'/Users/user/.gem/ruby/1.8/bin/crowd:19

There is no error when the links are over http. Are there a few caveats for using https? Should the original post be over https as well? Will there be some need to run this on port 443 for ssl?

Thank you ahead of time for your help on this.

commented

Well...you can pretty much close this one.


    def download(url, path)
      if url.match(FILE_URL)
        FileUtils.cp(url.sub(FILE_URL, ''), path)
      else
        uri = URI(url)
        Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
          request = Net::HTTP::Get.new uri.request_uri

          http.request request do |response|
            File.open(path, 'w+') do |file|
                response.read_body do |chunk|
                file.write chunk
              end
            end
          end
        end      
      end
      path
    end

This let's you use https. FYI, you need to get on ruby 1.9.2 so you can do this.