S3 - Connection Reset By Peer - Over Https
forgotpw1 opened this issue · comments
This is a fantastic tool and exactly what I am looking for.
In my testing, I hit a Connection Reset By Peer when my input urls are for resources on s3 and I request them over https.
Here is the error reported in the node console
Connection reset by peer
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/timeout.rb:62:in `timeout'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/timeout.rb:93:in `timeout'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2009:in `read_new'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:1050:in `__request__'/Users/user/.gem/ruby/1.8/gems/rest-client-1.6.1/lib/restclient/net_http_ext.rb:17:in `request'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:948:in `request_get'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:380:in `get_response'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:543:in `start'/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:379:in `get_response'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/action.rb:49:in `download'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/action.rb:48:in `open'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/action.rb:48:in `download'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/action.rb:123:in `download_input'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/action.rb:120:in `chdir'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/action.rb:120:in `download_input'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/action.rb:35:in `initialize'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/worker.rb:77:in `new'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/worker.rb:77:in `run_work_unit'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/worker.rb:102:in `run'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/node.rb:60:in `POST /work'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/node.rb:60:in `fork'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/node.rb:60:in `POST /work'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:809:in `call'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:809:in `route'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:488:in `instance_eval'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:488:in `route_eval'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:477:in `route!'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:474:in `catch'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:474:in `route!'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:453:in `each'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:453:in `route!'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:569:in `dispatch!'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:388:in `call!'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:536:in `instance_eval'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:536:in `invoke'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:536:in `catch'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:536:in `invoke'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:388:in `call!'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:377:in `call'/Users/user/.gem/ruby/1.8/gems/thin-1.2.11/lib/thin/connection.rb:84:in `pre_process'/Users/user/.gem/ruby/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `catch'/Users/user/.gem/ruby/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `pre_process'/Users/user/.gem/ruby/1.8/gems/thin-1.2.11/lib/thin/connection.rb:57:in `process'/Users/user/.gem/ruby/1.8/gems/thin-1.2.11/lib/thin/connection.rb:42:in `receive_data'/Users/user/.gem/ruby/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'/Users/user/.gem/ruby/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'/Users/user/.gem/ruby/1.8/gems/thin-1.2.11/lib/thin/backends/base.rb:61:in `start'/Users/user/.gem/ruby/1.8/gems/thin-1.2.11/lib/thin/server.rb:159:in `start'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/node.rb:93:in `start'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/node.rb:93:in `initialize'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/node.rb:93:in `new'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/node.rb:93:in `start'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/lib/cloud_crowd/node.rb:79:in `initialize'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:923:in `new'/Users/user/.gem/ruby/1.8/gems/sinatra-0.9.6/lib/sinatra/base.rb:923:in `new'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/bin/../lib/cloud_crowd/command_line.rb:118:in `start_node'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/bin/../lib/cloud_crowd/command_line.rb:107:in `run_node'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/bin/../lib/cloud_crowd/command_line.rb:43:in `initialize'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/bin/crowd:5:in `new'/Users/user/.gem/ruby/1.8/gems/cloud-crowd-0.6.2/bin/crowd:5/Users/user/.gem/ruby/1.8/bin/crowd:19:in `load'/Users/user/.gem/ruby/1.8/bin/crowd:19
There is no error when the links are over http. Are there a few caveats for using https? Should the original post be over https as well? Will there be some need to run this on port 443 for ssl?
Thank you ahead of time for your help on this.
Well...you can pretty much close this one.
def download(url, path)
if url.match(FILE_URL)
FileUtils.cp(url.sub(FILE_URL, ''), path)
else
uri = URI(url)
Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
request = Net::HTTP::Get.new uri.request_uri
http.request request do |response|
File.open(path, 'w+') do |file|
response.read_body do |chunk|
file.write chunk
end
end
end
end
end
path
end
This let's you use https. FYI, you need to get on ruby 1.9.2 so you can do this.