.open fails where .download works with HTTP Basic Authentication

[Brandon Zylstra]

I'm not 100% sure this is a bug, but it's certainly unexpected (to me).

The following code works fine (without Down):

    @config   = Anyway::Config.for(:campus_access_manager)
    data      = open supervisor.access_letter, http_basic_authentication: [@config['username'],@config['password']]
    send_data data.read, :type => data.content_type, :disposition => 'inline'

but I'd rather use Down, and I'd rather not save the file to disk. Switching to using Down seems fine (at least for the first step of fetching the file):

    data    = Down.download supervisor.access_letter, http_basic_authentication: [@config['username'], @config['password']]

but if I try to avoid downloading the file and passing chunks while it is being downloaded, to avoid saving it to disk, it fails with Down::ClientError - 401 Unauthorized.

    data    = Down.open supervisor.access_letter, http_basic_authentication: [@config['username'], @config['password']]
    data.each_chunk { |chunk| chunk }
    data.close

Why would HTTP Basic Auth work with download and not with open?

Is support for HTTP Basic just not implemented for open? Is this something that may be added in the near future?

[Janko Marohnić]

Down::NetHttp#open currently only supports HTTP basic authentication via the URL, i.e. https://username:[e-mail]/some/path. The reason Down::NetHttp#download supports the :http_basic_authentication option is because open-uri supports it, which #download calls internally (but not #open).

Alternatively, you can use the Down::Http backend instead, which is backed by http.rb, and where #download and #open have the same interface:

data = Down::Http.open(supervisor.access_letter) do |client|
  client.basic_auth(user: @config['username'], pass: @config['password'])
end

Note that, in order to avoid writing to disk, you should pass rewindable: false to the #open method.

[Brandon Zylstra]

@janko Awesome—thank you so much for the example code and the excellent software!

I tried using the code you provided, and I'm getting HTTP::StateError - body has already been consumed. (I had actually gotten this same error previously, and I thought I must be misunderstanding something terribly, but since it is happening with this code, I'm guessing maybe only a small tweak is needed.)

I'm using essentially the exact code you provided, with the rewindable: false added (although I get the same result without it). Here's my controller action:

  def access_letter
    @config     = Anyway::Config.for(:campus_access_manager)
    remote_file = Down::Http.open(supervisor.access_letter, rewindable: false) do |client|
      client.basic_auth(user: @config['username'], pass: @config['password'])
    end

    send_data remote_file.read, :type => remote_file.content_type, :disposition => 'inline'
    remote_file.close
  end

It's occuring in the stream! method of http 4.4.1, in lib/http/response/body.rb:

      def stream!
        raise StateError, "body has already been consumed" if @streaming == false
        @streaming = true
      end

I tried commenting out the line that raises the error (since I couldn't figure out any way that @streaming would ever not be false, and that did allow me to define remote_file but not get any data from remote_file.read. A binding.pry after defining remote_file allowed me to see a Down::ChunkedIO object and its attributes. But still no dice trying to read it. I just got back an empty string.

It's really unclear to me if this is a bug in Down, or if there's something else my code needs added to it.

(The remote_file.content_type might be an additional problem but it is separate.)

[Janko Marohnić]

I added HTTP basic auth option to NetHttp#open in the latest Down version, I will investigate the http.rb issue.

[Janko Marohnić]

I couldn't reproduce the issue with Down::Http on httpbin.org:

require "down/http"

io = Down::Http.open("https://httpbin.org/basic-auth/john/secret") do |client|
  client.basic_auth(user: "john", pass: "secret")
end

puts io.read

{
  "authenticated": true,
  "user": "john"
}

This can only happen if something called HTTP::Response::Body#to_s before Down::Http called #readpartial. I'm wondering what could have called it in your case 🤔