Response streaming example/docs incomplete
spacebat opened this issue · comments
Having fun with the into: IO.stream() example:
iex(3)> Req.get!("https://www.abc.net.au/news").body |> String.slice(0..39)
"<!DOCTYPE html><html lang=\"en-AU\"><head>"
iex(4)> Req.get!("https://www.abc.net.au/news", into: IO.stream())
** (ArgumentError) argument error
(stdlib 5.0.2) io.erl:99: :io.put_chars(:standard_io, <<31, 139, 8, 0, 0, 0, 0, 0, 0, 3, 236, 189, 235, 114, 219, 198, 210, 46, 252, 255, 187, 10, 188, 122, 171, 114, 168, 37, 80, 56, 31, 156, 56, 235, 147, 229, 248, 144, 37, 31, 98, 41, 182, 147, 85, 171, 88, 3, 96, 64, ...>>)
(req 0.4.8) lib/req/steps.ex:760: anonymous fn/3 in Req.Steps.finch_stream_into_collectable/5
...
If I instead stream into a File.Stream there is no such error but the resulting file is a gzipped file. That explains the error above.
My question is how do we obtain a stream of the uncompressed response body?
Thank you for the report. I believe this is a bug in Req that we don't decompress before streaming a chunk. By default we ask the server to send compressed response (using compressed step which sends the accept-encoding header). As a workaround in the meantime you can turn it off like this:
iex> Req.get!("https://www.abc.net.au/news", compressed: false, into: IO.stream())
# <!DOCTYPE html><html lang="en-AU">...