janko / down

Streaming downloads using Net::HTTP, http.rb or HTTPX

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

ChunkedIO does not implement #gets correctly, and therefore does not parse CSV while streaming.

edlebert opened this issue · comments

Hi,

First of all, thank you for writing this software. I tried fixing this myself but could not get the ChunkedIO CSV test to work. Currently, ruby's csv calls the io#gets with a nil separator and a limit. In ChunkedIO's current implementation, this causes ChunkedIO to read the entire data source, because it always does so when separator is nil.

When using CSV.new(Chunk.open('http://some/large/file.csv')).each {...}, I used print statements to verify that the entire file is downloaded before any parsing occurs.

I created a PR here: #30

I just released version 4.8.1 with fixes described in #30, so hopefully that should resolve the issue.