possibility of adding aiofiles
GitHK opened this issue · comments
Was aiofiles considered in the Downloader
If you would value such a feature I will PR it.
If there is a clear benefit to doing so I am not adverse to the idea, although I do have a few concerns. My main worries stem from the fact that it's already pretty hard to trace errors which occur down at the socket / file level at the moment and adding aiofiles where the writes are being executed in a thread pool wouldn't make this any easier. I would certainly want aiofiles to be an optional thing that can be used if available and can be disabled by an environment variable or flag even if it was installed.
On a higher level have you run into a problem where your downloads are being slowed down by the file write? The way this works at the moment is that small chunks are read out of the socket and written to disk, while those sockets refill before the loop goes back to them an pulls more data. We did consider the value of adding aiofiles when we first implemented the chunked download but my intuition at the time was that the sync writes were unlikely to be a bottleneck in the code. Have you experienced a workload where benchmarks show this is an issue, or do you have other reasons why you desire threaded write?
Thanks for opening an issue!
Thanks for the very fast answer. Just what I wanted to hear. My use case matches none of the above.
I would like to integrate parfive's Downloader inside an existing async application which acts as webserver. My concern was with the write operations blocking the loop. I do not care that much if the overall speed degradation with using aiofiles. The main focus is to keep the loop going.
I am also not aware if there is a better technique/library for async file operations.
Integrating parfive into existing async applications is something which hasn't got a lot of attention, but is something I am willing to support.
As I said, I am happy with using aiofiles (I also know of no better solution) but I would like it to be opt-in to not create more complexity for people who don't need the support. Does that sound good?
Thanks I agree with you. Will try to submit a PR and we'll discuss about it.