Async iterator loads entire file before allowing iteration

[Brian Begy]

Describe the bug

I'm trying to download a large (500MB) file, parsing it, and doing something async like writing to a database or calling a webhook without loading the whole file into memory.

The docs suggest using an async iterator to accomplish this, but when I do that, my memory usage spikes as it loads the entire file before allowing me to iterate.

To Reproduce

Create a parsers, open a file stream and pipe it to the parser. Try to async iterate, measuring memory usage before and after.

Additional context

What is the recommended process for doing something like this?

[Worms David]

You should pipe to the writer, for example using the pipe sample where you replace process.stdout with your own writable stream.

[Brian Begy]

Ok thanks. Now I'm finding that the pipe example's use of transformer() doesn't match the published type definitions.

[Worms David]

I you have another problem, then you should open a new issue. Closing the current issue.