Docs: Async example for parsing a large CSV via streams

[aeddie-zapidhire]

Summary

As a user, I would like an example of how to parse a very large CSV file using streams AND doing some async operations as a part of the stream pipeline.

Motivation

It is not easy to find a clear of example of how to do this. Took a lot of Googling.

Draft

Could this example, or similar, be added to the docs:

Using the pipeline API (async)

import { parse } from 'csv-parse'
import fs from 'fs'
import { pipeline } from 'stream/promises'

/**
 * Do something async with a row.
 * @param {*} row A row of the CSV as an object.
 */
async function handleRow(row) {
  // Do something async
}

/**
 * Read the CSV using a stream and return the number of rows handled.
 */
async function readCsv(filePath) {
  let count = 0

  await pipeline(
    fs.createReadStream(filePath),
    parse({
      skip_empty_lines: true,
      columns: true
    }),
    async function* (source) {
      for await (const chunk of source) {
        yield await handleRow(chunk)
        count++
      }
    }
  )

  return count
}

[Worms David]

Is this page not enough: https://csv.js.org/parse/recipes/async_iterator/

[aeddie-zapidhire]

@wdavidw well, there you go. I see the search terms I needed to use to get those pages above the fold. Apologies for the distraction.

[aeddie-zapidhire]

Oh, I see what I did wrong. So, I saw that the iterator code was outside of the stream pipeline, so I obviously skimmed that example too quickly and thought that it was just loading all the records into memory and then iterating over them.

[Dariusz Górak]

I am sorry for commenting on this closed issue but I am facing issues with pipeline + to param of parser, this is minimal repo:

const { pipeline } = require("stream/promises");
const { open, readFile } = require("fs/promises");
const { parse } = require("csv-parse");
const { Readable } = require("stream");

const parser = parse({
    trim              : true,
    columns           : true,
    skip_empty_lines  : true,
    relax_column_count: true,
    from              : 1,
    to                : 1001
});

const fn = async () => {
    const buffer = await readFile("csv.csv");
    await pipeline(
        Readable.from(buffer),
        parser,
        async function* csvRow (source) {
            for await (const chunk of source) {
                yield chunk;
            }
        },
        async function* handleSingleRow (source) {
            for await (const row of source) {
                console.log("ROW:");
                console.log(row);
            }
            yield
        }
    );
    console.log("AFTER PIPELINE");
}

console.log("START");
fn().then(() => console.log("FINISHED")).catch(error => console.error("error", error));

csv.csv is CSV file with 2160 records: (+ header number,text)

1,test

Now executign this piece of code never prints AFTER PIPELINE or FINISHED, neither any error. Process just exists.
If I remove:
to : 1001
line from parser options then it works and finishes with correct logs.

Is this an oversight on my part?

Node: 16.20.0
csv-parse: 5.3.10