olivere / ndjson

Reader and Writer for ndjson-compatible data.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Reader error `bufio.Scanner: token too long` when a line is very large (> 64K)

mlaflamm opened this issue · comments

Thank you for this very handy little library. I had to read from a file with lines larger than 64K and I got this error bufio.Scanner: token too long.

I've already have a working solution based on this answer.
https://stackoverflow.com/questions/21124327/how-to-read-a-text-file-line-by-line-in-go-when-some-lines-are-long-enough-to-ca#answer-37455465

A PR is coming soon.