Parquet is missing rows

Question

Parquet is missing rows

mariussoutier opened this issue 2 years ago · comments

I have a Parquet file that should have 30,000+ rows, but SELECT COUNT(*) FROM {} returns 7000. Another one with more than 40,000 rows returns exactly 8000. Converting the same data to JSON works fine.

Phil Eaton · Answer 1 · Tue Jun 28 2022 18:38:42 GMT+0800 (China Standard Time)

Thanks for the report! Can you share a parquet file that has this issue?

Marius Soutier · Answer 2 · Tue Jun 28 2022 19:33:44 GMT+0800 (China Standard Time)

Unfortunately no, it's business-related. But nothing special, 30 or so columns with mostly UTF8 and two INT32 types.

Marius Soutier · Answer 3 · Tue Jun 28 2022 21:02:57 GMT+0800 (China Standard Time)

One of column does contain very large values, but other than that, normal stuff.

Phil Eaton · Answer 4 · Wed Jun 29 2022 02:30:38 GMT+0800 (China Standard Time)

Sounds like @Sajuno reproduced the issue and is thinking of a fix. Thanks @Sajuno!

Ben Schuddeboom · Answer 5 · Wed Jun 29 2022 05:16:14 GMT+0800 (China Standard Time)

@mariussoutier multiprocessio/datastation#278 should fix it!

Phil Eaton · Answer 6 · Thu Jun 30 2022 01:34:33 GMT+0800 (China Standard Time)

Closed in #82 now available in dsq 0.21.0