Wrongly converting columns to nullable may cause momory problem
shuai-xu opened this issue · comments
Describe what's wrong
When I run a query with gouping set, It failed with exception java.lang.UnsupportedOperationException: Not enough spark off-heap execution memory. Acquired: 17592186700480, granted: 2143226266. The stack shows as following:
It took a long time for me to find that this is due to wrongly convert columns to nullable in ExpandTransform. This action may cause that data in Block is nullable, but the header in the next transform is not nullable, when seriaze the data and then deseriaze it, it will throw exception.