[FlightRPC] Flight fallback mechanism.

[vnkesarwani]

Hi Ryan,
This may be out of scope question related to this git repo.

I am planning to implement following Flight client + server scenario to handle huge data shared between spark and other system
I couldn't find any api or design to handle fall back mechanism in case data is not fitting in memory.

Cases:
1- Once memory buffer is nearing full, data should spill over to disk.
2- Spilling over disk or memory mapped file.
3- Should it be .arrow file or feather format on Disk.
4- Should it be compressed? any design suggestion?
Do you have any suggestion or reference to achieve this. As this may be application level module example: spark

-Vinay

[Ryan Murray]

Hey Vinay,

apologies for the delay, been a busy start to the year. I would echo Wes' comment from the arrow mailing list on this topic.