FlatSharp and ApacheArrow
kimmolinna opened this issue · comments
Is there any plans to integrate FlatSharp with a present ApacheArrow/.Net interface?
Happy to help here, but you'll need to educate me a bit first! What is the relationship between FlatBuffers and Apache Arrow? Does Arrow use FlatBuffers as its internal format?
Yes, it does. You will find more infromation at page (https://github.com/apache/arrow/blob/master/csharp/README.md).
"Updating FlatBuffers code
See Flatbuffers guide for how to get the flatc executable.
Run flatc --csharp on each .fbs file in the format folder. And replace the checked in .cs files under FlatBuf with the generated files.
Update the non-generated FlatBuffers .cs files with the files from the google/flatbuffers repo."
Thanks for the pointer. I've looked through that repo a little bit, and it seems like they take a hard dependency on flatc, the Google Flatbuffer compiler. FlatSharp is designed to be more idiomatic than the Google version, so it's not just a drop-in replacement. FlatSharp could presumably be used in lieu of flatc, but you'd need to make a code change in the Arrow repository. FlatSharp also requires some additional schema annotations to specify C# specific behavior. I suppose one avenue would be to spin up a new "ArrowSharp" branch that just swapped in FlatSharp, but I'm not sure what I can do from this repo.
On a side note, the Arrow schema also causes the FlatSharp compiler to choke somewhere, so I need to look into that separately. It seems to just be a compiler issue as the runtime-generated code works just fine. I'll try to get a fix out for that in the next day or so.
Thank you very much for your effort. ArrowSharp sounds fantastic. :) And if you could use dictionary coded arrays then it would be the fastest if I have understood correctly. The present implementation won't support those.
#155 Should fix the compiler issues. Added an Apache Arrow test.
This is turning out to be pretty nontrivial, as you might expect. I'll play with this a little bit as time permits, but Flatbuffers is integrated very tightly with Arrow.
Just FYI: I'm probably not going to be working on this -- it's just too far out of scope for this branch and I don't want to maintain a fork of Arrow.
I'm working on a few changes to get FlatSharp fully compatible with Apache Arrow (need to supported the required attribute).