Giters

jamescourtney / FlatSharp

Fast, idiomatic C# implementation of Flatbuffers

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Reducing compiler generated file

duckdoom4 opened this issue · comments

commented

Hi there! We've been using the flatbuffer compiler for our project pkNX (https://github.com/kwsch/pkNX).
But we've been running into an issue due to the compiler generating insanely large files.

Each pkNX.Structures.FlatBuffers.X folder contains a large set of .fbs files that get compiled into a single .cs unit (one for each project). The file for the Arceus project, is currently 606353 lines long. The size of this file slows down everything a lot.

Any suggestions on how we can reduce the generated file size?
If not, could we get some flags or code patches to reduce the size?

It would probably also help if the generated file would try to merge some 'branches' and reduce overall characters.

eg. this occurs often:

#pragma warning restore CS8618
#pragma warning disable CS8618

These could be simply removed. Lots of code use fully qualified types, but this is often not necessary

commented

Thanks for reaching out! I am aware that the files can get pretty large; FlatSharp's test code is also several hundred thousand lines long after generation.

I do want to clarify your concern a bit. You mention you are concerned about the size, but I want to ensure that I understand the concern correctly. Do you mean:

  • The number of bytes in the file?
  • Number of lines?
  • The number of tokens (ie, using fully-qualified namespaces when a simple type name would suffice)
  • The fact that this is is one big file instead of multiple smaller ones?

Thanks!

commented

I'd also like to understand what about this is unacceptable in your toolchain. In my experience the FlatSharp tests aren't fast, but nor are they particularly slow to compile / execute.

commented

Well, I can't say for sure what the exact cause of the issue is. What we are currently facing is extremely slow intelli-sense features.

In short, things like go to definition, intelli-sense loading, suggestions (dot auto complete) and the likes. I'm also using resharper, which slows down even more. We know the issue is with this generated file because it only occurred after we swapped to the compiler instead of manually building the classes with attributes.

I personally don't really care about build times. I'm used to c++ builds times, so yeah..

Given that, I can only assume it's the number of tokens that the intellisense parser struggles with. Maybe memory ?

commented

Yeah, I get that. I've also had issues with intellisense in the past with FlatSharp tests. I can't imagine Resharper helps much, as you said. I don't personally use Resharper, but when I have in the past things did not get faster :)

I've thought of a few things off the top of my head:

  • Pass a SerializerTypes flag to the FlatSharp compiler. If you only wanted Lazy semantics, this could cut the size of the file by 75%, which is an easy win. Of course, you'd get NotImplementedExceptions for .ParseProgressive and whatnot.
  • Have the FlatSharp compiler emit a DLL instead of source code. I don't know for sure that that this would make things faster, but it might save intellisense from having to trawl through internal and private members.

I'm more interested in option #2 right now, at least as a prototype.

commented

Update: I've played with prototyping option 2, and while it seems possible to do this automatically, I'm not convinced it's wise to do so. I've prototyped an alternative structure that may make things faster that I'd like to ask you to try out if possible. The basic idea is to put all of the FBS files in their own project, but to reference that project with an assembly reference instead of a project reference, and use the MSBuild task in a target to build the project. I've done a sample of that with the FlatSharp tests here:

https://github.com/jamescourtney/FlatSharp/tree/indirect_reference_experiment/src/test

I'm curious if this helps your use case. I realize it's not an ideal solution, and represents some extra steps that shouldn't be required. However, it seems like your schema is particularly large.

commented

Thanks for the suggestions. I'll give them a go and update you on the results. How does the SerializerTypes option work? I couldn't find anything about it in the documentation.

Edit: Also another note: We really only need the greedy serializer, but it seems the code for all other serializers is still generated. Would be a huge save if it only does one. Can we make that happen somehow? Would that also allow the removal of the virtual keyword from the generated code?

We have a very large amount of fbs files and I'd rather not have to edit each of them to add additional attributes

commented

Oh -- that's something I was proposing here. I haven't built it (yet), but am open to doing so. I wanted to first see how the different reference style helped before diving down that hole.

commented

Oh I see! Sorry for misunderstanding. That would be a huge help in our case.

I'm working on implementing your other suggestion now. Might take a while though, don't have too much free time next to work

commented

PR #386 is done-ish. I have a couple more small items I want to pull into version 7.2. The gist is that you'll be able to pass in the deserialization options you want:

<PropertyGroup>
    <!-- Supports Lazy,Progressive,Greedy,GreedyMutable, All, and None (for write-only) -->
    <FlatSharpDeserializers>Lazy,Progressive</FlatSharpDeserializers>
</PropertyGroup>
dotnet FlatSharp.Compiler.dll ... --deserializers All
commented

If you attempt to use a deserializer that was excluded, you'll get a NotImplementedException back from FlatSharp.

commented

Awesome, looking forward to trying it out :)

commented

Other question, I'm trying to separate some fbs files into a different shared project.

So we have Project A, Project B, Shared Project.

Shared Project contains common fbs files that are shared between A and B
Project A contains fbs files that need to include Shared project's fbs files.

Currently this doesn't seem to work. I get errors telling me that The type 'SharedType' in 'path\to\ProjectA\FlatSharp.generated.cs' conflicts with the imported type 'SharedType' in 'SharedProject, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. Using the type defined in 'path\to\ProjectA\FlatSharp.generated.cs'.

Any way to get this working?

commented

Any way to get this working?

This is going to be a little tricky and is worth tracking in its own thread. The short answer is "not right now". FlatSharp currently scoops up everything recursively through your imports. This has the advantage of behaving predictably, since the whole object graph is available. The downside is the issue you are describing. Please raise a new issue with some details about your use case and what you want to get out of it, and I'll be happy to work with you on it.

commented

Will be fixed in 7.2.0

commented

When can we expect 7.2.0 (on nuget)?