Codestripping with IL2CPP

Question

Codestripping with IL2CPP

manoharprabhu opened this issue a year ago · comments

Hi,
I'm evaluating FlatSharp 7.1.1 for my project and I created a new Unity project in Unity 2020.3.
I have added the following DLLs to the Assets/Plugins folder:

FlatSharp.Runtime.dll
Google.FlatBuffers.dll
System.Buffers.dll
System.Memory.dll
System.Runtime.CompilerServices.Unsafe.dll

I generated the .cs file for the FBS schema using FlatSharp.Compiler.

The project compiles and runs well on the unity editor, but when I build it for Android and run it on the device, I get a MethodAccessException when I try to serialize the data.

MethodAccessException: Attempt to access method 'FlatSharp.Internal.IGeneratedSerializer<System.Object>.Write' on type 'FlatSharp.Compiler.Generated.ND6E0DEEE460176C11A9DFC2F61E71896F92D7DFEFFB94D52CCCA66C01E5E0635.Serializer' failed.
04-02 13:22:30.590 27236 27274 E Unity   :   at FlatSharp.SpanWriter.InvokeWrite[TItemType] (FlatSharp.Internal.IGeneratedSerializer`1[T] serializer, System.Span`1[T] destination, TItemType item, FlatSharp.Internal.SerializationContext context) [0x00000] in <00000000000000000000000000000000>:0
04-02 13:22:30.590 27236 27274 E Unity   :   at FlatSharp.GeneratedSerializerWrapper`1[T].Write[TSpanWriter] (TSpanWriter writer, System.Span`1[T] destination, T item) [0x00000] in <00000000000000000000000000000000>:0
04-02 13:22:30.590 27236 27274 E Unity   :   at FlatSharp.ISerializerExtensions.Write[T] (FlatSharp.ISerializer`1[T] serializer, System.Span`1[T] buffer, T item) [0x00000] in <00000000000000000000000000000000>:0
04-02 13:22:30.590 27236 27274 E Unity   :   at FlatSharp.ISerializerExtensions.Write[T] (FlatSharp.ISerializer`1[T] serializer, System.Byte[] buffer, T item) [0x00000] in <00000000000000000000000000000000>:0

I suspected IL2CPP stripping out some of the code, so I changed the backend from IL2CPP to mono.
After doing this change, the serialization and deserialization completed successfully without any problems.

FBS file

namespace PayloadNS;
attribute "fs_serializer";

table Payload (fs_serializer) {
    Ids:[int];
    Names:[string];
    points:[float];
}

Serialization:

        int bytesToWrite = Payload.Serializer.GetMaxSize(payload);
        byte[] bytes = new byte[bytesToWrite];
        int bytesWritten = Payload.Serializer.Write(bytes, payload);

A couple of questions here:

Is my project setup correct? Am I missing anything critical here?
Do I need to do any additional steps if I have to use IL2CPP?

James Courtney · Answer 1 · Mon Apr 03 2023 04:34:41 GMT+0800 (China Standard Time)

Is my project setup correct? Am I missing anything critical here?

I think so. Full disclosure, I'm not a Unity developer and have never worked in game dev, so I'm unable to provide robust Unity guidance. @joncham could probably give you better direction than I.

You shouldn't need the reference to Google.FlatBuffers.

I see this from the output: Attempt to access method 'FlatSharp.Internal.IGeneratedSerializer<System.Object>.Write' on type 'FlatSharp.Compiler.Generated.ND6E0DEEE460176C11A9DFC2F61E71896F92D7DFEFFB94D52CCCA66C01E5E0635.Serializer' failed.

FlatSharp's IGeneratedSerializer<T> implementations do not work for object. I see from the stack trace that you are using the correct generic methods above that, though. I know that static analysis tools sometimes can't figure out generics correctly. FlatSharp does emit some AotHelpers to help grease the wheels on this, but it seems like it's not working in your case. You might try adding some to that to see if you can convince the AOT analyzer to not strip the code. Here's what FlatSharp emits for your schema:

        // Method generated to help AOT compilers make good decisions about generics.
        [System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverage]
        public void __AotHelper()
        {
            checked
            {
                this.Write<ISpanWriter>(default!, new byte[10], default!, default!);
                this.Write<SpanWriter>(default!, new byte[10], default!, default!);

                this.ParseLazy<IInputBuffer>(default!, default);
                this.ParseLazy<MemoryInputBuffer>(default!, default);
                this.ParseLazy<ReadOnlyMemoryInputBuffer>(default!, default);
                this.ParseLazy<ArrayInputBuffer>(default!, default);
                this.ParseLazy<ArraySegmentInputBuffer>(default!, default);

                this.ParseProgressive<IInputBuffer>(default!, default);
                this.ParseProgressive<MemoryInputBuffer>(default!, default);
                this.ParseProgressive<ReadOnlyMemoryInputBuffer>(default!, default);
                this.ParseProgressive<ArrayInputBuffer>(default!, default);
                this.ParseProgressive<ArraySegmentInputBuffer>(default!, default);

                this.ParseGreedy<IInputBuffer>(default!, default);
                this.ParseGreedy<MemoryInputBuffer>(default!, default);
                this.ParseGreedy<ReadOnlyMemoryInputBuffer>(default!, default);
                this.ParseGreedy<ArrayInputBuffer>(default!, default);
                this.ParseGreedy<ArraySegmentInputBuffer>(default!, default);

                this.ParseGreedyMutable<IInputBuffer>(default!, default);
                this.ParseGreedyMutable<MemoryInputBuffer>(default!, default);
                this.ParseGreedyMutable<ReadOnlyMemoryInputBuffer>(default!, default);
                this.ParseGreedyMutable<ArrayInputBuffer>(default!, default);
                this.ParseGreedyMutable<ArraySegmentInputBuffer>(default!, default);

                throw new InvalidOperationException("__AotHelper is not intended to be invoked");
            }
        }

Do I need to do any additional steps if I have to use IL2CPP?

Apparently, since it isn't working :) I've used Mono AOT in the past to try and mimic Unity's toolchain, but it's not a perfect solution. I don't have a full setup handy right now, but will try to take a look in the next couple of days.

James Courtney · Answer 2 · Mon Apr 10 2023 10:53:06 GMT+0800 (China Standard Time)

Quick update here: I've confirmed that the FlatSharp samples project runs successfully with a full AOT pass with mono per the documentation here: https://www.mono-project.com/docs/advanced/aot/

My process was roughly:

Change FlatSharp samples to target net472.
Resolve all build errors (disabled gRPC sample)
Copied the binaries to a Linux system and ran with Mono AOT=full
Confirmed samples ran with expected outputs

Unfortunately, this is the best I know to do for Native Code approaches.

Manohar Prabhu · Answer 3 · Mon Apr 10 2023 11:53:22 GMT+0800 (China Standard Time)

Hey James, thanks for checking it out.👍

James Courtney · Answer 4 · Mon Apr 10 2023 13:03:54 GMT+0800 (China Standard Time)

Hey James, thanks for checking it out.👍

Please let me know how you'd like to proceed from here. I'd like to get FlatSharp to emit code that works for your case, so I don't want to close this issue.

Is there anything else I can help with? This seems like a case of the codestripping tool simply removing more than it probably should, so you might be able to give it hints with explicit generic usages like I had above.

Manohar Prabhu · Answer 5 · Mon Apr 10 2023 16:36:13 GMT+0800 (China Standard Time)

I'm currently experimenting with link.xml and adding [Preserve] attributes to the generated code in postprocessor to prevent these methods from getting stripped.

I'll let you know if I find any solution for this.

James Courtney · Answer 6 · Wed Apr 12 2023 04:18:24 GMT+0800 (China Standard Time)

If you figure it out, let me know. I'll be happy to implement it into FlatSharp proper (even if I have to annotate every method with [Preserve]), thought that's likely not ideal given the size of the generated code.

James Courtney · Answer 7 · Fri Jul 21 2023 13:07:10 GMT+0800 (China Standard Time)

Stale -- please reopen if there is any more I can do to help.