Investigate support for Unity
jtattermusch opened this issue · comments
quoting @jskeet:
"There's suspicion the current version of C# protobufs won't work with Unity, but we can add another build target for that later when we find a need - we can create a new minor version of the NuGet package for that with no problems.
My experience with the "old" protobuf-csharp-port is that Unity on iOS has various issues, so I think we'll want to investigate them pretty thoroughly before trying to support it anyway - and we may need to conditionally compile out some features. We'll see... but I'd like to avoid having Unity support block a dotnetcore release."
also see #638
+1
+1
+1
Three +1s in the space of a few minutes suggests this is being linked to from somewhere - it would be nice if someone could provide the context. It would be even nicer if someone with Unity support could try this and inform us where there are problems :)
I can only speak for myself, but in my case we are developing a mobile game with Unity and a Java backend. In order to keep our client compatible with iOS, we are using protobuf-net. Unfortunately, protobuf-net has some known bugs and does not support the latest Protobuf versions, so an official solution from Google would be very interesting for us.
@jayallanjethwa: But how did you happen to reach this issue? It seems unlikely that three people would find an issue raised nearly two months ago entirely independently but within the same 5 minute window. I'm only asking so that I can try to get more context that might help resolve the issue. (I'd love to be able to support Unity - but I foresee some tricky issues, based on previous experience of bug reports.)
@jskeet I started to use protobuf v3 in Unity, and I found two problems.
First of all, Unity is using .NET 3.5 and there were some compile errors when I copied to Unity project, so I had to modify some code with .NET 4.5 specific. This made me working protobuf v3 with Unity for simple case. Secondly, when I trying JsonFormatter.Format(IMessage) in iOS, Unity will threw exception of System.Reflection.Emit is not permitted., because iOS doesn't allow JIT.
So far I found these problems, but there may be other problems because I am not using full functions of v3.
@asuuma: Right. Supporting both .NET 3.5 and modern PCLs is painful. I'm hoping that it won't be too long before Unity moves on in terms of runtime - after all, .NET 4.0 was released a long time ago. Could you give the details of the stack trace around Reflection.Emit? I suspect the problem is the use of expression trees, which would be feasible to fix, though annoying.
(I'm going to need to balance the importance of supporting a wide variety of platforms with the additional code maintenance headaches and possible performance degredations of doing so. The more I know before making this judgement call, the better...)
@jskeet Yeah, everyone are hoping to move on newer runtime in near future... They opened roadmap of Unity and it says they are researching about upgrading .NET profile.
Here is the stack trace.
NotSupportedException: /Users/builduser/buildslave/unity/build/Tools/il2cpp/il2cpp/libil2cpp/icalls/mscorlib/System.Reflection.Emit/DynamicMethod.cpp(21) : Unsupported internal call for IL2CPP:DynamicMethod::create_dynamic_method - System.Reflection.Emit is not supported.
at System.Reflection.Emit.DynamicMethod.CreateDelegate (System.Type delegateType, System.Object target) [0x00000] in <filename unknown>:0
at System.Linq.Expressions.Expression`1[TDelegate].Compile () [0x00000] in <filename unknown>:0
at Google.Protobuf.Reflection.FieldAccessorBase..ctor (System.Reflection.PropertyInfo property, Google.Protobuf.Reflection.FieldDescriptor descriptor) [0x00000] in <filename unknown>:0
at Google.Protobuf.Reflection.SingleFieldAccessor..ctor (System.Reflection.PropertyInfo property, Google.Protobuf.Reflection.FieldDescriptor descriptor) [0x00000] in <filename unknown>:0
at Google.Protobuf.Reflection.FieldDescriptor.CreateAccessor (System.String propertyName) [0x00000] in <filename unknown>:0
at Google.Protobuf.Reflection.FieldDescriptor.CrossLink () [0x00000] in <filename unknown>:0
at Google.Protobuf.Reflection.MessageDescriptor.CrossLink () [0x00000] in <filename unknown>:0
at Google.Protobuf.Reflection.FileDescriptor.CrossLink () [0x00000] in <filename unknown>:0
at Google.Protobuf.Reflection.FileDescriptor.BuildFrom (Google.Protobuf.ByteString descriptorData, Google.Protobuf.Reflection.FileDescriptorProto proto, Google.Protobuf.Reflection.FileDescriptor[] dependencies, Boolean allowUnknownDependencies, Google.Protobuf.Reflection.GeneratedCodeInfo generatedCodeInfo) [0x00000] in <filename unknown>:0
at Google.Protobuf.Reflection.FileDescriptor.InternalBuildGeneratedFileFrom (System.Byte[] descriptorData, Google.Protobuf.Reflection.FileDescriptor[] dependencies, Google.Protobuf.Reflection.GeneratedCodeInfo generatedCodeInfo) [0x00000] in <filename unknown>:0
at Test..cctor () [0x00000] in <filename unknown>:0
at Test.get_Descriptor () [0x00000] in <filename unknown>:0
at Google.Protobuf.JsonFormatter.Format (IMessage message) [0x00000] in <filename unknown>:0
Right, thanks - so that is expression tree compilation. It's definitely feasible to write a workaround for that particular one.
@asuuma Could you elaborate on what you did to workaround the .NET version mismatch?
@dreis2211 Oh very sorry, I missed your message.
Here is diff of what I changed, just fixing compile errors.
diff -u -r Google.Protobuf/Compatibility/PropertyInfoExtensions.cs protubuf-unity/Compatibility/PropertyInfoExtensions.cs
--- Google.Protobuf/Compatibility/PropertyInfoExtensions.cs 2015-09-10 20:05:51.000000000 +0900
+++ protubuf-unity/Compatibility/PropertyInfoExtensions.cs 2015-10-06 12:36:43.000000000 +0900
@@ -47,7 +47,7 @@
/// </summary>
internal static MethodInfo GetGetMethod(this PropertyInfo target)
{
- var method = target.GetMethod;
+ var method = target.GetGetMethod();
return method != null && method.IsPublic ? method : null;
}
@@ -57,7 +57,7 @@
/// </summary>
internal static MethodInfo GetSetMethod(this PropertyInfo target)
{
- var method = target.SetMethod;
+ var method = target.GetSetMethod();
return method != null && method.IsPublic ? method : null;
}
}
diff -u -r Google.Protobuf/Compatibility/TypeExtensions.cs protubuf-unity/Compatibility/TypeExtensions.cs
--- Google.Protobuf/Compatibility/TypeExtensions.cs 2015-09-10 20:05:51.000000000 +0900
+++ protubuf-unity/Compatibility/TypeExtensions.cs 2015-10-06 12:36:43.000000000 +0900
@@ -51,7 +51,8 @@
/// </summary>
internal static bool IsValueType(this Type target)
{
- return target.GetTypeInfo().IsValueType;
+// return target.GetTypeInfo().IsValueType;
+ return target.GetType().IsValueType;
}
/// <summary>
@@ -59,7 +60,8 @@
/// </summary>
internal static bool IsAssignableFrom(this Type target, Type c)
{
- return target.GetTypeInfo().IsAssignableFrom(c.GetTypeInfo());
+// return target.GetTypeInfo().IsAssignableFrom(c.GetTypeInfo());
+ return target.GetType().IsAssignableFrom(c.GetType());
}
/// <summary>
@@ -72,9 +74,9 @@
// GetDeclaredProperty only returns properties declared in the given type, so we need to recurse.
while (target != null)
{
- var typeInfo = target.GetTypeInfo();
- var ret = typeInfo.GetDeclaredProperty(name);
- if (ret != null && ((ret.CanRead && ret.GetMethod.IsPublic) || (ret.CanWrite && ret.SetMethod.IsPublic)))
+ var typeInfo = target.GetType();
+ var ret = typeInfo.GetProperty(name);
+ if (ret != null && ((ret.CanRead && ret.GetGetMethod().IsPublic) || (ret.CanWrite && ret.GetSetMethod().IsPublic)))
{
return ret;
}
@@ -99,8 +101,8 @@
// GetDeclaredMethod only returns methods declared in the given type, so we need to recurse.
while (target != null)
{
- var typeInfo = target.GetTypeInfo();
- var ret = typeInfo.GetDeclaredMethod(name);
+ var typeInfo = target.GetType();
+ var ret = typeInfo.GetMethod(name);
if (ret != null && ret.IsPublic)
{
return ret;
diff -u -r Google.Protobuf/JsonFormatter.cs protubuf-unity/JsonFormatter.cs
--- Google.Protobuf/JsonFormatter.cs 2015-09-10 20:05:51.000000000 +0900
+++ protubuf-unity/JsonFormatter.cs 2015-10-23 10:14:41.000000000 +0900
@@ -496,7 +496,7 @@
private void WriteFieldMask(StringBuilder builder, IMessage value)
{
IList paths = (IList) value.Descriptor.Fields[FieldMask.PathsFieldNumber].Accessor.GetValue(value);
- AppendEscapedString(builder, string.Join(",", paths.Cast<string>().Select(ToCamelCase)));
+ AppendEscapedString(builder, string.Join(",", paths.Cast<string>().Select(s => ToCamelCase(s)).ToArray()));
}
/// <summary>
diff -u -r Google.Protobuf/WellKnownTypes/TimeExtensions.cs protubuf-unity/WellKnownTypes/TimeExtensions.cs
--- Google.Protobuf/WellKnownTypes/TimeExtensions.cs 2015-09-10 20:05:51.000000000 +0900
+++ protubuf-unity/WellKnownTypes/TimeExtensions.cs 2015-10-23 10:13:48.000000000 +0900
@@ -34,7 +34,6 @@
using System.Collections.Generic;
using System.Linq;
using System.Text;
-using System.Threading.Tasks;
namespace Google.Protobuf.WellKnownTypes
{
I've applied this patch (without the commented out code) and can confirm this work in unity3d 5.2.2f
This also keeps working when used in a WebGL export. Although i am not using any fancy protobuf features (only string, int, etc and repeated)
Note that with this patch, it won't work with CoreCLR, which is a more important initial target for us. I don't think it's feasible to support both platforms with a single binary.
Agreed, but it would be nice if unity could be supported somehow untill they finally decide to move away from their ancient mono version.
Even a simple readme note for unity users revering to this issue ticket would be enough.
I wouldn't feel comfortable claiming something was "supported" via just a patch with consumers building the code themselves afterwards. This patch is a good starting point for fuller support, but I'd rather get the first version publicly released with just CoreCLR (and therefore normal desktop .NET) support to start with, then revisit.
+1
+1
I've also applied the patch, and have protobuf v3.0.0-beta-2 working in my mod for Kerbal Space Program (running in Unity). I also ran the nunit tests which all passed (although I removed Google.Protobuf.Test.Compatibility)
@jskeet Thanks for your statement. Do you have a rough idea of how the timeline for this could look like?
I don't have any plans at the moment... I'd be very surprised to see it before protobuf 3.0 goes GA. We can look at it again after that.
It's not just a matter of fixing the code itself - it's making sure we can test and maintain it appropriately.
(Any sign of Unity moving off the ancient CLR they use yet? I'd hope that they're looking into CoreCLR now it's nearing release...)
I think it is unlikely that Unity moves off their CLR. They have their own Mono fork and it contains a bug which is already fixed in original Mono repo 5 years ago. I guess merging branches forked at least 5 years ago would be a heavy toll. Plus they have to make their il2cpp work after the update.
But still, I wish that would happen. I'd love to use protobuf and google.apis.v3 in Unity.
You are able to use Protobuf just fine in unity3d though? I'm currently using it for serializing my network packages.
@zeroZshadow: My guess is that you're using protobuf-csharp-port, the old proto2 implementation... which apparently worked on Unity if you were lucky. We repeatedly ran into issues to do with JIT or AOT compilation, suspecting them to be bugs with the very old version of Mono being used.
@jskeet I'm using whatever is here https://github.com/google/protobuf/tree/master/csharp/src/Google.Protobuf together with proto3
@zeroZshadow: So how are you using that? I wouldn't have expected you to be able to install the NuGet package, for example - it targets CLRv4. Which version of Unity3D are you using?
@jskeet I've copied that entire folder to Assets/Plugins, then applied the fix that is posted by asuuma.
I'm using Unity 5.3.2, but earlyer versions also had no issues.
Okay - that makes a bit more sense. You should be aware that this isn't an officially-supported scenario at the moment. I make no guarantees about it - you may well find it works in simply situations, but that some more advanced features break :( Obviously we'd like to support Unity eventually - but the nature of the platform does make it hard to support properly, and we want to get the support for more modern platforms rock-solid first.
I've tried copying the folder to Assets/Plugins and applying the posted fix, but I get numerous other errors with Unity 5.3.2, many around string interpolation (i.e. $"...{...}..."), some around defaulted properties, some others sprinkled in as well. Since the above comments are ~10 days old, it seems unlikely (though possible) that these additional errors were all introduced recently. Is there something else I need to do to make this work? I understand this isn't a supported scenario, but I'm hoping that some of the people here can offer advice.
@dmuir: Yes, you need C# 6 support to build the Google.Protobuf library, but not to build generated code. Basically you'd need to build the library with VS2015, and then add it as a reference within Unity. But as you say, it's not supported right now..
FWIW we're successfully using protobuf-csharp-port in our Unity project. The app is live on the store for Android and iOS. We've added Google.ProtocolBuffersLite.dll and Google.ProtocolBuffersLite.Serialization.dll to Plugins, and compiling all messages to C# with the "lite" option.
@zakyg: If that's working for you and you can keep using proto2, then I suggest you stick with that for the moment. But as I say, I've seen issues related to Unity in protobuf-csharp-port and when I was doing that just in my free time, it was hard to provide the level of support I wanted.
@jskeet yes we're not planning on upgrading for now, the comment was to help anyone who looks for a way to integrate some version of Protocol Buffers into their Unity project. The OP mentioned they assumed they would run into issues with iOS, but the problematic Emit call was not included in the C# Proto classes compiled with the lite option. We're using protobuf-csharp-port for all our client-server communication and for persisting data as well. Runs good on production on both iOS & Android.
Hi -
The simplistic patch above does not fully work (it may be incomplete).
In particular, unit tests (packing) fails.
I have a more extensive patch in which all unit tests pass. All changes are straightforward (if tedious):
- cloned csproj and removed nuspec, etc. set framework to 3.5.
- removed all extension methods (replace with fully qualified static methods)
- removed all GetTypeInfo calls (not supported in 3.5)
- replaced ReadOnlyDictionary with IDictionary (former not supported in 3.5)
Per your comment above, I don't think it makes sense to try and have a single binary for all platforms. But it would be great if this could be rolled into mainline (in some form) to at least have nominal support for Unity. I don't think Unity will upgrade their framework anytime soon.
I can submit a patch or pull request... please let me know how I can help.
Hi,
we made a Unity compatible version of the C# Protocol Buffers runtime library.
It is available here (https://github.com/bitcraftCoLtd/protobuf3-for-unity) and based on the tag v3.0.0-beta-2.
We fixed the compiler errors to support the runtime 3.5, and all the unit test pass.
Once a new beta version or the final version of the library will be out, we plan to update our repository if Unity is not yet support as this time.
@hobein Thanks for your work ! Works fine for me :) !
Hi @cperras, no we do not provide NuGet packages, sorry about that. This is because Unity sadly does not make it easy to use NuGet packages. However, we provide prebuilt versions of the DLL here: https://github.com/bitcraftCoLtd/protobuf3-for-unity/releases
I hope this help.
@hobein , @TanukiSharp , I tried the protobuf3-for-unity and found it would throw exception "System.Reflection.Emit is not supported" when trying to print out a message on iOS.
After some investigation, I found it was caused by Google.Protobuf.Reflection.ReflectionUtil which is using System.Linq.Expressions to generate code at runtime.
eg.
internal static Func<IMessage, object> CreateFuncIMessageObject(MethodInfo method)
{
ParameterExpression parameter = Expression.Parameter(typeof(IMessage), "p");
Expression downcast = Expression.Convert(parameter, method.DeclaringType);
Expression call = Expression.Call(downcast, method);
Expression upcast = Expression.Convert(call, typeof(object));
return Expression.Lambda<Func<IMessage, object>>(upcast, parameter).Compile();
}
I replaced these code like below:
return (IMessage message) =>
{
var returnValue = method.Invoke(message, null) as object;
return returnValue;
};
It seems to resolve the issue on iOS.
Here is my changes:
To be honest, I am not very experienced in C#, so it would be helpful if somebody could help review these changes.
thanks.
@zhangzhibin Just mentioning the reply to the issue you opened on our repo:
bitcraftCoLtd/protobuf3-for-unity#4 (comment)
So basically, it'd be great if this could land in https://github.com/google/protobuf so we are certain that these changes do not impact the rest of the userbase (wondering if the original syntax was a conscious choice)
Unity on iOS is a bit of a complex topic due to the pipeline:
C# -> IL -> cpp
which is why reflection is a bit of a sensitive matter.
From what we gathered so far, it appears that the contributors of this repo do not wish to prioritize Unity-related limitations, so this change will only be accepted here if it makes sense beyond the Unity scope.
If your changes are not accepted here, then we will update the protobuf3-for-unity repo, so please keep us updated.
Note that using reflection-based invocation will be considerably slower than using compiled expression trees. The alternative would be to use Delegate.CreateDelegate, but it's more fiddly than the expression tree version.
@zhangzhibin Yes @jskeet is right, there is a huge performance cost here, and if you are interested in understanding why, the reason is that with the expression tree, the call to the method argument (MethodInfo) gets "natively" inserted in the expression, which will result in a regular ".NET native" call, whereas with a lamda, the method argument is closured, and then called through reflection each time the lambda is executed, which is very slow.
I've run a basic performance test, and got something from 5 times to 20 times slower (could be even worst).
@jskeet As you said, using method.CreateDelegate would require to cache the compiled methods in a dictionary for a real performance gain, but that would just clutter the code and consume more memory, etc... so I agree, this is not a good idea.
@zhangzhibin Since the Google repo does not pretend to support iOS and/or Android (Unity) they are not likely to have an #if IOS in their code to fallback to lambdas, moreover this would mean they would have to add a specific build configuration to make iOS builds, which sucks. And for the same reasons I've enumerated before, we (bitcraft) are not going to accept this change either.
I would advice you to fork our repo (or Google's one) and put this change in yours, so you can always rebase it once Google and/or us update our repos.
Another advice, because you could get in trouble with two different versions of the same protobuf dll (one for iOS with lambdas, and another for all the other platforms with expression trees), you could have only one dll with the platform check done at runtime, using UnityEngine.Application.platform. Note that this would make your version of protobuf dll to be tightly-coupled to Unity, and that's also why Google devs are never going to let this happen.
I hope this helped.
@TanukiSharp: No, no dictionary would be required. These are already being cached, not in a dictionary but in the descriptor. (Compiling the expression tree each time would also be expensive.) So this could be done, but it would be uglier code - I'd rather not do that at the moment.
@jskeet Sorry I'm not in charge of that project at bitcraft, just a kind of .NET advisory for the team, so I don't know that much about the protobuf code base.
So do you mean the 4 methods ReflectionUtil.Create*IMessage*() are guaranteed to be called only once per MethodInfo ?
Because the method argument would need to be compiled to delegate outside of the lambda, and so only the delegate would need to be closured in the lambda, since lambda will be evaluated each time it is called.
Actually we may implement this in our fork anyway, because at some point we will need to support iOS, no matter what.
@TanukiSharp: The usage of ReflectionUtil is (at least currently) in creating descriptors, which are created once per message type basically. I didn't follow your discussion of the delegate being "closured in the lambda" - I was proposing not needing a lambda expression at all within ReflectionUtil, if possible. It's possible that one level of indirection would be required anyway, basically to do the casting... but that would have minimal performance impact.
@jskeet OK I'm interested in your solution to avoid using code emission at runtime.
Actually I think CreateDelegate will not work, because we would have to know the type of delegate at compile time, since produced Delegate can't be cast to Func<X, Y>.
Anyway, if you have a solution to simply and efficiently avoid code emission, that would be much appreciated.
Thanks.
@TanukiSharp: I don't have time to do this now, I'm afraid. But you can absolutely create a delegate of the right type with Delegate.CreateDelegate and cast it. We know the target type using the generic type parameter.
To make sure that the project owners don't underestimate the importance of Unity support, I'd add my 5 cents. According to the Unity website, it has been downloaded by 5.5M users. 280K Unity-based games have been released so far. It's a big market. And what is really important, most of the games do sophisticated client-server communication. Protobuffers solve the serialization problem just perfectly, providing support for protocol backward compatibility, which is vital for mobile and online games, and in general making cross-platform interaction a breeze. Please don't drop Unity support on the floor.
@TanukiSharp @qbrandon @jskeet Thanks for feedback and pointing out the performance impact.
@amlinux I agree with you on this. It would be very helpful if Unity is added to the official support list.
However, as I dig deeper, trying to make grpc working with Unity, it is very sad to find out that how much Unity is drawing back by its unwillingness to upgrade .Net. So, even now the Protobuf supports Unity, there would still be a large gap between Unity and modern .Net world.
So, I understand that the much more update-to-date libraries not support out-of-date Unity when Unity team is more interested in il2cpp than upgrading .Net. I feel it is not right forcing this kind of backward compatibility.
@jskeet Just to make sure we are on the same page, we can't use Delegate.CreateDelegate method since it is not available in .NET Standard 1.0, which you guys are targeting.
Hereafter is why this is causing problems:
(let's consider the method ReflectionUtil.CreateFuncIMessageObject for the moment)
Trial 1:
internal static Func<IMessage, object> CreateFuncIMessageObject(MethodInfo method)
{
return (IMessage message) =>
{
Type type = typeof(Func<object>);
Delegate d = method.CreateDelegate(type, message); // <-- Not working
};
}
Trial 2:
internal static Func<IMessage, object> CreateFuncIMessageObject(MethodInfo method)
{
return (IMessage message) =>
{
Type type = typeof(Func<>);
type = type.MakeGenericType(method.ReturnType);
Delegate d = method.CreateDelegate(type, message); // <-- Working
Func<object> f = (Func<object>)d; // <-- Not working
};
}
Trial 3:
internal static Func<IMessage, object> CreateFuncIMessageObject(MethodInfo method)
{
return (IMessage message) =>
{
Type delegateType = typeof(Func<>);
type = type.MakeGenericType(method.ReturnType);
Delegate d = method.CreateDelegate(type, message);
var f = (Func<?>)d; // <-- We need to know the return type at compile time
};
}
About the conversation "closured in the lambda" that I did not explain clearly enought (sorry about that), you can see in all the trials that the CreateDelegate happens in the lambda, which means that it will be performed each time the lambda is called, which is probably not good. The best would be to create the delegate outside the lambda, and put only the already created delegate inside the lambda, as follow:
internal static Func<IMessage, object> CreateFuncIMessageObject(MethodInfo method)
{
Type type = typeof(Func<,>);
type = type.MakeGenericType(typeof(IMessage), method.ReturnType);
Delegate d = method.CreateDelegate(type); // <-- Not working, member method needs the target
return (IMessage message) =>
{
d(message); // <-- Obviously can't call it that way, this is just for the demo
};
}
So for the moment, since nobody have time to spend on that issue, I guess the best remains what @zhangzhibin proposed. I mean, for a Unity specific build, so we are probably going to implement that in our fork.
At the moment, it is more important for us that it can run on all platforms than having crazy performances.
Anyway, if you (@jskeet) have pointers to solve this issue in a better way for all platforms, that would be much appreciated.
Thanks.
@amlinux @zhangzhibin I do understand the Unity support problems, since I'm working with Unity daily since many years, but I do agree with @jskeet, I believe it is not up to the world to adjust to Unity, but Unity to upgrade to a modern version of .NET, people just can't keep living in the middle-age because of Unity...
@TanukiSharp Yes, can't agree more.
@TanukiSharp: Eek, I hadn't realized that Delegate.CreateDelegate wasn't available in .NET Standard 1.0. That's rather surprising given how long it's been available in regular .NET :(
If it were, the way to avoid creating the delegate in the lambda expression is simply to create an open delegate, so you can pass the message in - see my 2008 blog post for details of how to do that: https://codeblog.jonskeet.uk/2008/08/09/making-reflection-fly-and-exploring-delegates/ (ironically, when creating the previous version of Protocol Buffers). However, as we can't use it, that doesn't help :( We could potentially use plain, slow reflection in .NET Standard 1.0 and Delegate.CreateDelegate (or expression trees) for later versions... but I'd rather not.
Just a little update for those interested to know a bit more about this issue.
I've run some tests and perform some investigations and trials, and I found out that at some point, no matter what, we need to know the real type of the target, at compile time, because it just can't simply be cast to object or IMessage in delegates. Even using @jskeet's MagicMethod trick at some point led me to a typing dead-end.
So we are going to stick with the basic fix we did, based on @zhangzhibin contribution (bitcraftCoLtd/protobuf3-for-unity@985243b)
The new Unity beta supports .NET 4.6, this should solve a lot of the problems.
+1
Does not support IL2CPP of Unity engine yet?
Nope. We have a PR that would work, I believe, but it still needs careful review - and then build and release work too.
+1!
I tested latest Protobuf 3 with Unity 2017.1.2p3 + .Net 4.6 : IL2CPP, it seems working correctly normal serialization/deserialization into MemoryStream as binary. But deserializing from JsonString via JsonParser is failed (app crashes).
@kkc0923 Were you using the stock version or @jskeet's PR #3794? And were you testing on iOS where there is no Reflection.Emit capability?
I'm trying to get the current version up and running on Unity 2017.3.1+.NET3.5+iOS, but am hitting failures because of the Reflection.Emit requirement. Trying to figure out if moving to .NET 4.6 alone will fix things or if I need to try to pick up the changes in PR #3794.
Thanks
Quick followup to my comment. The Unity 2017.3.1+.NET3.5+iOS does work for me out of the box as long as I avoid accessing the Descriptor property on any of the generated message types. (My test code was accessing IMessage.Descriptor.Name to get a name for the message type).
@mdouglass I used a latest Protobuf package in Release section. And I tested only Android devices. Also I tested only simple serialization and deserialization with direct accessing to message type's property (like MyCustomData.myCustomProperty = testValue;)
It seems that official release is not yet implemented for Refelction limitations.
@jtattermusch this is relevant to our ongoing work to get GRPC working in Unity. Once I started introducing more complicated messages to my project (instead of the simple Healthcheck I was running), I ran into this System.Reflection.Emit issue. It looks like my particular errors are related to the google.protobuf.Timestamp well-known type, so I'm going to first try replacing its usage in my project.
I'd also be interested @jskeet 's fork, but need to research how to cross-compile the necessary DLL on OSX, as the default installation instructions obviously yield dylibs.
Given that this issue has been open for almost 3 years now... any chance we can drive this home?
edit After replacing all instances of google.protobuf.Timestamp in my protobuf definitions with a simple double (to encode unix epoch time instead), everything worked using the current NuGet Protobuf package :)
I was able to get grpc up and running on iOS 11 + Unity 2017 + .NET4.6, I documented the process in my blog. Hope this could help for you guys.
I was thinking creating a branch for Unity support, but I was only able to get it up and running on iOS and specifically .NET 4.6, so maybe there should be a better way to support this.
Sounds Great! Awesome work! I'm looking forward to see grpc officially support Unity!
@zaneclaes can you link to the System.Reflection.Emit issue?
Ad the question how to proceed with improving support:
- the best way to help with stuff that has been stuck for long time is to come up with a series of small PRs that incrementally fix the issue. Contributions are welcome.
@jtattermusch er, this is the System.Reflection.Emit issue -- see the stacktraces above. My understanding is that @jskeet 's open PR fixes the issue by avoiding the use of expression trees, but I have been unable to validate this (if someone can provide a compiled DLL of his work as a drop-in replacement for the NuGet package, I'd be happy to do so): #3794
Here's my own stacktrace to add to the mix. Note that I still have this issue if I attempt to stringify any of my messages on the iOS client. For example, the following stacktrace is triggered by $"Message {msg}" where msg is an instance of a BootstrapRes (pb::IMessage):
NotSupportedException: /Users/builduser/buildslave/unity/build/External/il2cpp/il2cpp/libil2cpp/icalls/mscorlib/System.Reflection.Emit/AssemblyBuilder.cpp(20) : Unsupported internal call for IL2CPP:AssemblyBuilder::basic_init - System.Reflection.Emit is not supported.
at System.Reflection.Emit.AssemblyBuilder..ctor (System.Reflection.AssemblyName n, System.String directory, System.Reflection.Emit.AssemblyBuilderAccess access, System.Boolean corlib_internal) [0x00000] in <00000000000000000000000000000000>:0
at System.AppDomain.DefineDynamicAssembly (System.Reflection.AssemblyName name, System.Reflection.Emit.AssemblyBuilderAccess access, System.String dir, System.Security.Policy.Evidence evidence, System.Security.PermissionSet requiredPermissions, System.Security.PermissionSet optionalPermissions, System.Security.PermissionSet refusedPermissions, System.Boolean isSynchronized) [0x00000] in <00000000000000000000000000000000>:0
at System.AppDomain.DefineDynamicAssembly (System.Reflection.AssemblyName name, System.Reflection.Emit.AssemblyBuilderAccess access) [0x00000] in <00000000000000000000000000000000>:0
at System.Reflection.Emit.DynamicMethod+AnonHostModuleHolder..cctor () [0x00000] in <00000000000000000000000000000000>:0
at System.Reflection.Emit.DynamicMethod..ctor (System.String name, System.Reflection.MethodAttributes attributes, System.Reflection.CallingConventions callingConvention, System.Type returnType, System.Type[] parameterTypes, System.Type owner, System.Reflection.Module m, System.Boolean skipVisibility, System.Boolean anonHosted) [0x00000] in <00000000000000000000000000000000>:0
at System.Reflection.Emit.DynamicMethod..ctor (System.String name, System.Type returnType, System.Type[] parameterTypes, System.Boolean restrictedSkipVisibility) [0x00000] in <00000000000000000000000000000000>:0
at System.Linq.Expressions.Compiler.LambdaCompiler..ctor (System.Linq.Expressions.Compiler.AnalyzedTree tree, System.Linq.Expressions.LambdaExpression lambda) [0x00000] in <00000000000000000000000000000000>:0
at System.Linq.Expressions.Compiler.LambdaCompiler.Compile (System.Linq.Expressions.LambdaExpression lambda) [0x00000] in <00000000000000000000000000000000>:0
at System.Linq.Expressions.Expression`1[TDelegate].Compile (System.Boolean preferInterpretation) [0x00000] in <00000000000000000000000000000000>:0
at Google.Protobuf.Reflection.FieldAccessorBase..ctor (System.Reflection.PropertyInfo property, Google.Protobuf.Reflection.FieldDescriptor descriptor) [0x00000] in <00000000000000000000000000000000>:0
at Google.Protobuf.Reflection.SingleFieldAccessor..ctor (System.Reflection.PropertyInfo property, Google.Protobuf.Reflection.FieldDescriptor descriptor) [0x00000] in <00000000000000000000000000000000>:0
at Google.Protobuf.Reflection.FieldDescriptor.CreateAccessor () [0x00000] in <00000000000000000000000000000000>:0
at Google.Protobuf.Reflection.FieldDescriptor.CrossLink () [0x00000] in <00000000000000000000000000000000>:0
at Google.Protobuf.Reflection.MessageDescriptor.CrossLink () [0x00000] in <00000000000000000000000000000000>:0
at Google.Protobuf.Reflection.FileDescriptor.CrossLink () [0x00000] in <00000000000000000000000000000000>:0
at Google.Protobuf.Reflection.FileDescriptor.BuildFrom (Google.Protobuf.ByteString descriptorData, Google.Protobuf.Reflection.FileDescriptorProto proto, Google.Protobuf.Reflection.FileDescriptor[] dependencies, System.Boolean allowUnknownDependencies, Google.Protobuf.Reflection.GeneratedClrTypeInfo generatedCodeInfo) [0x00000] in <00000000000000000000000000000000>:0
at Google.Protobuf.Reflection.FileDescriptor.FromGeneratedCode (System.Byte[] descriptorData, Google.Protobuf.Reflection.FileDescriptor[] dependencies, Google.Protobuf.Reflection.GeneratedClrTypeInfo generatedCodeInfo) [0x00000] in <00000000000000000000000000000000>:0
at Moongate.GatewayReflection..cctor () [0x00000] in <00000000000000000000000000000000>:0
at Moongate.BootstrapRes.get_Descriptor () [0x00000] in <00000000000000000000000000000000>:0
@zaneclaes do you know about this one #3794?
Yep; I referenced it in my last two comments @jtattermusch ; again:
My understanding is that @jskeet 's open PR fixes the issue by avoiding the use of expression trees, but I have been unable to validate this (if someone can provide a compiled DLL of his work as a drop-in replacement for the NuGet package, I'd be happy to do so)
Since I have a work-around by avoiding the use of the methods which trigger the error, I haven't invested in cross-compiling the PR just yet because I'm on OSX ;)
@zaneclaes building the Google.Protobuf assembly should be trivial:
- install dotnet SDK
- dotnet restore Google.Protobuf.sln
- dotnet build Google.Protobuf.sln
#3794 has been merged.
Excellent, thanks @jtattermusch and @jskeet ! Sorry I didn't get to test this before it got merged; this isn't my main work and I have limited time each day ;) Look forward to NuGet being updated!
Closing as this appears to be fixed.
@haberman I don't think this is really "fixed." There is limited experimental support for Android alone. It still doesn't work on many platforms, and even those it does work on require a lot of voodoo and undocumented tweaking to use.
@zaneclaes Can you help give a summary on what's working and what's not working?
I can try. The core challenge is that grpc_csharp_ext needs to compiled natively for each platform, and placed into the Assets/Plugins directory of Unity.
- If you use the NuGet package, you can dig into the compiled binaries and copy
grpc_csharp_ext.dllandlibgrpc_csharp_ext.sointo this directory to make Windows and Linux work. - To make OSX work, you need to rename
libgrpc_csharp_ext.dylibto abundlebefore copying. - To make Android work, you need to use the special compilation implemented in this thread and copy its
libgrpc_csharp_ext.so - There is a partially working iOS solution I've detailed on my blog (it doesn't support bitcode yet).
- There is no working WebAssembly solution yet.
- Other platforms may also be broken (I've only enumerated the "big ones.")
I may have also made some additional small tweaks to GRPC core, which I'll have to dig a little deeper on. But that covers the main points.
FWIW, Unity has its own distribution channel (the asset store), which is probably the better route for distributing a working GRPC build to Unity users, which I'd be very interested in building/maintaining, if I could get some strong support from the rest of the community.
Those sound like gRPC issues - this issue is about just protobuf, which I believe is in a rather better state for Unity now, although it still requires some code to get reflection working correctly.
Oh, gosh. So sorry; I spent much more time interacting with the gRPC side of this and totally overlooked that this issue was in Protobufs. It seems that this can be closed! Apologies.
Thanks for the updates!