NPE when trying to serialize trait values with generic values using shapeless

Question

NPE when trying to serialize trait values with generic values using shapeless

bbarker opened this issue 6 years ago · comments

Brandon Elam Barker commented 6 years ago

This occurs in both JS and the JVM; I packaged up a somewhat minimal example of this. Here is the problematic trait:

  sealed trait JobWrap {
    type J
    val job: J
    val ev: Job[J]
  }

JVM errror:

[info] boopickling various API data structures
[info] - should not throw an error (46 milliseconds)
[info] - should not throw an error when JobWrapped *** FAILED *** (4 milliseconds)
[info]   java.lang.NullPointerException:
[info]   at boopickle.PickleImpl$.apply(Default.scala:76)
[info]   at boopickle.PickleImpl$.intoBytes(Default.scala:81)
[info]   at org.xsede.jobrunner.model.MainSpec.$anonfun$new$3(MainSpec.scala:32)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1691)
[info]   at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
[info]   at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
[info]   at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1685)
[info]   at org.scalatest.FlatSpecLike.invokeWithFixture$1(FlatSpecLike.scala:1689)
[info]   at org.scalatest.FlatSpecLike.$anonfun$runTest$1(FlatSpecLike.scala:1701)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
[info]   at org.scalatest.FlatSpecLike.runTest(FlatSpecLike.scala:1701)
[info]   at org.scalatest.FlatSpecLike.runTest$(FlatSpecLike.scala:1683)
[info]   at org.scalatest.FlatSpec.runTest(FlatSpec.scala:1685)
[info]   at org.scalatest.FlatSpecLike.$anonfun$runTests$1(FlatSpecLike.scala:1759)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
[info]   at scala.collection.immutable.List.foreach(List.scala:389)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:373)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:410)
[info]   at scala.collection.immutable.List.foreach(List.scala:389)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
[info]   at org.scalatest.FlatSpecLike.runTests(FlatSpecLike.scala:1759)
[info]   at org.scalatest.FlatSpecLike.runTests$(FlatSpecLike.scala:1758)
[info]   at org.scalatest.FlatSpec.runTests(FlatSpec.scala:1685)
[info]   at org.scalatest.Suite.run(Suite.scala:1147)
[info]   at org.scalatest.Suite.run$(Suite.scala:1129)
[info]   at org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1685)
[info]   at org.scalatest.FlatSpecLike.$anonfun$run$1(FlatSpecLike.scala:1804)
[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
[info]   at org.scalatest.FlatSpecLike.run(FlatSpecLike.scala:1804)
[info]   at org.scalatest.FlatSpecLike.run$(FlatSpecLike.scala:1802)
[info]   at org.scalatest.FlatSpec.run(FlatSpec.scala:1685)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513)
[info]   at sbt.TestRunner.runTest$1(TestFramework.scala:76)
[info]   at sbt.TestRunner.run(TestFramework.scala:85)
[info]   at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
[info]   at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
[info]   at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:185)
[info]   at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
[info]   at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
[info]   at sbt.TestFunction.apply(TestFramework.scala:207)
[info]   at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
[info]   at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
[info]   at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
[info]   at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
[info]   at sbt.std.Transform$$anon$4.work(System.scala:63)
[info]   at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
[info]   at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
[info]   at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
[info]   at sbt.Execute.work(Execute.scala:237)
[info]   at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
[info]   at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
[info]   at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
[info]   at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info]   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info]   at java.lang.Thread.run(Thread.java:748)

JS error:

[info] boopickling various API data structures
[info] - should not throw an error (6 milliseconds)
[info] - should not throw an error when JobWrapped *** FAILED *** (2 milliseconds)
[info]   scala.scalajs.js.JavaScriptException: TypeError: Cannot read property 'pickle__O__Lboopickle_PickleState__V' of null
[info]   at boopickle.PickleImpl$.apply(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:2875:5)
[info]   at boopickle.PickleImpl$.intoBytes(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:2879:21)
[info]   at {anonymous}()(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:70295:80)
[info]   at scala.scalajs.runtime.AnonFunction0.apply(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:34267:23)
[info]   at org.scalatest.OutcomeOf.outcomeOf(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:1496:7)
[info]   at org.scalatest.Transformer.apply(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:46734:10)
[info]   at org.scalatest.FlatSpecLike$$anon$1.apply(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:35766:72)
[info]   at org.scalatest.FlatSpecLike.invokeWithFixture$1(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:65615:15)
[info]   at org.scalatest.FlatSpecLike.runTest(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:65818:5)
[info]   at {anonymous}()(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:65931:14)

Otto Chrons · Answer 1 · Thu Aug 30 2018 00:55:07 GMT+0800 (China Standard Time)

There is no way a macro could generate a pickler for such trait as it has no type parameters and no implementations, and could contain anything. Container types must have a way to reference the pickler for the contained data, You'll have to provide your own, custom pickler for such types.

johannes karoff · Answer 2 · Thu Aug 30 2018 04:37:51 GMT+0800 (China Standard Time)

@bbarker What you are seeing here is one of the annoyances of scala's initializers, you have defined an implicit pickler in the jobwrap companion:

implicit val jwPickler: Pickler[JobWrap] = generatePickler[JobWrap]

Now, in the shapeless module, generatePickler is defined like this:

def generatePickler[T](implicit pickler: Pickler[T]): Pickler[T] = pickler

So, when calling generatePickler on the rhs of the implicit val jwPickler, it will resolve the implicit pickler with the currently uninitialized field jwPickler. In this case, the pickler is null. If you just try to call generatePickler without the implicit in scope, it will tell you that there is no derivable pickler.

In the normal boopickle.Default object, we defined the generatePickler method in a different way. It calls a macro directly. With shapeless, we get that implicitly.

@ochrons It think, we should make both functions behave the same way in order to avoid this unexpected behavior. But not sure yet, how we could do this here. What do you think?