ProcolEx unable to resolve cross-umbrella-app implementations on Elixir 1.15+
DaTrader opened this issue · comments
Elixir and Erlang/OTP versions
Erlang/OTP 25 [erts-13.2.2.9] [source] [64-bit] [smp:24:24] [ds:24:24:10] [async-threads:1] [jit:ns]
Elixir 1.15.7 (compiled with Erlang/OTP 25)
Operating system
Linux Mint
Current behavior
As of Elixir 1.15, the ProtocolEx library can no longer resolve the implementations of protocols residing in a different umbrella app. Prior to Elixir 1.15 this used to work just fine.
The following demo app reproduces the bug: https://github.com/DaTrader/demo_umbrella_1_15
Works properly:
asdf global elixir 1.14.5-otp-25
mix deps.get
mix compile
mix test # maybe needed to repeat it once again and the tests will pass for sureWorks only with implementations residing in the same umbrella app as the protocols they respectively implement.
asdf global elixir 1.15.7-otp-25
mix deps.get
mix compile
mix test # always fails the same testExpected behavior
The tests should pass on all supported Elixir versions, including 1.15.x and 1.16.x.
Can you please do a smaller reproduction sample? This includes all of LiveView, which is probably unrelated to the reported issue? Thank you.
I was able to dig a bit deeper into this and this is related to code path pruning. There is an actual bug in the library, which is why you mentioned sometimes it requires to run tests twice even in earlier Elixir versions:
mix test # maybe needed to repeat it once again and the tests will pass for sure
When you are inside an umbrella project, and you run mix test inside demo, you have only a subset of all dependencies. In versions prior to v1.15, we did not prune code paths, which meant you may or may not have demo_web in your code path. Elixir v1.15 makes it consistent, demo_web is never available when compiling demo, so the bug is always reproducible, since protocol_ex compiles protocols inside demo first (the first dependency to compile).
The way we solved this for Elixir protocols is to have three distinct consolidation paths, one per app, and one for the umbrella. The library would have to mirror that, instead of relying on a single application having a global view of the system (which was not true before and it is deterministically not true now).
Thanks a lot @josevalim
I am reporting this explanation of yours to the library author in hope he fixes it soon.
I'm testing by putting the compiler call 'on' the umbrella app itself, but it doesn't seem to get called (in fact calling mix compile in the umbrella directory rather than a specific app doesn't seem to compile anything except the dependencies listed within the umbrella itself, doesn't even compile the umbrellas apps?).
Continuing here: OvermindDL1/protocol_ex#67 (comment)
@josevalim I think this might actually be a mix bug? The ProtocolEx compiler isn't getting called at all for each app but the first it seems. It's getting called once, which is then compiling things up to that point, but it's not getting called again, even though each app explicitly adds it to the compiler list to run (as seen in DaTrader's example project above).
Copy/pasted message from that linked thread:
Looks like because 'demo' is comprehensive and 'demo_web' isn't doing much, hmm...
If I remove the protocol_ex compile from 'demo' so it's just in demo_web since demo_web contains demo then it does indeed compile there:
... snip demo ...
==> demo_web
Compiling 12 files (.ex)
Compiled lib/demo_web/application.ex
Compiled lib/demo_web/controllers/error_json.ex
Compiled lib/demo_web.ex
Compiled lib/demo_web/telemetry.ex
Compiled lib/demo_web/bar.ex
Compiled lib/demo_web/controllers/page_controller.ex
Compiled lib/demo_web/endpoint.ex
Compiled lib/demo_web/router.ex
Compiled lib/demo_web/components/core_components.ex
Compiled lib/demo_web/controllers/error_html.ex
Compiled lib/demo_web/controllers/page_html.ex
Compiled lib/demo_web/components/layouts.ex
Generated demo_web app
Consolidating ProtocolEx's project-wide...
ProtocolEx beam module Elixir.Demo.Fooable.beam with implementations [Demo.Fooable.DemoWebBar, Demo.Fooable.DemoFoo]
Consolidating ProtocolEx's project-wide complete.
Consolidated Plug.Exception
... snip more consolidated ...But if the compiler is in both then it gets invoked on demo, but not demo_web:
==> demo
Compiling 4 files (.ex)
Compiled lib/demo/application.ex
Compiled lib/demo.ex
Compiled lib/demo/fooable.ex
Compiled lib/demo/foo.ex
Generated demo app
Consolidating ProtocolEx's project-wide...
ProtocolEx beam module Elixir.Demo.Fooable.beam with implementations [Demo.Fooable.DemoFoo]
Consolidating ProtocolEx's project-wide complete.
==> demo_web
Compiling 12 files (.ex)
Compiled lib/demo_web.ex
Compiled lib/demo_web/controllers/error_json.ex
Compiled lib/demo_web/application.ex
Compiled lib/demo_web/telemetry.ex
Compiled lib/demo_web/bar.ex
Compiled lib/demo_web/controllers/page_controller.ex
Compiled lib/demo_web/endpoint.ex
Compiled lib/demo_web/router.ex
Compiled lib/demo_web/components/core_components.ex
Compiled lib/demo_web/controllers/error_html.ex
Compiled lib/demo_web/controllers/page_html.ex
Compiled lib/demo_web/components/layouts.ex
Generated demo_web app
Consolidated Plug.Exception
... snip more consolidated ...
If the ProtocolEx compiler were getting called at all then it should be working, but it's not getting called. Everything a given implementation needs is in the dependency tree so 'other' apps shouldn't matter.
I'm... like 80% sure this is a mix bug? It's not invoked the compiler in all cases, seems to just do it once and then never again, when it needs to be done for each app that requests it...
DaTrader: You could possibly work around this for now by having one app depend on all your others and have the protocol_ex compiler added on 'just' that one instead of all of them even if that app does nothing else?
Mix tasks are only executed once per project unless reenabled. That’s not a bug, Mix has always behaved as such. :)
In case of umbrellas, tasks are only executed on the parent umbrella, unless marked as recursive.
And even if it compiled per app, you would need different directories to compile down to. Otherwise you would depend on the order the last one is compiled (which would always win).
How can I run a compiler 'per' app then? Each should have its own consolidation as they should only know about their own tree (so 'others' that aren't in the tree and aren't loaded can't "infect" an implementation with code that would then crash due to missing functions to call at runtime)?
Mark the task as @recursive. I mentioned that in a previous comment. :)
I thought that was for tasks. Are compiler callbacks considered tasks?
Yes!