Upgrading from 2.2.11 to 2.2.12 causes JSC_UNDEFINED_VARIABLE exceptions in mdoc:js

Question

Upgrading from 2.2.11 to 2.2.12 causes JSC_UNDEFINED_VARIABLE exceptions in mdoc:js

raquo opened this issue 3 years ago · comments

Hello,

I have a build that works with mdoc 2.2.11, but if I upgrade mdoc to 2.2.12, then mdoc:js fails with the following error when I run docusarusCreateSite:

UPDATE 2: see the most minimal reproduction in Anton's comment below.

info: JSC_UNDEFINED_VARIABLE. variable $c_Lcom_raquo_airstream_eventstream_DelayEventStream is undeclared at file:///Users/raquo/code/scala/airstream/src/main/scala/com/raquo/airstream/eventstream/EventStream.scala line 35 : 4
info: JSC_UNDEFINED_VARIABLE. variable $c_Lcom_raquo_airstream_eventstream_MergeEventStream is undeclared at file:///Users/raquo/code/scala/airstream/src/main/scala/com/raquo/airstream/eventstream/EventStream.scala line 227 : 4
info: JSC_UNDEFINED_VARIABLE. variable $f_Lcom_raquo_airstream_core_Observable__mapTo__F0__Lcom_raquo_airstream_core_Observable is undeclared at file:///Users/raquo/code/scala/airstream/src/main/scala/com/raquo/airstream/eventstream/SwitchEventStream.scala line 27 : 6
info: JSC_UNDEFINED_VARIABLE. variable $c_Lcom_raquo_airstream_eventbus_EventBus is undeclared at virtualfile:time.md line 28 : 15
info: JSC_UNDEFINED_VARIABLE. variable $m_Lcom_raquo_airstream_core_Observable$ is undeclared at virtualfile:time.md line 32 : 26
info: Closure: 5 error(s), 0 warning(s)
error: mdoc:js exception
mdoc.internal.markdown.ModifierException: mdoc:js exception
Caused by: org.scalajs.linker.interface.LinkingException: There were errors when applying the Google Closure Compiler
	at org.scalajs.linker.backend.closure.ClosureLinkerBackend.compile(ClosureLinkerBackend.scala:136)
	at org.scalajs.linker.backend.closure.ClosureLinkerBackend.$anonfun$emit$5(ClosureLinkerBackend.scala:105)
	at org.scalajs.logging.Logger.time(Logger.scala:42)
	at org.scalajs.logging.Logger.time$(Logger.scala:40)
	at mdoc.modifiers.JsModifier$$anon$1.time(JsModifier.scala:46)
	at org.scalajs.linker.backend.closure.ClosureLinkerBackend.$anonfun$emit$3(ClosureLinkerBackend.scala:94)
	at scala.Option.map(Option.scala:242)
	at org.scalajs.linker.backend.closure.ClosureLinkerBackend.emit(ClosureLinkerBackend.scala:88)
	at org.scalajs.linker.standard.StandardLinkerImpl.$anonfun$link$2(StandardLinkerImpl.scala:47)
	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:433)
	at scala.concurrent.BatchingExecutor$AbstractBatch.runN(BatchingExecutor.scala:134)
	at scala.concurrent.BatchingExecutor$AsyncBatch.apply(BatchingExecutor.scala:163)
	at scala.concurrent.BatchingExecutor$AsyncBatch.apply(BatchingExecutor.scala:146)
	at scala.concurrent.BlockContext$.usingBlockContext(BlockContext.scala:107)
	at scala.concurrent.BatchingExecutor$AsyncBatch.run(BatchingExecutor.scala:154)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1425)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1665)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1598)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
info: Compiled in 19.89s (1 error)

Steps to reproduce on master branch of raquo/Laminar:

run website/docusaurusCreateSite. It will work fine.
In project/plugins.sbt, bump mdoc version to 2.2.12 (it uses 2.2.9 but I checked that 2.2.11 is the last version that works)
run website/docusaurusCreateSite again. It will fail with the error above now.

For context, com.raquo.airstream is a dependency of com.raquo.laminar. I'm the author of both, and I don't know what is special about the specific airstream-related variables that are not being found.

The Scala.js version that the Laminar project uses is 1.1.0. However, I tried upgrading all dependencies to 1.3.1, and making sure that 1.3.1 is selected by sbt, and that results in the same error. I'm just trying to narrow down the problem. If mdoc 2.2.12 is not supposed to support Scala.js 1.1.0, I can provide a build that uses Scala.js 1.3.1 and still fails with the same error.

Build files:

Laminar/project/plugins.sbt (website is the docs project)
Laminar/build.sbt

Is my expectation right that this upgrade should have worked with no other code changes? If not, sorry. But then, what am I missing?

Nikita Gazarov · Answer 1 · Mon Feb 01 2021 11:20:14 GMT+0800 (China Standard Time)

Some corrections:

Laminar/project/build.properties currently specifies sbt 1.3.13, sorry I was jumping between a few branches and didn't notice that. If I bump this to 1.4.6, website/docusaurusCreateSite works with Scala.js 1.1.0 and mdoc 2.2.12.

However, I still can't find a configuration that works with Scala.js 1.3.x. The following configuration does not work:

Laminar/project/build.properties:

sbt.version = 1.4.6    // 1.4.7 doesn't work either

Laminar/project/plugins.sbt:

addSbtPlugin("org.scala-js" % "sbt-scalajs" % "1.3.0")              // 1.3.1 doesn't work either
addSbtPlugin("ch.epfl.scala" % "sbt-scalajs-bundler" % "0.20.0")    // bump required for scala.js 1.3.x
addSbtPlugin("org.scalameta" % "sbt-mdoc" % "2.2.12")               // 2.2.16 doesn't work either
// ... plus unrelated plugins

I switched to testing with Scala.js 1.3.0 in case 1.3.1 isn't supposed to work.

The error message is the same except it complains about a slightly different set of Airstream class names:

info: JSC_UNDEFINED_VARIABLE. variable $f_Lcom_raquo_airstream_core_Observable__mapTo__F0__Lcom_raquo_airstream_core_Observable is undeclared at file:///Users/raquo/code/scala/airstream/src/main/scala/com/raquo/airstream/eventstream/ConcurrentFutureStream.scala line 18 : 6
info: JSC_UNDEFINED_VARIABLE. variable $c_Lcom_raquo_airstream_eventstream_DelayEventStream is undeclared at file:///Users/raquo/code/scala/airstream/src/main/scala/com/raquo/airstream/eventstream/EventStream.scala line 35 : 4
info: JSC_UNDEFINED_VARIABLE. variable $c_Lcom_raquo_airstream_eventstream_MergeEventStream is undeclared at file:///Users/raquo/code/scala/airstream/src/main/scala/com/raquo/airstream/eventstream/EventStream.scala line 227 : 4
info: JSC_UNDEFINED_VARIABLE. variable $c_Lcom_raquo_airstream_eventbus_EventBus is undeclared at virtualfile:time.md line 28 : 15
info: JSC_UNDEFINED_VARIABLE. variable $m_Lcom_raquo_airstream_core_Observable$ is undeclared at virtualfile:time.md line 32 : 26
info: Closure: 5 error(s), 0 warning(s)

Nikita Gazarov · Answer 2 · Wed Feb 03 2021 17:40:21 GMT+0800 (China Standard Time)

I made a small reproduction with mdoc failing to include Scala's built-in Success type. This confirms that this has nothing to do with my code or libraries.

I made a minimal branch on my project that demonstrates the issue.

Run sbt website/mdoc (or sbt website/docusaurusCreateSite) and see the following error:

[info] running mdoc.Main
info: Compiling 3 files to /Users/raquo/code/scala/laminar/website/target/mdoc
info: Closure: 0 error(s), 0 warning(s)
info: Closure: 0 error(s), 0 warning(s)
info: JSC_UNDEFINED_VARIABLE. variable $c_s_util_Success is undeclared at virtualfile:a.md line 11 : 7
info: Closure: 1 error(s), 0 warning(s)
error: mdoc:js exception
mdoc.internal.markdown.ModifierException: mdoc:js exception
Caused by: org.scalajs.linker.interface.LinkingException: There were errors when applying the Google Closure Compiler
	at org.scalajs.linker.backend.closure.ClosureLinkerBackend.compile(ClosureLinkerBackend.scala:136)
...

As you can see in the branch, in website/docs/examples, this requires a set of three very specific files, a.md, b.md, and c.md with the following contents:

a.md:

---
title: A
---

```scala mdoc:js
import scala.util.Success
Success("foo")
```

```scala mdoc:js
import scala.util.Success
Success("bar")
```

b.md:

---
title: B
---

```scala mdoc:js
println("b")
```

c.md:

---
title: C
---

```scala mdoc:js
println("c1")
```

```scala mdoc:js
println("c2")
```

I believe this is a minimal reproduction. If I remove any of those files or any of those code blocks, mdoc will succeed. Moreover, the naming of files is important. The .md file names must be in this order, alphabetically, to trigger the error.

I tested that this error happens with mdoc 2.2.12 and mdoc 2.2.16 regardless of Scala.js version (1.1.0, 1.2.0, 1.3.0) (ETA: I meant Scala.js version in my project, but it turns out that mdoc uses the Scala.js version defined in mdoc itself, which is 1.3.0 for mdoc 2.2.12+ and 1.1.0 for mdoc 2.2.11). On the other hand, mdoc 2.2.9 and 2.2.11 do work (with Scala.js 1.1.0). So, I guess there must be a regression somewhere in 2.2.12 or its dependency bumps, if any.

Anton Sviridov · Answer 3 · Fri Feb 05 2021 07:20:23 GMT+0800 (China Standard Time)

I've managed to achieve an even smaller reproduction - just using CLI and coursier, no dependencies: https://github.com/keynmol/mdoc-scalajs-problem/runs/1834922392

This should be a good starting point to play around with the order of files and whatever else affects it.

My attempt to reproduce it in tests in #455 is so far not successful.

Sébastien Doeraene · Answer 4 · Fri Feb 05 2021 13:13:46 GMT+0800 (China Standard Time)

Some thought: it seems that mdoc invokes the same linker multiple times, for each snippet within some set. That's great, as it means that it can leverage Scala.js' incremental linker.

It does mean that the bug we see here could be caused by a bug in the incremental linker of Scala.js.

Anton Sviridov · Answer 5 · Fri Feb 05 2021 17:51:40 GMT+0800 (China Standard Time)

Yep, creating a new linker solves this (tested locally).

In this scenario, the linker is created once for each file, and all snippets are processed with the same linker.

@sjrd any particular internal state of the linker I can look at?

In terms of raw Scala files, the linker receives the results of compilations of the following files:

info: Compiling 3 files to /tmp/tmp.84Xc0e6rbq
Linking a.md: Input.VirtualFile("a.md", "object mdocjs {
@_root_.scala.scalajs.js.annotation.JSExportTopLevel("mdoc_js_run0")
def run0(node: _root_.org.scalajs.dom.raw.HTMLElement): Unit = {
import scala.util.Success
Success("foo")
}

@_root_.scala.scalajs.js.annotation.JSExportTopLevel("mdoc_js_run1")
def run1(node: _root_.org.scalajs.dom.raw.HTMLElement): Unit = {
import scala.util.Success
Success("bar")
}

}
")
info: Closure: 0 error(s), 0 warning(s)


Linking b.md: Input.VirtualFile("b.md", "object mdocjs {
@_root_.scala.scalajs.js.annotation.JSExportTopLevel("mdoc_js_run0")
def run0(node: _root_.org.scalajs.dom.raw.HTMLElement): Unit = {
println("b")
}

}
")
info: Closure: 0 error(s), 0 warning(s)


Linking c.md: Input.VirtualFile("c.md", "object mdocjs {
@_root_.scala.scalajs.js.annotation.JSExportTopLevel("mdoc_js_run0")
def run0(node: _root_.org.scalajs.dom.raw.HTMLElement): Unit = {
println("c1")
}

@_root_.scala.scalajs.js.annotation.JSExportTopLevel("mdoc_js_run1")
def run1(node: _root_.org.scalajs.dom.raw.HTMLElement): Unit = {
println("c2")
}

}
")
info: JSC_UNDEFINED_VARIABLE. variable $c_s_util_Success is undeclared at virtualfile:a.md line 11 : 7
info: Closure: 1 error(s), 0 warning(s)
error: mdoc:js exception
mdoc.internal.markdown.ModifierException: mdoc:js exception
Caused by: org.scalajs.linker.interface.LinkingException: There were errors when applying the Google Closure Compiler
        at org.scalajs.linker.backend.closure.ClosureLinkerBackend.compile(ClosureLinkerBackend.scala:136)
        at org.scalajs.linker.backend.closure.ClosureLinkerBackend.$anonfun$emit$5(ClosureLinkerBackend.scala:105)

I tried to reproduce it by just using those scala files - but the compiler would not allow this set of files to exist in the same package (mdocjs defined multiple times) - could this be confusing the linker and getting it into a bad state?

Notice how it only breaks after linking the last file (c.md) and the variable is c_s_util_Success

Sébastien Doeraene · Answer 6 · Fri Feb 05 2021 17:54:53 GMT+0800 (China Standard Time)

Woot! I can reproduce this without mdoc! scala-js/scala-js#4416

Sébastien Doeraene · Answer 7 · Fri Feb 05 2021 17:56:27 GMT+0800 (China Standard Time)

Ah ah! I see we concurrently went through the same process @keynmol :p

Anton Sviridov · Answer 8 · Fri Feb 05 2021 17:56:53 GMT+0800 (China Standard Time)

You were more successful :) I started writing a bash script that was applying those changes in sequence :D