Port SASS plugin to use the Dart Sass JS compiler

Question

Port SASS plugin to use the Dart Sass JS compiler

longwa opened this issue 3 years ago · comments

Given that libsass is deprecated and the official recommendation is to use the Dart Sass compiler, it seems reasonable to try and port the asset pipeline sass plugin to use this compiler.

This would also allow for native compatibility with the Apple M1 machines, which a lot of developers are starting to use (including me).

Per my discussion with @davydotcom, I have started prototyping a solution using ~~J2V8~~ Javet to execute JavaScript natively in the JVM.

At a high level, my initial implementation is:

Use the NPM gradle plugin to pull down the dart-sass package as part of the plugin build
Run webpack to bundle it into a single JS file which is included in the plugin JAR
Create a Javet V8 runtime in the SassProcessor and load the sass.js bundle
Implement the importers callbacks in Sass via Java Function support in Javet to resolve imports via the classpath.
Build a native M1 version of the Javet library (stretch goal)

There are some technical challenges still with this approach:

J2V8 seems to be a hot mess from a build standpoint. The MacOS builds are very old and so far I can't get them to build locally due to numerous compile issues. It's not even clear if they support running on MacOS anymore let alone the M1.
~~J2V8~~ Javet needs custom libraries for different platforms. How will that work with this plugin? Will we need to have people install a version built against the version for their platform? That seems nasty. Maybe it can include all of the native libraries and just choose the correct one at runtime. A little heavier but more user friendly.
The current MacOS version of J2V8 is too old to process the resulting sass.js bundle that webpack builds. I tried transpiling to ES6, but that also doesn't seem to work. Currently I'm trying to build a newer MacOS version of J2V8, but see item 1 on this list for more on that.

Aaron · Answer 1 · Wed Nov 24 2021 13:24:03 GMT+0800 (China Standard Time)

The author of the Javet framework suggested on a J2V8 issue that his library might be more up-to-date and suitable:

eclipsesource/J2V8#498

https://github.com/caoccao/Javet

I'm able to load the sass.js bundle and execute successfully using Javet, so I'm going to continue down that road.

Sam Cao · Answer 2 · Thu Nov 25 2021 08:54:10 GMT+0800 (China Standard Time)

So far, Javet doesn't support Apple Silicon because I don't own such device. It would be great if you could help support that. I look forward to your pull requests. Good luck!

Aaron · Answer 3 · Thu Nov 25 2021 09:23:57 GMT+0800 (China Standard Time)

@caoccao Absolutely, assuming I can get the Dart Sass compiler fully working with Javet, I will definitely work on porting natively to the M1. Thanks for Javet and the MacOS version, I honestly don't know what our options would be without something like this.

Sam Cao · Answer 4 · Thu Nov 25 2021 10:03:56 GMT+0800 (China Standard Time)

Sure, there is always a way out and that keeps motivating me on maintaining Javet.

By the way, as the JS infra is moving towards Rust, I wonder if asset-pipeline will embrace that trend in the future. I tested swc (with ~20X performance improvement) and confirmed it can work well with Javet on Linux and Windows. So, basically, the Rust based JS infra can be supported by Javet natively. Hope that makes sense to this project.

Aaron · Answer 5 · Fri Nov 26 2021 05:58:48 GMT+0800 (China Standard Time)

@caoccao I was wondering, is there a way with Javet to choose, at runtime, which native library is used?

I'd like to be able to distribute this asset-pipeline plugin for all platforms and choose the native Javet implementation at runtime as opposed to forcing people of this library to choose a platform and arch.

Aaron · Answer 6 · Fri Nov 26 2021 06:29:12 GMT+0800 (China Standard Time)

I have a working version on x86 that successfully compiles our project assets (which have a pretty complex set of imports including BS 4.6).

Sam Cao · Answer 7 · Fri Nov 26 2021 09:06:25 GMT+0800 (China Standard Time)

@caoccao I was wondering, is there a way with Javet to choose, at runtime, which native library is used?

I'd like to be able to distribute this asset-pipeline plugin for all platforms and choose the native Javet implementation at runtime as opposed to forcing people of this library to choose a platform and arch.

Yes, Javet allows custom native library deployment.

Javet classes can be completely separated from the native libraries. The current packaging strategy is a trade-off between package size and end-user experience.

All native libraries (Android, Mac OS, Linux and Windows x Node.js and V8) can be packaged together, of course, the jar file size can reach 100+MB.
Some Javet users just customize the jar file by embedding the native libraries they need. The jar file size can be less than 10MB.
Some Javet users just package the classes in the jar file and tell Javet where to load the native libraries. This allows Javet to work with the native image.

I guess option 3 might be the one you are looking for. Here is what I would do if I were you.

Tweak the Javet official jar file by removing all the native libraries. That will give you a ~200KB jar file.
Detect the runtime OS and CPU arch.
Download the corresponding zipped native library (~5MB) somewhere if you have the infra, or directly from maven central.
Put the native library to a local folder.
Follow this doc to tell Javet where the native library is.
Load Javet and Voilà 😉

Please let me know if you have any questions. You may reach me at discord.

Aaron · Answer 8 · Mon Nov 29 2021 01:53:55 GMT+0800 (China Standard Time)

I was able to build a native M1 version of Javet from scratch (building V8 and Node.js for M1 as well). All of the sass-asset tests are passing and our main project using BS 4.6 is also compiling successfully on the M1.

Just have to solve the packaging problem which Sam has given direction on above as well as a problem where the Gradle daemon is holding onto the JNI library and causing a failure when trying to start another daemon.

Caused by: java.lang.UnsatisfiedLinkError: Native Library /private/var/folders/19/ph165v_n45j288nr69_djr5h0000gn/T/javet/88301/libjavet-node-macos-arm64.v.1.0.5.dylib already loaded in another classloader

I think unloading and loading the library on demand is probably the fix for both issues.

Sam Cao · Answer 9 · Mon Nov 29 2021 16:28:02 GMT+0800 (China Standard Time)

Congratulations on your wonderful progress!

Regarding this error, if everything runs smoothly, it can be ignored. Javet default logger just logs it as an error when the JNI lib is being loaded to the same JVM twice. Basically, it doesn't mean things are breaking.

You may inject your custom logger into Javet to mute this error, I think. Or, I may update Javet to read a system property for skipping this error.

Behind the scene, JVM only allows one memory copy of a particular JNI lib regardless of which classloader the lib is being loaded to. I think Gradle reuses the same JVM process when restarting a new daemon and this issue occurs.

Demian Seiler @ Intrepid Networks · Answer 10 · Mon Nov 29 2021 22:38:04 GMT+0800 (China Standard Time)

I just want drop in here and say thank you for working on this. I just got my new work MacBook Pro M1x and ran across this in setting up my local dev environment but haven't had time to allocate to figure out the issue due to year end commitments. I look forward to the progress on this and again, thank you. 🥇

Aaron · Answer 11 · Thu Dec 02 2021 12:18:03 GMT+0800 (China Standard Time)

Linking the M1 Javet support ticket: caoccao/Javet#120

Aaron · Answer 12 · Tue Dec 07 2021 01:41:17 GMT+0800 (China Standard Time)

Waiting for Javet 1.0.6 before opening a PR for the new plugin. There are a few fixes for dynamically loading the native libraries needed in that release as well as the native M1 support.

azavisha-snap · Answer 13 · Tue Dec 07 2021 01:45:07 GMT+0800 (China Standard Time)

Thank you for your contributions @longwa. You saved me a lot of time! Looking forward to this release.

Sam Cao · Answer 14 · Wed Dec 08 2021 08:23:27 GMT+0800 (China Standard Time)

Waiting for Javet 1.0.6 before opening a PR for the new plugin. There are a few fixes for dynamically loading the native libraries needed in that release as well as the native M1 support.

Javet v1.0.6 was released.

Aaron · Answer 15 · Fri Dec 10 2021 03:07:49 GMT+0800 (China Standard Time)

I have integrated the 1.0.6 version of Javet and have our team using it on our Macs (both x86 and ARM) as well as using it in our CI/CD pipelines. I'm going to give it a day or two internally to make sure there are not major issues and then I'll push forward with the PR.

Aaron · Answer 16 · Wed Dec 15 2021 22:36:00 GMT+0800 (China Standard Time)

PR #290 has been opened for this issue.

Moritz Kobel · Answer 17 · Tue Mar 08 2022 08:06:32 GMT+0800 (China Standard Time)

PR #299 allows to resolve absolute paths as in the old sass-asset-pipeline

Moritz Kobel · Answer 18 · Thu Apr 21 2022 01:35:07 GMT+0800 (China Standard Time)

PR #299 allows to resolve absolute paths as in the old sass-asset-pipeline

Is there anything I can do/improve for the acceptance of my PR #299 ?

Moritz Kobel · Answer 19 · Fri Jun 10 2022 21:06:20 GMT+0800 (China Standard Time)

PR #299 allows to resolve absolute paths as in the old sass-asset-pipeline
@longwa @davydotcom Is there anything I can do/improve for the acceptance of my PR #299 ?

Galeno de Melo · Answer 20 · Tue Aug 08 2023 02:49:32 GMT+0800 (China Standard Time)

For those who might experience it in the future: Grails 5 + Java 11.0.18-amzn (Corretto) don't work well because it's missing some native methods.

The solution, in our case, was to move to Temurin version. So far, so good.

I'd like to take the opportunity to thank you guys for the great work <3

Stacktrace

Caused by: java.lang.UnsatisfiedLinkError: 'long com.caoccao.javet.interop.V8Native.createV8Runtime(java.lang.Object)'
	at com.caoccao.javet.interop.V8Native.createV8Runtime(Native Method)
	at com.caoccao.javet.interop.V8Host.createV8Runtime(V8Host.java:266)
	at asset.pipeline.dart.SassProcessor.process(SassProcessor.groovy:80)
	at asset.pipeline.Processor$process$0.call(Unknown Source)
	at asset.pipeline.Processor$process$0.call(Unknown Source)
	at asset.pipeline.AbstractAssetFile.processedStream(AbstractAssetFile.groovy:171)
	at asset.pipeline.AbstractAssetFile.processedStream(AbstractAssetFile.groovy)
	at asset.pipeline.DirectiveProcessor.fileContents(DirectiveProcessor.groovy:305)
	at asset.pipeline.DirectiveProcessor$fileContents$18.callCurrent(Unknown Source)
	at asset.pipeline.DirectiveProcessor.loadContentsForTree(DirectiveProcessor.groovy:127)
	at asset.pipeline.DirectiveProcessor.compile(DirectiveProcessor.groovy:65)
	at asset.pipeline.DirectiveProcessor$compile$6.call(Unknown Source)
	at asset.pipeline.AssetCompiler$_compile_closure4.doCall(AssetCompiler.groovy:160)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)