Rust-only CLI with plugins

[David Sherret]

Problems

1. Annoying to maintain both a JS and Rust CLI, which have different config formats.
2. JS CLI is slower. Rust CLI is much faster because it parallelizes work.
3. JS CLI depends on node and npm. In the future if this supports other ecosystems then people needing to install npm wouldn’t make sense.
4. No plugins in the Rust CLI... would be nice to change that.

Solution

I think in order to do this the config file (ex. dprint.config.json) would have a new plugins array that contains a list of urls or file paths to .wasm files:

{
  "projectType": "openSource",
  "lineWidth": 80,
  "typescript": {
    "lineWidth": 120
  },
  "json": {
    "useTabs": true
  },
  "includes": [
    "**/*.{ts,tsx,js,jsx,json}"
  ],
  "plugins": [
    "https://plugins.dprint.dev/typescript-0.14.1.wasm",
    "https://plugins.dprint.dev/json-0.2.0.wasm"
  ]
}

On load, the CLI will:

1. Check the "plugins" array.
2. Check if the plugins have been downloaded in the past (via a cache), if not, downloads them to a cache -- should tell the user what it's doing so they know what's going on (and so they understand why it’s slow. Definitely have progress bars).
3. Loads the plugins from the file system (look up dynamic linking in Rust -- use WASM).
4. Continue as usual.

Benefits

• No dependency on a certain kind of package manager (ex. npm), though the cli binary will be distributed still on npm along with other methods (the plugin versions are not read from an npm package.json file or other config)
• Additional plugins can be added beyond the official ones.
• The plugin version specified in the config file is the version that will be used to format, so people will be able to format multiple projects with the version defined in the config (in other words, dprint can be installed globally and work fine across projects).

Additional CLI Changes

• Config file will be required in order to format (similar to how it's required for the JS CLI)
• Definitely an dprint --upgrade command would be nice. Perhaps stripping the version off the url would return meta data on the plugin... will need to come up with some format... could be website specific defined in the cli—ex. dprint.dev and github.com specific support... I’d prefer it being locked down at the start)
• Future: dprint --clear-plugin-cache or some way to clear the plugin cache (perhaps the cache should keep track of last access time and just delete plugins that haven't been used for X number of days... could be a cache retention days setting or something... maybe that's too complicated)
• Probably a way to specify cache directory.

Initial Implementation Steps

1. Investigate dynamic linking in Rust and try using it hardcoded in the Rust CLI. - Use web assembly.
2. Create the caching system.
3. Update the CLI to use the caching system.

[Khải]

(look up dynamic linking in Rust).

I don't think this is scalable. Different platform handle dynamic libraries different ways, not to mention Node.js.

I have two mutual-exclusive suggestions:

1. Use WASM modules. Node.js can already load WASM, and Rust has libraries to load WASM.
2. Use UNIX interfaces, i.e. command-line and stdio pipe. Compile the plugins to executable programs that read input code via stdin and print formatted code via stdout. For Node.js, use WASI.

[David Sherret]

@KSXGitHub right now on my machine I have a basic prototype working. It seems when compiling to a dylib (https://doc.rust-lang.org/reference/linkage.html) Rust will create a .dll file for Windows, .dylib on osx, and .so on Linux. I assume the .dll and .dylib files will be portable, but maybe not the .so? I’m not a linux user at the moment (used to be) and I’m unfamiliar with this. Do you know?

I don’t want to use wasm modules because I don’t think they support multi-threading at the moment (I could be wrong though). I’ll compile something separately for use in node.js. I’m very inexperienced in this area.

Yeah, creating executables per plugin was my original idea then have subprocesses running in the background (at least on windows, then use what you said on linux).

Thanks for your help!

[David Sherret]

I was talking with a friend and we both agreed that sandboxing is most important. Downloading libraries that can do whatever is a big concern, so I think using WASM modules here would be best. It would also simplify things so much. I think I can use https://wasmer.io

Concern would be performance (no multithreading), but I think I can create a wasm instance pool and share those between threads.

[Khải]

@dsherret Sandboxing is to protect against unknown third-party programs, but since TypeScript plugin and JSON plugin are both developed by you, it is not any more risky than to use dprint in the first place. So for these two, dynamic libraries is fine. As for actual third-party plugins, you may create another plugin that allow executing WASM.

There're still people who do not want to trust any part of dprint, you may release entirety of dprint as a WASI program for those people.

Doing all that might be a lot of work, so it is understandable that you just want to make WASM only for simplicity.

[Bartek Iwańczuk]

@dsherret I'll be interested in your findings on the subject; I feel the same challenges are gonna appear in deno lint.

[David Sherret]

This is released now.

@bartlomieju what was helpful for this was to use https://wasmer.io and to cache the compiles. By doing that, plugins can then be instantiated in 5-45ms depending on the plugin. Right now, the initial compile on first load is somewhat slow (maybe 2-10 seconds), but wasmer has done a rewrite to make it much faster so I'm going to try that out soon (it's in beta).

Also, what was helpful was to create a macro that generates the web assembly interface. See creating a plugin and the generate_plugin_code macro. I think probably I'm not doing something as optimized as it could be, but so far it's pretty good and fast.

[Bartek Iwańczuk]

@dsherret for you for the info and blazing the trails on this one! I will revisit this topic in the coming weeks when I'll work on plugin system for deno_lint.