Create Plugin System for CLI using Dynamic Imports
tgadam opened this issue · comments
Create Plugin System for CLI using Dynamic Imports. Example https://medium.com/deno-the-complete-reference/dynamic-imports-in-deno-5e9eb7f66238
@juanneilson we shared pair and discuss this further.
@juanneilson feel free to hand this off to someone else
@AndresPerezTesela could be looking into this ticket, but will chat with @tnrich too.
We are having challenges using dynamic imports and compiling to a single executable using with deno.
A little bit more research is needed. High priority to this ticket.
Research was done for the following tools, which follow the plan on bundling the CLI in an executable.
-
Javy: Shopify's beta JS to WASM tool.
- Would allow us to continue with Deno using their Web Assembly API.
- It's still in beta and error handling isnt very user friendly.
- Plugin development for the community might be a frustrating process.
-
nexe/nexe: Node to executable tool.
- Not for Deno. We'd need to move out from Deno into maybe TS.
- Easy to use and errors are more clear (pretty similar to actual nodejs error messages)
- Not all npm dependencies can be used. Specially not those that run shell commands which depend on the platform.
- Does NOT support cross-platform compiling. nexe/nexe#688
- Supports Dynamic Importing.
-
vercel/pkg: Node to executable tool.
- Not for Deno. We'd need to move out from Deno into maybe TS.
- Does not support ESM. But this can be overcome by ESM to CommonJS converters.
- Supports cross-platform compiling.
- Does not support Dynamic Importing.
Nexe seems like a good choice. Although http(s) module imports are supported only in experimental mode via the flag --experimental-network-imports, unfortunately I havent been able to make nexe compile node with this flag ON (seems like nexe doesnt support such a flag yet).
After some iteration seems like we are going to have to download the remote module's bundle, write it into the filesystem and then import it dynamically.
NOTES
- The whole repo is built using Nx and Deno, so if we decide to go this route we'll have to refactor most of the code. Specially the dependency management implemented for deno and the needed configurations to get the mono repo working with typescript.
- This route will allow us to compile the app into a single executable, however we would be leaving all the benefits that deno offers behind. I think it's wise to discuss this more in depth so we're all in agreement before starting to work on this. Im not so very convinced this course change its worth it, we can always implement an installer script anyways...
Here are some additional thoughts from discussions we've had around this:
Background:
Deno was originally selected because it supports dynamic and remote imports. This would enable people extending the tools to create and host their plugins anywhere on the internet and the tools could download the plugins on-demand over the network. Deno was also selected because it would compile to a single executable that targeted all major platforms providing a low effort distribution and installation path as well as avoiding any potential issues regarding runtime version mismatches.
It was only recently discovered that these two features, dynamic imports and compiled executables are not compatible together in Deno. If you use dynamic imports then you can't compile to a single executable.
This means we have to adjust our approach either regarding installation, regarding extensibility, or using the Deno platform.
Additional Options
In addition to the options already evaluated. we've also discussed but not evaluated:
- Using a 3rd party installer to package the entire Deno runtime with the code into a single file.
- Creating or adopting a DSL (domain specific language) for plugins and implementing an interpreter for the DSL in the tools.
- Making the plugin framework essentially a "shell" runner. Meaning that plugin authors would need to package their plugins up as single exe's as well and then we would download them and run through as separate processes.
We're going to move forward with Deno and figure out a way to build our own installation packages.
Had this branch made where deno is ditched in favour of typescript in case we decide to go back in the future: https://github.com/TeselaGen/fsml.org/tree/refactor/ap/deno-out-ts-in
@tgadam This is a real problem for us in the CLI that we are developing for Deno as well. At a minimum would love to find out about any solution you come up with, but also we're open to collaborating on it if you want help.
@tgadam This is a real problem for us in the CLI that we are developing for Deno as well. At a minimum would love to find out about any solution you come up with, but also we're open to collaborating on it if you want help.
@cowboyd for now we're thinking of adapting the packaging and installation scripts that OClif uses for Deno.
We're close to having a plugin framework and example Microbyre plugin working
@tgadam sounds amazing. I'm very curious how you ended up loading your plugins dynamically.
@tgadam sounds amazing. I'm very curious how you ended up loading your plugins dynamically.
@cowboyd we used Deno's dynamic import. Here's an article that is a good starting point https://medium.com/deno-the-complete-reference/dynamic-imports-in-deno-5e9eb7f66238
Ah I see, and then bundle Deno itself using the OClif style packaging.... makes sense.
Making progress with the plugin systems. Some preliminary tests are being run with the Microbyre dataset/plugin