Error bundling @oclif/core into my project

[Isaac Roldán]

Describe the bug
Hi! We (shopify CLI) have a monorepo with our CLI and multiple plugins, and @oclif/core is a dependency on all of them.

We are trying to bundle each package with its dependencies, but when including @oclif/core in the bundle we can't make it work:
Running any command will crash on line 34 of command.ts

const pjson = requireJson<PJSON>(__dirname, '..', 'package.json')

I assume is looking for the @oclif/core package.json, but since the dependency is being bundled there isn't one.
Is there a way to avoid this? I see that this pjson is only used for a debug log, can we ignore the error if the file doesn't exist instead of crashing? I can make the PR if you agree with this approach :)

Thanks!

To Reproduce
Having a CLI, bundle it with esbuild, running any command should trigger this.

Expected behavior
Everything should work normally.

[Mike Donnalley]

@isaacroldan you're in luck! I just created a PR last week that fixes this: #945

Not sure on the timeline of getting that one merged (likely in the next couple weeks) but in the meantime it'd be a huge help if you could test it out for us using the dev tag.

You might also need oclif@dev if you run into any issues with running oclif readme

[Isaac Roldán]

Nice! I'll test it and let you know :)

[Isaac Roldán]

It works! although i'm not using the new explicit strategy, for now we are keeping our folder structure and just bundling dependencies, so i didn't test that part. But I love this new approach and being able to modify commands at runtime.

[Isaac Roldán]

Do you have plans to offer bundled versions of your plugins?
Otherwise bundling oclif doesn't make much sense because having @oclif/plugin-[plugins/commands/help] will force the installation of @oclif/core

[Mike Donnalley]

Great, thanks for testing it out 🏆

Do you have plans to offer bundled versions of your plugins?

I assume you mean bundled dependencies? As opposed to bundling the source code to a single file?

No plans at the moment. Although, I'm open to doing it. Are you imaging that there would be an unbundled @oclif/plugin-* and then a bundled @oclif/plugin-*-bundled?

[Isaac Roldán]

Yep, bundled dependencies for each plugin, exactly as you propose, having both versions would be ideal.

Asking because some plugins are really useful and even "needed", so if those don't have their dependencies bundled it makes it impossible really to bundle a CLI. That way I could keep @oclif/plugin-plugins-bundled as external knowing that it won't introduce any transitive dependencies.

Because bundling @oclif/plugin-plugins itself in my project doesn't work, I assume because of the way oclif loads plugins. Is there a workaround for that?

[Mike Donnalley]

so if those don't have their dependencies bundled it makes it impossible really to bundle a CLI. That way I could keep @oclif/plugin-plugins-bundled as external knowing that it won't introduce any transitive dependencies.

Can you help me understand why you would want or need to mark plugins as external? It seems to me that you could successfully bundled all your dependencies (including those of oclif plugins) if you didn't mark them as external, right?

I don't have much experience with bundling so this might be a dumb question, but wouldn't you also be unnecessarily increasing the size of your CLI if all your plugins had their own bundled dependencies?

[Isaac Roldán]

Oh i don't want to mark them as external, ideally it'd be great to bundle everything. But oclif can't load bundled plugins.

In fact plugins are not even bundled automatically by esbuild because there is no direct import of them from your code. esbuild follows the import tree to bundle just the used code, so dynamically loaded dependencies like plugins are never bundled.

But even if we trick esbuild into bundling them, it will copy the code but not the package.json, so all the info in the oclif section is lost. Oclif won't know how to find these bundled plugins.

Maybe this can be solved if your CLI explicitly imports the commands and hooks from every plugin, that way esbuild will be able to follow the path and put the code in the right place. (is that even possible?)

Or am i missing something? What do you think, what are the options?

[Mike Donnalley]

Ah okay - that makes more sense to me.

Any chance that you could provide a repository that minimally replicates what you're trying to accomplish?

[Isaac Roldán]

Yep, is a bit late on my timezone but I'll prepare and share something tomorrow :)

[Isaac Roldán]

This is an example CLI where we are bundling with esbuild: https://github.com/isaacroldan/oclif-cli-bundled. This project includes the @oclif/plugin-commands.

After cloning you can notice the dist folder already contains the output, and there is no node_modules.

Without running npm i, run:

./bin/run.js 

It will work, because @oclif/core is bundled.
But it is complaining that it can't find the commands plugin.
An of course ./bin/run.js commands doesn't work

If you then run npm install, it will create the node_modules folder and ./bin/run.js commands will work, because oclif can find it in the node_modules folder.

So things happening here:

• the plugin is not being bundled because there is no direct import from anywhere in our code, so esbuild ignores it (you can see in the dist folder that there is no reference to it)
• in any case, oclif will try to find a package.json for the plugin, which will be imposible because will never have a node_modules folder.

Notes:

• to re-build just run npm run build, that runs the script in bin/bundle.js
• To test stuff, you need to remove the node_module folder, because oclif will try to dynamically load from there.

Let me know what you think :)

[Mike Donnalley]

Thanks @isaacroldan - this is incredibly helpful.

I tried to use globby to get all file paths of plugins and include that in the entryPoints

Doing that successfully includes plugin-commands in the dist folder but it causes some difficulties:

• oclif uses require.resolve to find the entry point of a plugin and then looks up the filesystem until it finds the directory containing a package.json (for example require.resolve('@oclif/plugin-commands') => node_modules/@oclif/plugin-commands/lib/index.js => node_modules/@oclif/plugin-commands)
• require.resolve won't work in the bundle (not sure the exact reason yet) and even if it did, it wouldn't be able to find the root directory since esbuild converts the package.json to a package.js

While it's probably possible to make oclif work under these circumstances I don't believe it's feasible since I'd like for bundling to work with most bundlers and most configurations. I don't want to tie people down to a very specific bundler + configuration.

The easiest path forward is to have oclif plugins export commands and then CLIs can use the explicit strategy to expose those commands:

{
  "oclif": {
    "commands": {
      "strategy": "explicit",
      "target": "./dist/index.js",
      "identifier": "COMMANDS"
    },
   "plugins": []
}

// @oclif/plugin-commands/src/index.ts
import Commands from './commands/commands.js'

export const commands = {
  commands: Commands,
}

// src/index.ts
import {run, Command} from '@oclif/core'
import {commands as PluginCommands} from '@oclif/plugin-commands'

import Hello from './commands/hello/index.js'
import HelloWorld from './commands/hello/world.js'

export default async function runCLI() {
  await run(process.argv.slice(2))
}

export const COMMANDS: Record<string, Command.Class> = {
  ...PluginCommands,
  hello: Hello,
  'hello:world': HelloWorld,
}

There are a couple downsides to this approach

• it becomes tedious to include plugins in your CLI
• You lose any distinction between plugins. For instance, <cli> plugins --core won't show any plugins or <cli> which plugins:install will tell the user that the plugins install command belongs to the root CLI plugin instead of @oclif/plugin-plugins

I'm inclined to go in this direction since it would provide the most flexibility and won't tie us down to a specific bundler and/or configuration. But I'm open to figuring out alternatives if it's not viable for your situation.

[Isaac Roldán]

Yeah, I think your proposal makes sense, explicitly loading the commands seems like the best alternative.

This should be possible after PR #945 is merged right? no need to do anything extra? I can try to do it with one of our plugins to test it.

We don't mind about plugin distinction and we are not adding core plugins that often to be honest, so I wouldn't mind the extra work for this.

My only question is, this solves the commands part, but how would hooks work?

[Mike Donnalley]

This should be possible after PR #945 is merged right? no need to do anything extra? I can try to do it with one of our plugins to test it.

Yeah #945 is the bulk of the work but we'd need to go and update our plugins to export commands so that you're able to import them.

My only question is, this solves the commands part, but how would hooks work?

Great question. For your own hooks, take a look at the oclif + Bundling section in the description of #945

For hooks owned by other plugins, you would need to also import them and register them in the package.json. Something like this:

// src/index.ts
export {hook as pluginNotFoundHook} from '@oclif/plugin-not-found'

"oclif": {
    "hooks": {
      "command_not_found": {
        "target": "./dist/index.js",
        "identifier": "pluginNotFoundHook"
      }
    }
}

Again, we'd need to update our plugins to also export the hooks in addition to the commands.

[Isaac Roldán]

Gotcha, I think this is the best approach for this.

Let me know if I can help with anything (maybe with updating the plugins), happy contribute back to oclif!