Support optional use of Yarn for packaging (instead of NPM)

Question

Support optional use of Yarn for packaging (instead of NPM)

RishitKedia opened this issue 7 years ago · comments

Hey! 👋

As discussed here with @HyperBrain, it'd be great if we could optionally use Yarn for packaging files, instead of NPM. 🙂

In particular, since serverless-webpack completely relies on NPM and uses it for packaging files, I'm having issues using serverless-webpack with Yarn Workspaces.

Currently, the bundled file is very huge (4 MB when used with Yarn Workspaces) compared to the same project when not used with Yarn Workspaces (25 kB). serverless-webpack bundles all the dependencies in one file even when using webpack-node-externals and webpackIncludeModules.

Thanks! 😃

Andrew Gerst · Answer 1 · Sat Feb 24 2018 02:38:31 GMT+0800 (China Standard Time)

What's the status on this? Why doesn't this work while using yarn? @HyperBrain

Frank Schmid · Answer 2 · Sat Feb 24 2018 03:15:12 GMT+0800 (China Standard Time)

Because the plugin internally uses npm - and these are orthogonal in some regions (esp. workspaces and links - and finally the lock files).

Regarding the timeline: I'll try my best to phase this in asap.

In general, it would work like this:
You will have a new configuration variable custom: webpack: packager which can be npm (default) | yarn. Depending on this setting all calls currently done by using npm would be done by calling yarn with the similar commands`.

But first, from the architecture of the project, the packager itself (npm) should be abstracted into separate classes, so that yarn just can be added as additional packager. This approach will keep the way open for other packagers.

Andrew Gerst · Answer 3 · Tue Feb 27 2018 00:16:47 GMT+0800 (China Standard Time)

Thanks! Would webpack-node-externals also need to be updated to support yarn or is this the only package that needs to be updated?

Frank Schmid · Answer 4 · Tue Feb 27 2018 00:18:53 GMT+0800 (China Standard Time)

No, only serverless-webpack. As far as I know node-externals reports external modules independently from the actually used packager.

Olzhas Alexandrov · Answer 5 · Sun Mar 04 2018 12:11:49 GMT+0800 (China Standard Time)

@HyperBrain Thank you very much for the package.
Please consider also adding support for --flat:
$ yarn install --flat
Or, in other words, support for resolutions in package.json.

Frank Schmid · Answer 6 · Mon Mar 05 2018 19:59:14 GMT+0800 (China Standard Time)

@OlzhasAlexandrov Thanks for the hint :) I will consider it. As soon as I have something available in a PR, I'll write here, so that people can start testing and verifying.

Frank Schmid · Answer 7 · Thu Mar 08 2018 10:39:50 GMT+0800 (China Standard Time)

Update: Seems the first version of the Yarn packager works:

Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: source-map-support@^0.5.3, flow-runtime@^0.16.0, bluebird@^3.5.1, lodash@^4.17.5, ...

This output is Yarn, not npm!

Serverless: Using configuration:
{
  "webpackConfig": "./webpack.config.js",
  "includeModules": true,
  "packager": "yarn",
  "packagerOptions": {},
  "packExternalModulesMaxBuffer": 204800
}

I will try to prepare a PR tomorrow, so that we have something that we can base further ideas (especially options and their defaults) onto. Stay tuned.

Frank Schmid · Answer 8 · Thu Mar 08 2018 21:03:01 GMT+0800 (China Standard Time)

At everyone here: The initial PR is available and ready for testing. I'll continue with the unit tests and proper documentation there. Please use "serverless-webpack": "github:serverless-heaven/serverless-webpack#yarn-support".

BUT: Bear in mind, that the PR is based on the just released version 5 of the plugin and you need to switch to the new configuration layout in serverless.yml to make it work with yarn. (low risk and easy)

[UPDATE: use the --verbose switch with the serverless command line to check what it is actually doing and if it is doing things right]

Frank Schmid · Answer 9 · Sun Mar 11 2018 20:49:56 GMT+0800 (China Standard Time)

I will merge the PR in the next few days. As mentioned in the PR, flat support is dropped for now. The feature in general should be functional.

If we encounter any issues, we should create separate issues for them and fix them or add missing stuff separately.

Blake Smith · Answer 10 · Mon Mar 12 2018 04:59:12 GMT+0800 (China Standard Time)

Does this have any impact for those of us not using webpack? The "Excluding development dependencies..." step takes awhile for me and I figure yarn would be faster than npm.

Edit: Nevermind, just saw the repository is specifically webpack.

Frank Schmid · Answer 11 · Mon Mar 12 2018 18:07:05 GMT+0800 (China Standard Time)

@bfsmith serverless-webpack does the packaging on its own and bypasses Serverless' packaging, so this is specific to this plugin. ... and you're right, using Yarn for packaging is a lot faster than NPM on most systems. If you use Node/JS as language, it would be worth a try to use webpack just to get an improved packaging behavior for the projects 😄

Noel Llevares · Answer 12 · Wed Mar 14 2018 06:50:55 GMT+0800 (China Standard Time)

Any idea why I'm getting Could not find packager 'yarn' when doing an sls deploy -v?

Frank Schmid · Answer 13 · Wed Mar 14 2018 20:56:14 GMT+0800 (China Standard Time)

@dashmug Then it does not find the yarn executable during packging (I assume you enabled it with `webpack: pacakger: yarn). Is it available in the serverless' command environment (path)?

The plugin spawns yarn install instead of npm install if configured for yarn, so the executable has to be found by the OS.

Noel Llevares · Answer 14 · Wed Mar 14 2018 21:04:40 GMT+0800 (China Standard Time)

Yes, yarn is available.

I think it might be just specific to my setup. I believe one of the following (or both) reasons may have caused it.

I'm using nodenv to pin my node version (v6.10) for this project. Global npm modules are installed differently when inside a nodenv environment.
yarn is installed using Homebrew so it uses my system's node version (v9.8.0). It was installed using brew install yarn not npm install -g yarn.

Frank Schmid · Answer 15 · Wed Mar 14 2018 21:08:02 GMT+0800 (China Standard Time)

(2) Might be the problem. As far as I remember the installation of global modules (like npm or yarn) is put into the specific node installation and tightly bound to the node version. I experienced that with nvm where every installed node version has its own globals.

So you should install yarn into the Node 6.10 context too, and it should work.

Frank Schmid · Answer 16 · Thu Mar 15 2018 02:48:41 GMT+0800 (China Standard Time)

Released with 5.1.0

Andrew Gerst · Answer 17 · Thu Mar 15 2018 04:35:09 GMT+0800 (China Standard Time)

Nice work @HyperBrain I'm impressed!

Noel Llevares · Answer 18 · Thu Mar 15 2018 07:55:01 GMT+0800 (China Standard Time)

@HyperBrain It turns out I got that error on v5.0.0 and I thought this PR was included in that release.

Updated to v5.1.0 now and it worked flawlessly despite nodenv and Homebrew yarn.

Good job!

dekryptic · Answer 19 · Sun May 06 2018 11:40:52 GMT+0800 (China Standard Time)

Is this meant to work with yarn workspaces? I'm getting error Your lockfile needs to be updated, but yarn was run with --frozen-lockfile., possibly because the yarn lockfile and the base package.json are different?

Also, for some reason I only have this issue when deploying to API Gateway and not when using serverless-offline.

Frank Schmid · Answer 20 · Sun May 06 2018 18:59:38 GMT+0800 (China Standard Time)

@dekryptic I did not try it with yarn workspaces, but the lockfile error normally means, that Yarn would change the lockfile during a Yarn install, i.e. the lockfile does not represent the contents of the package json. Do you have anything in your setup that changes only the package.json but does not update the lockfile? Can you try the very same with the latest serverless-webpack version (5.1.5)? I improved the error handling there and the plugin now checks that needed runtime dependencies are present in the dependencies section of the project.

I only have this issue when deploying to API Gateway and not when using serverless-offline.

That is expected. serverless-offline uses the locally available node_modules, but a deploy makes sure that only dependencies are packaged that are needed. You can do the packaging without an actual deployment for testing: serverless package.

dekryptic · Answer 21 · Sun May 06 2018 19:13:39 GMT+0800 (China Standard Time)

@HyperBrain Thank you for the quick response. I've upgraded to 5.1.5, but I think the problem is related to the nature of yarn workspaces. The basic premise is that you can have multiple package.json's in your subdirectories that specify your project's dependencies, so almost by definition the lockfile would have packages that are not present in the base package.json file. Well, at least this is what I understand.

Frank Schmid · Answer 22 · Sun May 06 2018 19:16:21 GMT+0800 (China Standard Time)

@dekryptic Yeah. I think you're right. Maybe the best way would be that you create a separate feature request for support of Yarn workspaces. We could then do explicit experiments for that and document the behavior of Yarn, its lockfiles and the workspaces there to be able to find a proper solution.