Unable to deploy a service using local packages

Question

Unable to deploy a service using local packages

mzmiric5 opened this issue 6 years ago · comments

This is a Bug Report

Description

When attempting to package or deploy a service which uses local packages either referenced via file: or yarn-workspace definitions, the deploy/package step that triggers the webpack build throws an error with

Command failed: yarn install --frozen-lockfile --non-interactive
error An unexpected error occurred: "Your lockfile needs to be updated, but yarn was run with `--frozen-lockfile`.".

Running yarn install --frozen-lockfile --non-interactive manually works just fine, so that points to an issue with serverless-webpack.

For bug reports:

What went wrong?
sls deploy and sls package fail with "Your lockfile needs to be updated, but yarn was run with `--frozen-lockfile`." error when using a local package
What did you expect should have happened?
The service should have been packaged and deployed successfully
What was the config you used?
What stacktrace or error message from your provider did you see?

@HyperBrain as requested in #369 I've opened a separate issue. I also went ahead and prepared a minimal repro repo with the config and dependencies set up that cause these issues to surface. https://github.com/mzmiric5/sls-wp-build-error-repro
Some notes:

services/sls-only is a template without webpack, which deploys correctly and has no problems running on AWS
shared/third-party-private is a placeholder for a package we are unable to share, but the fact that the package we are using has this one specific dependency is causing npm repro to fail with an error on npm install inside the sls package/deploy step
our team primarily uses yarn, so this is our primary concern, and services/sls-wp-yarn shows the issue the best, as it fails with both the third-party-private package and the moment-wrapper example package (which the npm packager doesn't have an issue with)

For feature proposals:

What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.
If there is additional config how would it look

Similar or dependent issue(s):

Additional Data

Serverless-Webpack Version you're using: 5.1.1
Webpack version you're using: 4.5.0
Serverless Framework Version you're using: 1.26.1
Operating System: macOS Sierra 10.12.6
Stack Trace (if available):

Nicholas Gates · Answer 1 · Tue Apr 24 2018 22:28:01 GMT+0800 (China Standard Time)

I ran into this exact same problem today. My yarn.lock file is up-to-date but the build still reports that it isn't. The command that fails in the build process runs without issue when executed manually.

Frank Schmid · Answer 2 · Wed Apr 25 2018 16:02:11 GMT+0800 (China Standard Time)

Hi @nickelstar ,

Can you check the package.json files that are generated by the plugin during deployment? You should find them after the error happens in the .webpack/dependencies and .webpack/service or .webpack/---function-name-- folders. It might be that a dependency you have is wrongly put into devDependencies and as a consequence is not installed and packaged during the deployment.

The frozen-lockfile error should reliably detect such errors, even if it does not show the exact reason without deeper inspection.

Nicholas Gates · Answer 3 · Wed Apr 25 2018 22:46:19 GMT+0800 (China Standard Time)

Thanks for the helpful info. When I looked at .webpack/dependencies/package.json, I noticed a package listed with an empty version number:

"dependencies": {
    "aws-sdk": "",

This struck me as odd since I don't use aws-sdk and it's not listed in my package.json. So I added it to my package.json via 'yarn add aws-sdk'. After that, the error went away and deployments started working with yarn specified as the webpack packager.

Frank Schmid · Answer 4 · Wed Apr 25 2018 22:50:11 GMT+0800 (China Standard Time)

@nickelstar Good to hear that. I think you can even move the aws-sdk to devDependencies as it will be found but eventually omitted when packaging, because AWS already provides it in theiy Lambda environment. That will reduce your package sizes again.

Francisco Guimarães (chico) · Answer 5 · Thu Apr 26 2018 02:39:57 GMT+0800 (China Standard Time)

@HyperBrain
I have this dependencies:

"dependencies": {
    "moment": "^2.20.1",
    "moment-timezone": "^0.5.14",
    "mongodb": "^2.2.33",
    "mssql": "^4.1.0",
    "object.values": "^1.0.4",
    "papaparse": "^4.3.6",
    "slug": "^0.9.1",
    "source-map-support": "^0.5.0",
    "sprintf-js": "^1.1.1",
    "underscore": "^1.8.3"
  },
  "devDependencies": {
    "aws-sdk": "^2.172.0",
    "babel-core": "^6.26.0",
    "babel-eslint": "^8.0.3",
    "babel-jest": "^22.1.0",
    "babel-loader": "^7.1.2",
    "babel-plugin-transform-runtime": "^6.23.0",
    "babel-preset-env": "^1.6.1",
    "babel-preset-flow": "^6.23.0",
    "eslint": "^4.16.0",
    "eslint-config-airbnb-base": "^12.1.0",
    "eslint-plugin-flowtype": "^2.40.1",
    "eslint-plugin-import": "^2.8.0",
    "eslint-plugin-jest": "^21.7.0",
    "flow-bin": "^0.63.1",
    "flow-typed": "^2.4.0",
    "glob": "^7.1.2",
    "jest": "^22.2.0",
    "serverless": "^1.25.0",
    "serverless-log-forwarding": "^1.1.6",
    "serverless-offline": "^3.16.0",
    "serverless-secrets": "https://github.com/franciscocpg/serverless-secrets.git#e623994662366b6fcfe7cc40f1d85fd19b5029cd",
    "serverless-webpack": "^5.1.1",
    "webpack": "^3.10.0",
    "webpack-node-externals": "^1.6.0",
    "webpack-plugin-copy": "^1.0.1"
  }

and then I see this .webpack/dependencies/package.json generated:

{
  (...)
  "dependencies": {
    "mongodb": "^2.2.33",
    "mssql": "^4.1.0",
    "object.values": "^1.0.4",
    "source-map-support": "^0.5.0",
    "moment": "^2.20.1",
    "lodash": "",
    "slug": "^0.9.1",
    "moment-timezone": "^0.5.14",
    "papaparse": "^4.3.6",
    "underscore": "^1.8.3",
    "sprintf-js": "^1.1.1"
  }
}

This empty lodash ("lodash":"") is causing the Your lockfile needs to be updated, but yarn was run with --frozen-lockfile error for me.

If I remove it yarn install --frozen-lockfile --non-interactive runs successfully. Do you have some idea what is causing this empty lodash?

Also when I run sls package I got a lot of warnings like this: WARNING: Could not determine version of module lodash.

Frank Schmid · Answer 6 · Thu Apr 26 2018 02:45:38 GMT+0800 (China Standard Time)

@franciscocpg That is strange. The lodash in the package.json points out that webpack deteceted it as a primary dependency of your code, but it seems to be missing in your original package.json.

This can happen if you use it directly in your code (by using import or require lodash) but forgot to add it as dependency. If any of your dependencies brings lodash in, you won't recognize the error locally, because it is implicitly in the node_modules folder. NPM might swallow such issues, but yarn - with it's better optimization - will bring it to the surface.
So I'm sure that the root cause is, that you use lodash somewhere but did not add it as explicit dependency.

Francisco Guimarães (chico) · Answer 7 · Thu Apr 26 2018 03:05:02 GMT+0800 (China Standard Time)

@HyperBrain
I've search for all occurrences of "lodash" in my entire project and I'm not using it anywhere.
I guess is some problem related to transitive dependencies and how yarn is resolving it.
Anyway I was able to workaround it by adding "lodash": "^4.17.10" to my dependencies.

Frank Schmid · Answer 8 · Thu Apr 26 2018 06:37:22 GMT+0800 (China Standard Time)

@franciscocpg Yes, you're right. This looks like a bug in the Yarn dependency tree transform (packager/yarn.js) and it seems to fail to retrieve the version numbers of 2nd level dependencies.
Cannot be much. I'll have a look later this week.
However, the case that a version cannot be retrieved can still happen even if the bug is fixed. Would it be wise and safe to assume @latest in these cases (additionally to the printed warning)?

Damon Maria · Answer 9 · Fri Apr 27 2018 12:12:03 GMT+0800 (China Standard Time)

Just struck this too (the original issue with local packages, not the lodash problem).

My local package is called shared. In the project directory it's referred to in files as:

package.json: "shared": "file:../shared/dist/",
yarn.lock: "shared@file:../shared/dist":

But in .webpack/dependencies it's referred to as:

package.json: "shared": "file:../../../shared/dist/",
yarn.lock: "shared@file:../shared/dist":

So the relative link in package.json has been fixed up with the move, but not the lock file. So yarn correctly fails but that might be tricky to correct for...

Frank Schmid · Answer 10 · Fri Apr 27 2018 16:56:29 GMT+0800 (China Standard Time)

@damonmaria Yes, that's exactly the problem. For npm we implemented a lock file rebasing for file references, for yarn I did not, because I missed it.

In theory the same approach as with npm should work (that's an easy one). I'll prepare a PR soon and handle this with high prio so that you can try and test that later.

Frank Schmid · Answer 11 · Fri Apr 27 2018 18:32:22 GMT+0800 (China Standard Time)

I'm working on a solution right now. However, there are 2 cases how local files can be integrated with yarn, so both need a proper handling:

case 1: yarn add my-module ../../somedir/mymodule
case 2: yarn add my-module file:../../somedir/mymodule
which lead to different representations in package.json as well as in yarn.lock

I'll do a first try to rebase any version reference that is file-like. If we encounter issues with that, we could - at least for now - restrict file references to use only the file: notation to make the use case work.

Frank Schmid · Answer 12 · Fri Apr 27 2018 20:00:14 GMT+0800 (China Standard Time)

@franciscocpg I think we should separate the 2 issues contained in this one into 2 issues.

I will fix the "local file references not handled" here and close it with the PR, but I think thet the dependency issue with yarn and transient dependencies is important too and should be solved and discussed properly.

Francisco Guimarães (chico) · Answer 13 · Fri Apr 27 2018 20:13:55 GMT+0800 (China Standard Time)

@HyperBrain
That makes sense.

I'm going to be out on the next 2 weeks so I'm unable to help during this period.

About the yarn and transient dependencies currently we can use this #370 (comment) as a workaround so I think is not too critical.

Frank Schmid · Answer 14 · Fri Apr 27 2018 20:21:37 GMT+0800 (China Standard Time)

@franciscocpg Agree & no problem 😄 . Just tested the attached PR #376 and for me it worked now locally with yarn + file references. I'll add some unit tests, but anyone here (@damonmaria , @mzmiric5 ) can already test if it solves the problem.

Damon Maria · Answer 15 · Sat Apr 28 2018 07:18:12 GMT+0800 (China Standard Time)

I actually worked around this. My use case was sharing a project package across the different parts of the project. Once I struck this (and similar issues with the same shared package in non-Serverless parts of the project ) I instead installed the package (again as a file: URL) in the root of the project. It no longer needs to be moved as each part of the project is built, and when node and Webpack search for it they look up the tree, out of the sub-project into the root of the project. Webpack seems to handle this fine, pulling in the code at build time.

Miso Zmiric · Answer 16 · Sat Apr 28 2018 09:36:23 GMT+0800 (China Standard Time)

@HyperBrain just tested the PR branch with our private package and this fixed the issue we were having. Thanks a lot for a quick fix.

Frank Schmid · Answer 17 · Sat Apr 28 2018 18:26:19 GMT+0800 (China Standard Time)

Released with 5.1.3

Chris Feist · Answer 18 · Sat Nov 03 2018 00:29:30 GMT+0800 (China Standard Time)

I'm experiencing this issue using serverless-webpack 5.2.0. I have private scoped packages, and one of the dependencies that are defined in my private package is showing up without a version in .webpack/dependencies/package.json. However, it appears to be correct in the yarn.lock in that directory. I am whitelisting my scoped packages in the webpack config.

externals: [nodeExternals({
    whitelist: [/^@my-scope/],
  })],

Ken Snyder · Answer 19 · Wed Apr 10 2019 04:53:01 GMT+0800 (China Standard Time)

I am using serverless-webpack version 5.2.0 and also experiencing this error. Running the command yarn install --frozen-lockfile --non-interactive returns with no error and reports that it was already up-to-date. What should I be trying?

Ken Snyder · Answer 20 · Wed Apr 10 2019 04:56:13 GMT+0800 (China Standard Time)

I found that somehow that my dependency on axios which had been there had been removed. Not surprising it failed then but the error really misleading.

Dan Rivett · Answer 21 · Tue Jun 02 2020 21:12:03 GMT+0800 (China Standard Time)

I'd appreciate some advice on this issue to see if it's been resolved. I've been trying to deploy a serverless application that uses a local private package managed by Yarn Workspaces with no success, despite repeated different attempts I still get stuck with:

Error --------------------------------------------------

  yarn install --frozen-lockfile --non-interactive failed with code 1
  error An unexpected error occurred: "https://registry.yarnpkg.com/@private-package/foo: Not found".

(Even though things build outside of building with serverless-webpack).

I'm now about to try a different approach and publish the private package to a private npm repository, but before I start that I'm researching whether this is likely to suceed.

We are using Yarn as our package manager, so would definitely prefer to continue, but I would love to know if that is likely to succeed, and if switching to npm would make it more likely?

Thanks in advance.

Joseph Justus · Answer 22 · Thu Jun 11 2020 15:52:50 GMT+0800 (China Standard Time)

@danrivett, I was able to get serverless-webpack working in yarn monorepos by whitelisting them like so in the webpack config file..

externals: [
    nodeExternals({
      whitelist: ['private_package_1', 'private_package_2']
    })],

christopherchubb · Answer 23 · Wed Jul 29 2020 22:45:51 GMT+0800 (China Standard Time)

@danrivett, I was able to get serverless-webpack working in yarn monorepos by whitelisting them like so in the webpack config file..
externals: [
    nodeExternals({
      whitelist: ['private_package_1', 'private_package_2']
    })],

This worked perfectly for me, except the parameter name is different:

externals: [nodeExternals({ **allowlist**: ['private_package_1', 'private_package_2'], })],

Thank you for showing me the way...