npm gets permissions of node_modules all wrong

Question

npm gets permissions of node_modules all wrong

Cyberuben opened this issue 7 years ago · comments

I'm opening this issue because:

npm is crashing.
npm is producing an incorrect install.
npm is doing something I don't understand.
Other (see below for feature requests):

What's going wrong?

When installing modules in a freshly cloned git repo using npm install on npm version 5.0.0, I get the weird behavior of the node_modules folder having the wrong owner. Using ls -la I can see the following info is returned:

drwxr-xr-x 372  502 dialout 12288 May 28 11:16 node_modules/

When starting the application, I get several dependency errors. When installing the dependency that is reported to be missing, another entry appears.

When doing the same using npm at version 4.6.1, I get the following output:

drwxr-xr-x 373 root root 12288 May 28 11:19 node_modules/

Which is expected behavior, and no packages are missing

Note: The missing dependencies seem to be devDependencies from a few of the packages I depend on. There is something mentioned in the Breaking changes section of the NPM update changelog, but not about these packages not being installed when running npm install

How can the CLI team reproduce the problem?

npm install -g npm@5.0.0
cd /path/to/your/repo
npm install
ls -la

There is no npm-debug.log file, as there are no errors reported.

I'm positive I'm running npm as root:root.

root@node1:~# whoami
root
root@node1:~# groups
root

supporting information:

npm -v prints: 5.0.0
node -v prints: v7.10.0
npm config get registry prints: https://registry.npmjs.org/
Windows, OS X/macOS, or Linux?: Ubuntu 16.04 LTS
Network issues:
- Geographic location where npm was run: OVH, France
- I use a proxy to connect to the npm registry.
- I use a proxy to connect to the web.
- I use a proxy when downloading Git repos.
- I access the npm registry via a VPN
- I don't use a proxy, but have limited or unreliable internet access.
Container:
- I develop using Vagrant on Windows.
- I develop using Vagrant on OS X or Linux.
- I develop / deploy using Docker.
- I deploy to a PaaS (Triton, Heroku).

Alexander Sagen · Answer 1 · Sun May 28 2017 19:36:50 GMT+0800 (China Standard Time)

I just browsed through the npm code for keywords like mkdir, chown child_process to see if the problem would be obvious.

I didn't find anything in the npm code itself, but i found that Node.js uses the following syscall internally to execute mkdir: http://man7.org/linux/man-pages/man2/mkdir.2.html
This syscall creates a folder with the given name and mode, under the same user:group as the process that called it.

The newly created directory will be owned by the effective user ID of the process.

It looks like npm uses the module mkdirp to create folders, which in turn uses the built-in Node.js module fs, which in turn calls the above syscall.

If somehow npm runs under the user:group 502:dialout while creating the folder, this would be the cause of the problem. However, that must be a problem with npm itself and not with @Cyberuben's setup, because he is running npm as root:root.

Kat Marchán · Answer 2 · Mon May 29 2017 12:19:46 GMT+0800 (China Standard Time)

@alexrsagen The place to look is in pacote, combined with https://github.com/npm/npm/blob/latest/lib/config/pacote.js#L45-L47

It's possible I brainfarted about how our permissions-dropping works. I should check with @iarna about what the intentions were, but I thought we were supposed to drop perms to SUDO_UID and SUDO_GID for both node_modules (the issue being reported here), and ~/.npm (which we definitely want to do).

If it's not supposed to drop perms like this, the fix is just to path pacote to not use uid and gid for extractions (but still use them for caching package data).

Kat Marchán · Answer 3 · Mon May 29 2017 12:20:22 GMT+0800 (China Standard Time)

The fact that some packages are missing might well be a separate bug -- it might be useful to see what it is you tried to install that's missing. (I've seen some cases like #16744 where it's literally a completely invalid semver, and I have no idea how npm4 ever installed it).

Kat Marchán · Answer 4 · Mon May 29 2017 13:02:29 GMT+0800 (China Standard Time)

In the meantime: you should be able to blow away SUDO_UID and SUDO_GID and have this work as you're describing. That should work as a stopgap until we get this issue resolved. I just think there's a good chance it was never intended for npm4 to work that way. heh.

Ruben Rutten · Answer 5 · Mon May 29 2017 19:02:54 GMT+0800 (China Standard Time)

The list of dependencies I'm trying to install is as following:

"dependencies": {
    "argon2": "^0.15.0",
    "body-parser": "^1.17.2",
    "config": "^1.26.1",
    "connect-flash": "^0.1.1",
    "connect-redis": "^3.3.0",
    "cookie-parser": "^1.4.3",
    "es6-request": "^2.1.7",
    "express": "^4.15.3",
    "express-session": "^1.15.3",
    "fs-extra": "^3.0.1",
    "gm": "^1.23.0",
    "googleapis": "^19.0.0",
    "kue": "^0.11.5",
    "markdown": "^0.5.0",
    "moment": "^2.18.1",
    "moment-duration-format": "^1.3.0",
    "morgan": "^1.8.2",
    "multer": "^1.3.0",
    "nodemailer": "^4.0.1",
    "numeral": "^2.0.6",
    "nunjucks": "^3.0.1",
    "passport": "^0.3.2",
    "passport-local": "^1.0.0",
    "passport-remember-me": "0.0.1",
    "pg": "^6.2.3",
    "realworks": "0.0.7",
    "realworks-local-media": "0.0.2",
    "realworks-postgres-logger": "0.0.2",
    "realworks-postgres-property": "0.0.1",
    "recaptcha": "^1.2.1",
    "redis": "^2.7.1",
    "serve-favicon": "^2.4.3",
    "speakingurl": "^13.0.0",
    "sql-template-strings": "^2.2.2",
    "striptags": "^3.0.1"
  }

What happens when require()ing fs-extra under npm 5.0.0, is that it complains the module rimraf is not installed. When looking in node_modules, it really isn't there. When installing this module manually (npm install rimraf in project folder) it complains about another package, and if I repeat another and so on

Paul Colomiets · Answer 6 · Fri Jun 02 2017 17:45:57 GMT+0800 (China Standard Time)

I'm also observing user:group of 501:20 when installing by root (in a linux container), without SUDO_UID and SUDO_GID. But bigger problem is that we get the folowing syscall eventually:

4556 chown("/usr/local/lib/node_modules/.staging/invariant-a05eb0c3", 302912903, 1215288109) = -1 EINVAL (Invalid argument)

Which don't look like sensible user and group id.

Reproduced on npm@5.0.1

Ryan Grove · Answer 7 · Sat Jun 03 2017 06:07:42 GMT+0800 (China Standard Time)

I've also been experiencing this bug while trying to use npm5 in a Docker container on CircleCI 2.0. npm is being run by root and neither SUDO_UID nor SUDO_GID are set.

After reading this issue I employed the tried and true debugging method of deleting code at random until something works, and I discovered that the following change to lib/config/pacote.js fixes the problem for me:

--- pacote.js	2017-06-02 15:03:57.000000000 -0700
+++ pacote.fixed.js	2017-06-02 15:04:13.000000000 -0700
@@ -42,9 +42,7 @@
     userAgent: npm.config.get('user-agent')
   }

-  if (ownerStats.uid || ownerStats.gid) {
-    Object.assign(opts, ownerStats)
-  }
+  Object.assign(opts, ownerStats)

   npm.config.keys.forEach(function (k) {
     const authMatch = k[0] === '/' && k.match(

At least in my case, the problem seems to have been that since ownerStats.uid and ownerStats.gid were both 0, opts.uid and opts.gid were never being set.

Michael Hart · Answer 8 · Tue Jun 20 2017 02:01:52 GMT+0800 (China Standard Time)

I'm seeing 501:dialout on /usr/lib/node_modules/npm in my Alpine Docker images too: mhart/alpine-node#94

DanRagle · Answer 9 · Mon Jun 26 2017 23:05:01 GMT+0800 (China Standard Time)

@rgrove's suggestion above also seems on point for what we are experiencing: trying to do global installs while logged in as root results in files set to the uid/gid of the package maintainer (whomever created the source tarball). I.E., since uid/gid is 0/0 in that case, the code rgrove referenced above in the pacote.js file does not apply the uid/gid to the options at all. As a result the options (ultimately) are passed to the extraction process as uid/gid undefined/undefined, and the extraction process then just leaves them set to the ids in the tarball.

rgrove's suggestion in combination with the recent npmc patch for file/directory permissions (80c33cf) seems to correct both the file owners and permissions errors for us when installing global packages as root.

Petter · Answer 10 · Thu Jun 29 2017 02:05:53 GMT+0800 (China Standard Time)

I’m still seeing this in 5.0.4, even though the patch (80c33cf) seems to be in 5.0.4…?

Ryan Grove · Answer 11 · Thu Jun 29 2017 02:14:16 GMT+0800 (China Standard Time)

@haggholm 80c33cf doesn't appear to address the issue I described in #16766 (comment).

I think both changes are needed, as @DanRagle said in #16766 (comment), but I'm not confident enough in my knowledge of npm's internals to say for sure that what I proposed in my comment is the correct fix. It would be helpful if someone with a deeper understanding of npm and pacote could weigh in.

Kat Marchán · Answer 12 · Fri Jun 30 2017 02:09:11 GMT+0800 (China Standard Time)

@rgrove @Cyberuben @mhart @haggholm I've pushed a new canary, npmc@5.0.4-canary.7, which I believe fixes this. Could any of y'all verify this for me? Fingers crossed. :) (npm i -g npmc@latest && npmc i)

Ryan Grove · Answer 13 · Fri Jun 30 2017 02:18:06 GMT+0800 (China Standard Time)

@zkat npmc@5.0.4-canary.7 works like a charm for me. Thanks! 🎉

Michael Hart · Answer 14 · Fri Jun 30 2017 02:19:28 GMT+0800 (China Standard Time)

@zkat doesn't seem to work for me. At least in the sense that /usr/lib/node_modules/npmc is still owned by 501:dialout

You can reproduce with:

docker run mhart/alpine-node sh -c 'npm i -g npmc@latest && npmc i && ls -la /usr/lib/node_modules'

Michael Hart · Answer 15 · Fri Jun 30 2017 02:22:15 GMT+0800 (China Standard Time)

Oh wait, that's probably because npm installed npmc, right...?

Kat Marchán · Answer 16 · Fri Jun 30 2017 02:22:31 GMT+0800 (China Standard Time)

@mhart yes. You need to use npmc itself ;)

Michael Hart · Answer 17 · Fri Jun 30 2017 02:23:26 GMT+0800 (China Standard Time)

Right, duh 😸

Yes, this indeed works:

docker run mhart/alpine-node sh -c 'npm i -g npmc@latest && npmc i -g npm@latest && ls -la /usr/lib/node_modules'

Kat Marchán · Answer 18 · Fri Jun 30 2017 02:27:14 GMT+0800 (China Standard Time)

looks like we're done here. thanks all!

Jono Spiro · Answer 19 · Fri Jun 30 2017 15:31:00 GMT+0800 (China Standard Time)

Any idea when this patch is getting merged to mainline?

Kat Marchán · Answer 20 · Sat Jul 01 2017 11:00:05 GMT+0800 (China Standard Time)

@jspiro the release should be ~ wednesday next week.

Petter · Answer 21 · Sat Jul 08 2017 02:40:53 GMT+0800 (China Standard Time)

I’m still getting bad permissions when installing packages globally. I don’t know precisely what the cases are, but it seems to happen when running gyp; I find myself having to fix the permissions, change to the global node_modules/..., and run node rebuild.

Seweryn Zeman · Answer 22 · Fri Jan 26 2018 22:54:26 GMT+0800 (China Standard Time)

5.6.0 an this error still occurs for npm global builds for root. npm...

Andrew Palmer · Answer 23 · Sat Mar 17 2018 00:14:52 GMT+0800 (China Standard Time)

I'm still seeing global install permissions bugs in 5.7.1.

Niels · Answer 24 · Mon Apr 02 2018 19:43:39 GMT+0800 (China Standard Time)

Like @cadavre; errors on global root install for packages:

root@domoticz:~# npm install -g node-red-admin
npm WARN deprecated node-uuid@1.4.8: Use uuid module instead
/usr/bin/node-red-admin -> /usr/lib/node_modules/node-red-admin/node-red-admin.js

> bcrypt@0.8.7 install /usr/lib/node_modules/node-red-admin/node_modules/bcrypt
> node-gyp rebuild

gyp WARN EACCES user "root" does not have permission to access the dev dir "/root/.node-gyp/8.11.0"
gyp WARN EACCES attempting to reinstall using temporary dev dir "/tmp/.node-gyp"
gyp ERR! configure error
gyp ERR! stack Error: EACCES: permission denied, mkdir '/usr/lib/node_modules/node-red-admin/node_modules/bcrypt/build'
gyp ERR! System Linux 4.9.59-v7+
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /usr/lib/node_modules/node-red-admin/node_modules/bcrypt
gyp ERR! node -v v8.11.0
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: bcrypt@0.8.7 (node_modules/node-red-admin/node_modules/bcrypt):
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: bcrypt@0.8.7 install: `node-gyp rebuild`
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: Exit status 1

+ node-red-admin@0.1.3
updated 1 package in 9.898s

root@domoticz:~# npm -v
5.6.0

root@domoticz:~# node -v
v8.11.0

root@domoticz:~# ls -ahl /root/.node-gyp/
total 12K
drwxr-xr-x 3 root root    4.0K Mar 29 20:22 .
drwx------ 7 root root    4.0K Mar 29 20:22 ..
drwxr-xr-x 3  502 dialout 4.0K Mar 29 20:22 8.11.0

Babak · Answer 25 · Mon Apr 02 2018 23:08:49 GMT+0800 (China Standard Time)

@cadavre I had the same issue. Updating to npm 5.8.0 didn't solve my problem.

Although, the following command saved me:

npm config set unsafe-perm=true

or just

npm i npm -g --unsafe-perm=true

Andreas Urbanski · Answer 26 · Wed Jun 13 2018 00:24:13 GMT+0800 (China Standard Time)

Wow, this is still an issue in 6.1.0 Sadly for me the only satisfying resolution was to switch to yarn.