OSXFS specific issue

Question

OSXFS specific issue

alcohol opened this issue 7 years ago · comments

HOST

uname -a
Darwin macbookpro.local 17.2.0 Darwin Kernel Version 17.2.0: Fri Sep 29 18:27:05 PDT 2017; 
  root:xnu-4570.20.62~3/RELEASE_X86_64 x86_64 i386 MacBookPro11,4 Darwin

docker --version
Docker version 17.11.0-ce, build 1caf76c

FILES

package.json for reproduction:

{
  "name": "test",
  "private": true,
  "devDependencies": {
    "cucumber": "2.3.1"
  }
}

Dockerfile for reproduction:

FROM alpine:3.7

RUN apk add --no-cache --update git tini nodejs=8.9.1-r0

WORKDIR /workspace

ENTRYPOINT ["/sbin/tini", "--"]

REPRO

Repro steps host:

docker build --tag test-image .
docker run --rm -it -e HOME=/tmp -w /workspace -v $(pwd):/workspace test-image sh

Repro steps inside container:

npm install --verbose

OUTCOME

npm info it worked if it ends with ok
npm verb cli [ '/usr/bin/node', '/usr/bin/npm', 'install', '--verbose' ]
npm info using npm@5.5.1
npm info using node@v8.9.1
npm verb npm-session 8b372084f9346ee1
npm info lifecycle test@~preinstall: test@
npm http fetch GET 200 https://registry.npmjs.org/cucumber 128ms

...snipped...

npm info linkStuff cucumber@2.3.1
npm verb linkBins cucumber@2.3.1
npm verb linkBins [ { 'cucumber.js': './bin/cucumber.js',
npm verb linkBins     'cucumber-js': './bin/cucumber.js',
npm verb linkBins     cucumberjs: './bin/cucumber.js' },
npm verb linkBins   '/workspace/node_modules/.bin',
npm verb linkBins   false ]
npm verb linkMans cucumber@2.3.1
npm verb unlock done using /tmp/.npm/_locks/staging-afa1b9ac6beb89c7.lock for /workspace/node_modules/.staging
npm verb stack Error: ENOENT: no such file or directory, chmod '/workspace/node_modules/cucumber/bin/cucumber.js'
npm verb cwd /workspace
npm verb Linux 4.9.60-linuxkit-aufs
npm verb argv "/usr/bin/node" "/usr/bin/npm" "install" "--verbose"
npm verb node v8.9.1
npm verb npm  v5.5.1
npm ERR! path /workspace/node_modules/cucumber/bin/cucumber.js
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall chmod
npm ERR! enoent ENOENT: no such file or directory, chmod '/workspace/node_modules/cucumber/bin/cucumber.js'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent
npm verb exit [ -2, true ]

npm ERR! A complete log of this run can be found in:
npm ERR!     /tmp/.npm/_logs/2017-12-06T16_40_19_127Z-debug.log

EXPECTED

Cucumber gets installed.

NOTES

Also applies to 1.3.3, 2.1.0, 2.2.0, 2.3.0 (did not test lower than this).

Version 3.1.0 does not seem to have this issue.

Evgene Ivashkevich · Answer 1 · Wed Jan 10 2018 16:58:58 GMT+0800 (China Standard Time)

The same for me on linux machine. But sometimes it works.

Nicolas Bigaouette · Answer 2 · Wed Jan 17 2018 00:57:53 GMT+0800 (China Standard Time)

Same kind of error here working with serverless (1.25.0).

uname -a
Darwin m.local 17.3.0 Darwin Kernel Version 17.3.0: Thu Nov  9 18:09:22 PST 2017; root:xnu-4570.31.3~1/RELEASE_X86_64 x86_64
docker --version
Docker version 17.12.0-ce, build c97c6d6

Minimal (non-working) example using docker:

mkdir docker-issue-2296
cd docker-issue-2296
docker run -it --rm -v $(PWD):/cwd -w /cwd node:9.4.0-alpine npm init

Accepting all default values, resulting in following package.json:

{
  "name": "cwd",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC"
}

Then install fails:

docker run -it --rm -v $(PWD):/cwd -w /cwd node:9.4.0-alpine npm install --verbose serverless
npm info it worked if it ends with ok
npm verb cli [ '/usr/local/bin/node',
npm verb cli   '/usr/local/bin/npm',
npm verb cli   'install',
npm verb cli   '--verbose',
npm verb cli   'serverless' ]
npm info using npm@5.6.0
npm info using node@v9.4.0
npm verb npm-session ae591a1f8d148d0f
npm http fetch GET 200 https://registry.npmjs.org/serverless 138ms
[...]
npm info linkStuff serverless@1.25.0
npm verb linkBins serverless@1.25.0
npm verb linkBins [ { serverless: './bin/serverless',
npm verb linkBins     slss: './bin/serverless',
npm verb linkBins     sls: './bin/serverless' },
npm verb linkBins   '/cwd/node_modules/.bin',
npm verb linkBins   false ]
npm verb linkMans serverless@1.25.0
npm verb unlock done using /root/.npm/_locks/staging-09696d3793f12899.lock for /cwd/node_modules/.staging
npm WARN cwd@1.0.0 No description
npm WARN cwd@1.0.0 No repository field.

npm verb stack Error: ENOENT: no such file or directory, chmod '/cwd/node_modules/serverless/bin/serverless'
npm verb cwd /cwd
npm verb Linux 4.9.60-linuxkit-aufs
npm verb argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "--verbose" "serverless"
npm verb node v9.4.0
npm verb npm  v5.6.0
npm ERR! path /cwd/node_modules/serverless/bin/serverless
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall chmod
npm ERR! enoent ENOENT: no such file or directory, chmod '/cwd/node_modules/serverless/bin/serverless'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent
npm verb exit [ -2, true ]

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-01-16T16_44_31_237Z-debug.log

The node_modules directory is empty:

tree -a node_modules/
node_modules/
└── .bin
    ├── esparse -> ../esprima/bin/esparse.js
    ├── esvalidate -> ../esprima/bin/esvalidate.js
    ├── js-yaml -> ../js-yaml/bin/js-yaml.js
    ├── json-refs -> ../json-refs/bin/json-refs
    ├── loose-envify -> ../loose-envify/cli.js
    ├── mime -> ../mime/cli.js
    ├── mkdirp -> ../mkdirp/bin/cmd.js
    ├── raven -> ../raven/bin/raven
    ├── rc -> ../rc/cli.js
    ├── rimraf -> ../rimraf/bin.js
    ├── seek-bunzip -> ../seek-bzip/bin/seek-bunzip
    ├── seek-table -> ../seek-bzip/bin/seek-bzip-table
    ├── semver -> ../semver/bin/semver
    ├── serverless -> ../serverless/bin/serverless
    ├── sha.js -> ../sha.js/bin.js
    ├── sls -> ../serverless/bin/serverless
    ├── slss -> ../serverless/bin/serverless
    ├── tabtab -> ../tabtab/bin/tabtab
    └── which -> ../which/bin/which

1 directory, 19 files

Uploading:

a gzipped version of the full log as serverless.log.gz: serverless.log.gz
a gzipped version of npm's own log:
2018-01-16T16_54_05_430Z-debug.log.gz

Nicolas Bigaouette · Answer 3 · Wed Jan 17 2018 01:09:26 GMT+0800 (China Standard Time)

Interestingly enough, it works with a different docker image (using node v8.9.4):

rm -fr node_modules package-lock.json
docker run -it --rm -v $(PWD):/cwd -w /cwd node:8.9.4-alpine npm install serverless

> spawn-sync@1.0.15 postinstall /cwd/node_modules/spawn-sync
> node postinstall


> serverless@1.25.0 postinstall /cwd/node_modules/serverless
> node ./scripts/postinstall.js

npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN cwd@1.0.0 No description
npm WARN cwd@1.0.0 No repository field.

+ serverless@1.25.0
added 307 packages in 53.39s

EDIT: Talked to quickly; it now fails!

Pedro · Answer 4 · Fri Apr 13 2018 12:05:40 GMT+0800 (China Standard Time)

I'm getting the same thing using d3 with docker for mac:

test | npm ERR! path /app/node_modules/d3-dsv/bin/dsv2dsv
test | npm ERR! code ENOENT
test | npm ERR! errno -2
test | npm ERR! syscall chmod
test | npm ERR! enoent ENOENT: no such file or directory, chmod '/app/node_modules/d3-dsv/bin/dsv2dsv'
test | npm ERR! enoent This is related to npm not being able to find a file.
test | npm ERR! enoent

Sometimes it fails on a different file:

 test | npm ERR! enoent ENOENT: no such file or directory, chmod '/app/node_modules/d3-dsv/bin/json2dsv'

npm ERR! enoent ENOENT: no such file or directory, chmod '/app/node_modules/d3-dsv/bin/dsv2json'

Jorge Marin · Answer 5 · Wed Apr 18 2018 19:09:08 GMT+0800 (China Standard Time)

Same happens with cucumber@^3.2.0 in a Debian container running in MacOS High Sierra 10.13.4 with Docker version 18.03.0-ce

# npm install cucumber@^3.2.0
npm WARN enoent ENOENT: no such file or directory, open '/myVolume/projects/test/package.json'
npm WARN test No description
npm WARN test No repository field.
npm WARN test No README data
npm WARN test No license field.

npm ERR! path /myVolume/projects/test/node_modules/cucumber/bin/cucumber.js
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall chmod
npm ERR! enoent ENOENT: no such file or directory, chmod '/myVolume/projects/test/node_modules/cucumber/bin/cucumber.js'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-04-18T11_04_59_446Z-debug.log

Docker info:

$ docker info
Containers: 6
 Running: 1
 Paused: 0
 Stopped: 5
Images: 21
Server Version: 18.03.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.87-linuxkit-aufs
Operating System: Docker for Mac
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952GiB
Name: linuxkit-025000000001
ID: FXOM:7AVH:WCUR:ISF3:ZZU7:5J3U:H3LN:EQ46:B65I:PTRW:7SGC:M7AC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 42
 Goroutines: 61
 System Time: 2018-04-18T11:07:45.0285239Z
 EventsListeners: 2
HTTP Proxy: docker.for.mac.http.internal:3128
HTTPS Proxy: docker.for.mac.http.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Evgene Ivashkevich · Answer 6 · Wed May 23 2018 14:57:34 GMT+0800 (China Standard Time)

try to use "npm install --unsafe-perm"

Salomon Smeke · Answer 7 · Fri May 25 2018 04:24:35 GMT+0800 (China Standard Time)

+1 here. npm install --unsafe-perm didn't help.

:consistent didn't help.

gurucleff · Answer 8 · Tue Aug 14 2018 04:43:37 GMT+0800 (China Standard Time)

any updates on this issue?

Tom Widmer · Answer 9 · Fri Aug 24 2018 21:25:28 GMT+0800 (China Standard Time)

I have a workaround for this: as well as the volume mount of the local project, additionally mount a named volume to the node_modules folder, so it isn't shared with the host (it isn't really desirable to share node_modules between the OS and the container anyway, since the OS is different). E.g. in my docker-compose file I have something like:

services:
  my-service:
    image: my-service:local
    build: ../my-service
    volumes:
      - ../my-service:/var/app
      - my_service_node_modules:/var/app/node_modules
volumes:
  my_service_node_modules:

Should also help with performance.

George Song · Answer 10 · Fri Aug 24 2018 21:36:43 GMT+0800 (China Standard Time)

@tomwidmer When I tried this, I saw some weird behaviors when node_modules is also present in the host. Are you operating exclusively using Docker containers?

Tom Widmer · Answer 11 · Tue Aug 28 2018 01:12:51 GMT+0800 (China Standard Time)

@gsong I mostly just operate in the container - I use docker-compose exec <service> bash before doing any npm commands. However, I do run npm install outside the container to get the node_modules in place outside the container to help my IDE. Another option would be to copy node_modules out of the container and into the host folder manually with docker cp after installing/updating any modules.

I'd prefer a fix to the underlying problem of course.

cydergoth · Answer 12 · Tue Sep 11 2018 00:22:54 GMT+0800 (China Standard Time)

+1 Same issue

Darwin JCrisp-MBP-TX.local 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64

Docker version 18.06.0-ce, build 0ffa825

docker-robot · Answer 13 · Sun Dec 09 2018 10:00:24 GMT+0800 (China Standard Time)

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

Xingchen Hong · Answer 14 · Sun Dec 09 2018 15:53:15 GMT+0800 (China Standard Time)

@docker-for-desktop-robot WTH?

John Yani · Answer 15 · Fri Dec 21 2018 03:17:28 GMT+0800 (China Standard Time)

Didn't understand what are those Mac users are complaining about. Got a MacBook, now I have this issue too :(

And I'm using yarn, not npm.
It looks like osxfs return NOENT if it can't keep up with firehose of files created by npm/yarn.

Ashwini Kumar · Answer 16 · Sun Nov 10 2019 22:18:34 GMT+0800 (China Standard Time)

I am also running into the issue while installing d3, as @pcambra said. Any workaround?

Salomon Smeke · Answer 17 · Tue Nov 12 2019 07:30:07 GMT+0800 (China Standard Time)

Workaround for me was to mount a NPM volume that is container-only. Install NPM there. Then copy the directory over to its final location within another volume that is container-only.

Basically: dont sync it.

Ashwini Kumar · Answer 18 · Thu Nov 14 2019 02:12:11 GMT+0800 (China Standard Time)

/remove-lifecycle stale

docker-robot · Answer 19 · Tue Feb 11 2020 09:00:13 GMT+0800 (China Standard Time)

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

docker-robot · Answer 20 · Fri Jun 26 2020 11:06:17 GMT+0800 (China Standard Time)

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked