Giters

yarnpkg / yarn

The 1.x line is frozen - features and bugfixes now happen on https://github.com/yarnpkg/berry

Home Page:https://classic.yarnpkg.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

yarn install fails with `ENOENT: no such file or directory` occasionally

NiGhTTraX opened this issue · comments

commented

Running yarn install as part of a build step for a Docker image based on node:7 fails on Travis CI with ENOTEMPTY, EEXISTS errors. It always seems to error on the webdriverio package.

yarn install v0.19.1
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/webdriverio/-/webdriverio-4.6.2.tgz: ENOENT: no such file or directory, open '/usr/local/share/.cache/yarn/npm-webdriverio-4.6.2-dd095ee618896a21c8f1b9d4278736d85a64ca0f/lib/protocol/timeouts.js'".

When Travis runs yarn install as part of the install phase it works just fine. The error only happens when building a Docker image.

Repo which reproduces this issue.

node:7
OS: Docker + Travis CI
yarn: 0.19.1
package.json
yarn.lock

I've tried installing yarn both with npm install -g and with apt and both methods cause the failure on Travis.

Weirdly enough, the image builds successfully on my local machine which runs Ubuntu 16.04.1 LTS with Docker version 1.13.0, build 49bf474.

commented

Interesting, so it only fails on Travis, but works when testing locally? That's very weird given Docker is supposed to ensure the environment is consistent.

commented

@Daniel15 I know right...

I've downgraded node to version 6 and it still fails on Travis. I've added the --verbose flag to yarn install and all I got was

verbose Performing "GET" request to "https://registry.yarnpkg.com/spawn-wrap/-/spawn-wrap-1.3.4.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/yargs/-/yargs-6.6.0.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/yargs-parser/-/yargs-parser-4.2.1.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/fibers/-/fibers-1.0.15.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/selenium-standalone/-/selenium-standalone-5.11.2.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/tcp-port-used/-/tcp-port-used-0.1.2.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/babel-runtime/-/babel-runtime-5.8.38.tgz".
verbose Error: ENOTEMPTY: directory not empty, rmdir '/usr/local/share/.cache/yarn/npm-webdriverio-4.6.2-dd095ee618896a21c8f1b9d4278736d85a64ca0f/lib/protocol'
    at Error (native)
error An unexpected error occurred: "ENOTEMPTY: directory not empty, rmdir '/usr/local/share/.cache/yarn/npm-webdriverio-4.6.2-dd095ee618896a21c8f1b9d4278736d85a64ca0f/lib/protocol'".

I am open to ideas on how to debug this.

commented

Downgrading to yarn 0.18.1 seemed to fix this for me. Seems like 0.19 might have a regression; see #1834

commented

I also have this problem with yarn 0.23.3, it's happening not when building an image but simply when running some CI.
The error is the following:

$ time yarn --frozen-lockfile
yarn install v0.20.3
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "ENOTEMPTY: directory not empty, rmdir '/builds/linagora/petals-cockpit/yarncache/npm-@angular/core-4.0.0-beta.8-8d9c8a64e7c26ff7208404e716deea94bb509cd7/src'".
info If you think this is a bug, please open a bug report with the information provided in "/builds/linagora/petals-cockpit/frontend/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

real	0m9.812s
user	0m7.596s
sys	0m0.932s

I think there may be some strange way of doing remove of files…

Important point: the cache was empty!

commented

And on my machine, if I try to repro, I get this:

yarn install v0.20.3
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "http://docker0.gso.lan:8081/repository/npm/@angular/core/-/core-4.0.0-beta.8.tgz: EEXIST: file already exists, mkdir '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-beta.8-8d9c8a64e7c26ff7208404e716deea94bb509cd7/src/metadata'".
info If you think this is a bug, please open a bug report with the information provided in "/home/vnoel/Linagora/Petals/dev/git/petals-cockpit-new/frontend/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
commented

And with yarn 0.21.2:

yarn install v0.21.2
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "http://docker0.gso.lan:8081/repository/npm/@angular/core/-/core-4.0.0-beta.8.tgz: ENOENT: no such file or directory, lstat '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-beta.8-8d9c8a64e7c26ff7208404e716deea94bb509cd7/bundles/core.umd.js'".
info If you think this is a bug, please open a bug report with the information provided in "/home/vnoel/Linagora/Petals/dev/git/petals-cockpit-new/frontend/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

That's terrible!

commented

And I concur with @twooster about 0.18.1 working ok!

commented

@Daniel15 it doesn't work locally either. Actually, it NEVER work when the cache is empty for me!

commented

@victornoel the recent error could be #2714

commented

@bestander I did try 0.19.1 at the time and it didn't work…

I retried, and now the bug:

  • does not appear with an empty cache, but does appear in the following case (I really hope it is reproducible…):
    • rm -rf the yarn cache
    • clone https://gitlab.com/linagora/petals-cockpit.git
    • checkout 5f31ccb4b2357201baa50539b30702cffceb6992
    • run yarn in the frontend directory
    • checkout master
    • run yarn again in the frontend directory
    • I get: error An unexpected error occurred: "http://docker0.gso.lan:8081/repository/npm/@angular/core/-/core-4.0.0-rc.1.tgz: ENOENT: no such file or directory, utime '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/@angular/core/testing.js'". (I am using my own registry, but the same happens without it)
  • does appear with yarn 0.21.2, 0.19.1 but not with 0.18.2

So I don't think it is the same, let's hope you can reproduce it at least…

commented

(actually, I just tried again and reproduced the bug with an empty cache and yarn 0.21.2 while it wasn't the case before, maybe there is another file somewhere else that is the source of this problem, and that is not in the cache?)

commented

@bestander I will still test yarn as soon as #2744 is available as a nightly build :)

commented

Ping me if I can help.
The best action is to send a PR with a broken (and skipped) e2e test.

commented

@bestander well, nope, I still get errors such as:


➜  frontend git:(master) ✗ yarn
yarn install v0.22.0-20170227.1509
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/@angular/core/-/core-4.0.0-rc.1.tgz: ENOENT: no such file or directory, lstat '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/typings/src/facade/lang.d.ts'".

or:

➜  frontend git:(master) ✗ yarn
yarn install v0.22.0-20170227.1509
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/typescript/-/typescript-2.2.1.tgz: ENOENT: no such file or directory, lstat '/home/vnoel/.cache/yarn/npm-typescript-2.2.1-4862b662b988a4c8ff691cc7969622d24db76ae9/lib/typescriptServices.js'".

I will see if I can make an e2e test.

commented

@bestander anyway I can get a complete stacktrace of the error?

I only see this in the yarn-error.log:

Trace: 
  Error: http://docker0.gso.lan:8081/repository/npm/@angular/core/-/core-4.0.0-rc.1.tgz: ENOENT: no such file or directory, lstat '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/@angular/core.es5.js'
      at Error (native)

That's a bit useless :)

commented

the detailled error is:

{ Error: http://docker0.gso.lan:8081/repository/npm/@angular/core/-/core-4.0.0-rc.1.tgz: ENOENT: no such file or directory, lstat '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/@angular/core.js'
    at Error (native)
  errno: -2,
  code: 'ENOENT',
  syscall: 'lstat',
  path: '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/@angular/core.js',
  fstream_type: 'File',
  fstream_path: '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/@angular/core.js',
  fstream_class: 'FileWriter',
  fstream_stack: 
   [ '/home/vnoel/Linagora/Petals/dev/git/yarn/node_modules/fstream/lib/writer.js:285:28',
     '/home/vnoel/Linagora/Petals/dev/git/yarn/node_modules/graceful-fs/polyfills.js:284:29',
     'FSReqWrap.oncomplete (fs.js:123:15)' ] }

Not sure exactly what to do with this… it's happening at package-fetcher.js, line 56 exactly,  I have trouble finding the source though…

commented

It seems stupid, but I feel like it fails only when my networked npm mirror (a sonatype nexus in my company) has mirrored the @angular/core artefact. If it hasn't, things go well and then it fails on another artefact that is already mirrored (typescript in this case).

If I remove the artefact from the nexus mirror by hand, it works!

So… it is a bit like yarn can't keep up if things arrive too fast ^^ because when I use the normal npm registry without using my mirror, things usually goes well (we have a slow internet connection).
And it would explain why it often fails on CI systems because they usually have very fast internet connections…

That's a bit of stretch to conclude that, but it could help find the origin of the problem.
WDYT @bestander?

For the record, I think the error is emanating from tar.Extract step in the fetching pipeline, but I'm not totally sure ^^

commented

Thanks for researching more, @victornoel, you might be on something here.

commented

I can repro scenario from #2629 (comment).
Looking into it.

I get

error An unexpected error occurred: "https://registry.yarnpkg.com/typescript/-/typescript-2.2.1.tgz: ENOENT: no such file or directory, lstat '/Users/bestander/Library/Caches/Yarn/npm-typescript-2.2.1-4862b662b988a4c8ff691cc7969622d24db76ae9/lib/typescriptServices.js'".

However if I just try yarn install again and again it will eventually finish the installation.
Looks like exploding the .tgz file ends with error.

Update:

  • the .tgz seems fine, I can unzip manually the one that fails during fetch phase
  • I wonder if tar package is throwing this error for some reason, could it be concurrency?
commented

Some help investigating why untaring those few dependencies (in my case typescript and angular-core) causes error is welcome.
Concurrency? Bug in https://github.com/npm/node-tar?

commented

@victornoel, can you reproduce the bug with yarn install --network-concurrency 1?

commented

@bestander with --network-concurrency 1 the bug does not appear (while without it, it appears every time).
But what is the default value of this parameter? Whichever value I choose for it (1, 2, 4, 8), it works, while if I don't put it at all, it fails…

commented

The default is 15, I can repro the issue with concurrency 15 with a clean checkout of https://gitlab.com/linagora/petals-cockpit.git#075bac4c54fee466568c000c7ffe8025f593e212.

commented

Excellent news! One step further toward a solution AND a workaround :)

commented

Some results.

TL;DR I am out of ideas how to properly fix it for good, this needs some deeper Node.js knowledge.

  1. Network can be eliminated from possible issues.
    I've setup offline mirror for the .tgz files in yarn.lock and can reproduce the issue with packages installed from disk.

The issue is in unzip/untar stream in tarball-fetcher code.

  1. I tried a different library that extracts tar - https://github.com/mafintosh/tar-fs vs current https://github.com/npm/node-tar/. They both fail the same way.
    Going a bit deeper - exceptions are happening in node when doing multiple mkdirp operations
Error: ENOENT: no such file or directory, chmod '/Users/bestander/Library/Caches/Yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/typings/src/di/injector.d.ts'
  errno: -2,
  code: 'ENOENT',
  syscall: 'chmod',
  path: '/Users/bestander/Library/Caches/Yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/typings/src/di/injector.d.ts' }

I think core-4.0.0 and typescript-2.2.1 fail because they have quite a few files and deep folder structures, and they fail to install while making many concurrent mkdir/copy operations.

Every time there is a different syscall that fails: chmod, rmdir, mkdir, lstat, utime.

And it is not something obvious in the libraries' code.

  1. Fails the same on Node 4, 6 and 7.

  2. I could not reproduce the error with concurrency set to 8, so I'll send a PR to reduce default network concurrency.

  1. I was wondering how concurrency affects installation speed.

5.1. Using offline-mirror (no download), on my MBPro 13", clean cache and using node-tar to unzip files.
Concurrency 12 - fails
Concurrency 8 - 18 seconds
Concurrency 4 - 18 seconds
Concurrency 2 - 21 seconds

5.2. Using offline-mirror (no download), on my MBPro 13", clean cache and using tar-fs to unzip files.
Concurrency 12 - 15 seconds
Concurrency 8 - 15 seconds
Concurrency 4 - 17 seconds
Concurrency 2 - 18 seconds

5.3. Downloading packages from internet, on my MBPro 13", clean cache and using tar-fs to unzip files.
Concurrency 12 - failed once
Concurrency 8 - 21 seconds
Concurrency 4 - 23 seconds
Concurrency 2 - 34 seconds

Looks like setting concurrency to 8 is safe enough, also it makes sense to switch the tar library.
I'll follow up with a PR.

commented

A proper way to fix this would be forking https://github.com/mafintosh/tar-fs and doing smarter fs operations, e.g. using mkdir for every folder only once

commented

tar-fs maintainer seems to be active, maybe we could open an issue there and see what they know/propose about this?

commented

@victornoel, would you do that please?

commented

I ran into this error message in a somewhat similar scenario when testing out yarn on my jenkins' build agents.

Do you know what the conditions required to trigger this bug are? I'd like to replace my build system's npm calls with yarn for speed, but if I have to disable concurrency I'm worried it might negate any bonus there.

commented

@ProdigySim, as explained in #2829 (which was merged in yarn master), reducing the network concurrency has no big effect on the performance of yarn. You can simply set it to 8 and it should be alright. Anyway, even when downloading 8 dependencies at a time, I'm not sure most drive is going to follow in throughput so you won't loose much for sure :)

commented

@victornoel Thanks for the information. I'm not sure just reducing --network-concurrency will be enough in my case, since we would also run multiple instances of yarn in parallel.

I can repro this issue even with --network-concurrency 1, but maybe I should open a separate issue for that?

Using the same test repo you gave above:

#!/bin/bash
set -x # echo commands

# Clear yarn cache
rm -rf $(yarn cache dir)

# Clone the repo into two separate spots
git clone https://gitlab.com/linagora/petals-cockpit.git repo1
git clone https://gitlab.com/linagora/petals-cockpit.git repo2

# Run yarn on both in parallel
cd repo1/frontend && yarn --network-concurrency 1 &
cd repo2/frontend && yarn --network-concurrency 1 &

This nets me the error every time (4 for 4 so far)

error An unexpected error occurred: "https://registry.yarnpkg.com/@angular/core/-/core-4.0.0-rc.2.tgz: 
ENOENT: no such file or directory, lstat '/Users/<snip>/Library/Caches/Yarn/npm-@angular/core-4.0.0-rc.2-59535050e5d0e6141417186eee571296f8e9c3d0/@angular/core.es5.js'".

On yarn 0.21.3, node v4.5.0, OSX 10.11.6

So far I've been able to work around this issue by only including yarn on build jobs that will never run in parallel, or use completely different sets of packages, but even then I'm not sure it will be safe--hence asking about root conditions for this bug.

commented

@ProdigySim
This is a separate, but related, issue caused by Yarn's global download cache. A workaround is to create a different cache for each directory.

You can still run with --network-concurrency 8. (I actually have no issues with unlimited network concurrency.)

More context here.

commented

@bestander surprisingly, today, the problem reappeared (triggered by a tar for a new version of angular ^^) even with the network concurrency at 8, but only on my CI… I moved it to 2 and it works (and I don't really care if it takes a few more seconds to download dependencies, so it's ok for now).
We don't seem to get feedbacks from the tar-fs project… who else could we contact for help on that?

commented

having this issue as well on my Travis builds for OS X. I've clearned the cache and set network concurrency but nothing has helped.

commented

@kevingelion to which value did you set the network concurrency? be drastic and set something like 2 to see if the problem is this one :)

commented

@victornoel I set it to 1 and 2, and both options resulted in a fail. I used yarn --mutex network and no dice either.

commented

@bestander the following hack fixes (edit: NOT) the problem:

diff --git a/src/util/request-manager.js b/src/util/request-manager.js
index e0e134a2..995dac69 100644
--- a/src/util/request-manager.js
+++ b/src/util/request-manager.js
@@ -214,8 +214,7 @@ export default class RequestManager {
     }, params.headers);
 
     const promise = new Promise((resolve, reject) => {
-      this.queue.push({params, resolve, reject});
-      this.shiftQueue();
+      this.execute({params, resolve, reject});
     });
 
     // we can't cache a request with a processor

Obviously it is not a fix, it completely bypasses the request manager and its queuing system, but it shows that the problem is coming from this subsystem.

commented
commented

whoups not, it doesn't :D but it does improve things a bit

commented

Sorry for the false positive, I was too eager to report my findings :)

It changes things because before I could run yarn many times and it would never succeed downloading the angular-core dependency or the typescript (always these ones), but there it failed the first time, and succeeded the second time, and I forgot to remove the cache in between my tries so I thought it was working.

commented

Well it depends, now sometimes it works, sometimes it does not (with the hack, so it's not a total miss or my internet connection is too slow right now...)

commented

I'm running into this too in our CI builds. After a lot of testing, I've also finally been able to reproduce locally.

Again, it does sometimes work, but it fails often with either of the following errors (which makes it seem like there's some kind of race condition somewhere):

  • ENOENT: no such file or directory, lstat 'cache/directory/some-file'
  • EEXIST: file already exists, mkdir 'package-name'

I've isolated it to one package that we install directly from a private repository on GitHub. Interestingly enough, the package being referenced in the error message is always a dependency of this package (and it is always another package we install directly from GitHub, though not a private repository). So it seems like one repro case is installing packages from private GitHub URLs who have subdependencies that are also installed from GitHub repositories (not necessarily private).

Not sure if this helps at all...I'm happy to try to help out in any way I can!

Edit: Not sure if this is helpful or not, but the top level package is listed in the format of "git+ssh://git@github.com/org/package.git#v1.0.0", and in the error, the subdependency being downloaded looks like it's being downloaded over https with a url of "https://codeload.github.com/org/package/tar.gz/ljasdf08i234098aifj".

commented

I am investigating this a bit more.
Trying to build a standalone script for concurrent tar-fs extracts but I tend to think that the problem is in tar file breaking during download.

commented

Found it, Doh.

In the example from #2629 (comment) Yarn has duplicate packages that are getting downloaded and extracted in parallel.
The duplicated ones are @angular/core/-/core-4.0.0-rc.1 and typescript/-/typescript-2.2.1.tgz.

With high concurrency we just happen to do concurrent extraction into the same cache folder.
I'll investigate why Yarn does not deduplicate those two pacakages and send a fix.

No magic on OS or tar extraction level.

commented

haha, good job @bestander, glad we finally found the problem!

commented

Awesome work @bestander 🎉! Running into both #3090 and #3106 was what blocked us from using Yarn.

commented

thanks!

commented

I had this problem installing prop-types module. Each time I tried to install it would ENOENT on a different filename. For me the problem went away after installing npm 5.0.2

$ yarn add prop-types
yarn add v0.21.3
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/prop-types/-/prop-types-15.5.10.tgz: ENOENT: no such file or directory
....

$ npm -g install npm

# whoops, looks like npm installed itself to different location than apt-get did
$ npm -v 
3.5.2

# remove the cached link from shell so the right version can surface
$ hash -d npm
$ npm -v
5.0.2

$ yarn add prop-types
... properly installs prop-types as expected
commented

@skylize That's likely to be coincidental - Yarn doesn't use the npm client at all, so the version of npm does not affect execution of Yarn at all.

commented

This is causing my Travis builds to fail almost every single time, with a few different packages. Is there a solution yet?
error An unexpected error occurred: "https://registry.yarnpkg.com/apollo-client/-/apollo-client-1.8.0.tgz: ENOENT: no such file or directory, utime '/var/lib/jenkins/.cache/yarn/v1/npm-apollo-client-1.8.0-3b5d1976a06a0f82b2fc66fe71754868193dadb9/flow-typed/npm/webpack_vx.x.x.js'".

commented

@Redmega
Same here, but this works:

yarn install --network-concurrency 1
commented
commented

@victornoel I am using v0.27.5 on the jenkins machine, same as my local.

commented

Removing the yarn.lock file and yarn install fixed the problem for me.

commented

This also causes my Jenkins builds to fail occasionally. Usually it works after a second try but will fail again later.

commented

@ajcrites @Redmega @headione @benmerckx you should open another issue if you are experiencing this kind of problem. This issue has been fixed for sure, so your problem must be a different one, even if it shows some similar symptoms.
I'm pretty sure there is more chance for your problem to be solved if you open another issue :)

commented

We have the same problem, doing parallel builds of packages in Jenkins with Node 8.5. We currently need to stick to 0.27.5, until 1.0.2 is released fixing another bug. But thanks for your support and work anyway :)

commented

@floric I'm getting the same issue with the same context (Jenkins + Parallel) with node 8.9.4, has your issue been resolved ?

Edit: I'll try to use 8.11.1 to see if it includes a latest version of yarn without the bug.

commented

@Niceplace you may wanna try the --mutex option: https://yarnpkg.com/en/docs/cli#toc-concurrency-and-mutex

We have plans for adding better per-package locking to avoid this soon.

commented

I'm having intermittent errors with both ENOENT: no such file or directory, chmod and ENOENT: no such file or directory, lstat trying to run yarn --mutex=network on the root of a monorepo with Yarn workspace enabled...

It doesn't seem to be consistent, I get either one or the other randomly. (1.6.0 and node 8.11.1 and 9.11.1)

Specifically, the errors are:

error An unexpected error occurred: "ENOENT: no such file or directory, lstat '/Users/federicozivolo/test/packages/foobar/node_modules/detect-port-alt'".

and

error An unexpected error occurred: "ENOENT: no such file or directory, chmod '/Users/federicozivolo/test/packages/foobar/node_modules/jest/node_modules/.bin/jest'".
commented

I'm running Yarn 1.7.0 and I'm having the similar error. Yarn finally manages to install the package after several runs.

An unexpected error occurred: "ENOENT: no such file or directory, lstat '/home/nieltg/.cache/yarn/v1/npm-npm-registry-client-8.5.1-8115809c0a4b40938b8a109b8ea74d26c6f5d7f1/lib/dist-tags/fetch.js'".

EDIT:
I've used yarn --network-concurrency 1 but the error still occurs on me. Here is another sample of the error and yarn-error.log file.

An unexpected error occurred: "ENOENT: no such file or directory, copyfile '/home/nieltg/.cache/yarn/v1/npm-core-js-2.5.7-f972608ff0cead68b841a16a932d0b183791814e/library/fn/date/now.js' -> '/mnt/c/Users/nieltg/Projects/React/React-16-Demo/node_modules/core-js/library/fn/date/now.js'".
commented

I'm using Yarn 1.7.0. And I can confirm the same behavior still happening to me.

It's completely random. Sometimes it happens, sometimes it doesn't.

The last I received was:

error An unexpected error occurred: "ENOENT: no such file or directory, lstat '/root/.yarn-cache/v1/npm-@storybook/addon-actions-3.4.5-ba0d0c0c74357c0852e0b890b40
commented

I'm seeing this error quite frequently with Yarn 1.9.2 on Windows Subsystem for Linux.

commented

Today we got similar problems with broken packages on Jenkins CI where pipeline run yarn install in parallel. It was working fine jest few days ago.
Fixed with yarn install --network-concurrency 1 (as mentioned in a comment). Performance did not degraded much: ~7 sec -> ~8 sec.

commented

Why was this closed? It still happens:

Toms-MacBook-Pro-2:design-to-code tommedema$ yarn install
yarn install v1.9.4
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
error An unexpected error occurred: "ENOENT: no such file or directory, lstat '/Users/tommedema/projects/vg/design-to-code/packages/vgcli/node_modules/fs-extra'".
info If you think this is a bug, please open a bug report with the information provided in "/Users/tommedema/projects/vg/design-to-code/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Toms-MacBook-Pro-2:design-to-code tommedema$ yarn install --network-concurrency 1
yarn install v1.9.4
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 24.85s.

Note that in my case it disappeared after doing yarn remove fs-extra and yarn add fs-extra in two packages, effectively upgrading this dependency.

commented

Hi, I think I found something.

I was dabbling with a piece of code which list files in a specified directory recursively using fs and rxjs and I discovered that it was likely to fail if I don't wait for lstat call to be finished before calling another lstat.

I've made a simple NPM package, async-dirtree-test, to test if your environment is affected or not with this. I'm using WSL and I found that it is likely to fail while handling a directory which has many child directories, like node_modules, even with a low number of concurrency.

Well, I still don't know if this problem is WSL specific or not. Right now I can't test it in another environment, like Linux, Mac, etc.

commented

@nieltg I wanted to share an observation that might help shape some of the others. I'm using Docker CE in WSL and Docker for Windows as my host so when I work with Docker in WSL, it feels native, but in reality the host operates in the Windows world natively (so my Dockerfiles actually resolve /c/foobar to c:/foobar in the Docker engine). This is incredibly relevant when I use bindings (inside my container, I'm mounting my local folders so that /usr/src inside the container is ultimately at c:/src/foobar (though my Dockerfile would show the binding as /c/src/foobar:/usr/src (see the automatic translation in the path?)

This distinction is important because if I yarn install inside one of those local folders, inside my container, I get the same errors I do directly in WSL (no Docker involved).

On the other hand.... if I just mkdir /tmp/src && cp ./package.json /tmp/src/ && cd /tmp/src && yarn install, everything succeeds perfectly and I can just mv /tmp/src/node_modules /c/src/foobar/ and I'm good, so that's my present workaround. Mind you that /tmp exists as a docker store (all the IO looks like a single file to the OS because it's effectively a partition in a file).

I know that involving docker isn't ideal here, but it seems to suggest that rapid file handles might be an issue since IO itself isn't and might help others work around this.

...got distracted and submitted too soon. Anyhow, my brain is elsewhere at the moment, but I'll come back later and see if I can devise a test using your approach and Docker to draw further conclusions.

commented

REOPEN

commented

We have recently started seeing the same sort of error using yarn 1.10.1, running a CI build in Azure Devops (formerly Visual Studio Team Services).

The actual dependency that is failing seems to be random best I can tell, but yarn install is falling over intermittently with the ENOENT: no such file or directory, open '/usr/local/share/.cache/yarn........ error. One time it the build will work, the next it will fail.

The workaround of yarn install --network-concurrency 1 seems to work for us.

commented

@Marclev78 same error but yarn install --network-concurrency 1 doesn't seem to work for me

commented

@Marclev78 same thing here, using yarn 1.10.1 in Azure Devops and getting the error:

Error: https://registry.yarnpkg.com/core-js/-/core-js-1.2.7.tgz: ENOENT: no such file or directory, utime 'C:\Users\grpsshagent\AppData\Local\Yarn\Cache\v1\npm-core-js-1.2.7-652294c14651db28fa93bd2d5ff2983a4f08c636\fn\string\pad-left.js'

Locally everything works as expected.

commented

I'm here to simply say I too am seeing this error.

error An unexpected error occurred: "ENOENT: no such file or directory, chmod '/usr/local/opt/asdf/installs/nodejs/8.12.0/.npm/bin/atob'".

Unfortunately I think I have to abandoning yarn for my global node binaries and moving back to npm until this is fixed.

commented

Sadly, this problem started to plague our CI builds recently, too ;(

commented

@rainabba suggestion worked for me on WSL

commented

I think that write and read operations behave differently also. Even with my hacks, I frequently see errors while calling node's fs.writeFile (wrapped with bluebird promisify though). In every single instance, I can confirm the file exists immediately after I get the write error.

I'm sending a string (XML file content) to fs.writeFile(), which ultimately calls the following, but I'm not sure if I'm prepared to undertake the challenge that be required to setup a custom build with additional debugging output from this C++ project, so that I can confirm exactly where the right is actually feeling according to node or this C++ module.

Bottom line is that the writes are not failing, but node believes they are so the scenario that makes sense to me is that the c+ plus module is succeeding, but internally it checks for the file and fails, then reports not feel your back to node and then the actual write occurs so that when I go to check for the file, it is there and the error makes no sense.

https://github.com/nodejs/node/blob/master/src/node_file.cc#L1795

commented

@bestander any way to get that issue reopened? Clearly this is not fixed and impacting a lot of people.

commented

Confirming this still happens with yarn 1.12 and Azure Pipelines.

commented

Thanks for confirming everyone.
Looks like there are multiple reasons for that error.
I'll reopen the issue, community help to debug this is needed though.

commented

also happens with yarn 1.11, but not with 1.10

commented

@bestander - Related? #6312

If so, there's some fine repro work inthere

commented

I'm affected by this issue as well.

Windows 10 / WSL

"ENOENT: no such file or directory, lstat '/mnt/c/Users/<username>/.cache/yarn/v4/<random_file_in_random_package>"
commented

@limonte WSL had an error for a while that it would randomly throw a similar error when executing npm install / yarn install. It was happening when a lot of files at once were copied to the hard drive. Please make sure that you're on the latest Windows version (1809 or higher), as it might not be caused by the yarn itself.

commented

We are also seeing the problem of Extracting tar content of undefined.

error https://registry.yarnpkg.com/eslint/-/eslint-4.19.1.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, stat '/tmp/yarncache.KTKNZ/v4/npm-eslint-4.19.1-32d1d653e1d90408854bfb296f076ec7e186a300/node_modules/eslint/lib/rules/no-compare-neg-zero.js'"

So far we mitigated this with using only one concurrent network connections with the option --network-concurrency 1. But this is more a temporary solution.

commented

I also can confirm the issue on node:11.5.0-alpine.

error An unexpected error occurred: "ENOENT: no such file or directory, lstat '/app/node_modules/<random_pacakge>

commented

I noticed that the problems seemed to be related to linking to the git repository version of packages.

Reproduce

package.json

{
  "dependencies": {
    "react-navigation-core": "https://github.com/react-navigation/react-navigation-core",
    "react-navigation-hooks": "https://github.com/react-navigation/react-navigation-hooks"
  }
}

rm -rf node_modules && yarn cache clean && yarn

Workaround

Setting network-concurrency 1 fixes the problem for me every time.

Running npm install also works.

Notes

Removing either package from the dependency list doesn't cause the error, nor does using the published npm versions of those packages.

It seems to throw a different error every time. It seems to happen randomly to different files and with slightly different errors.

error https://registry.yarnpkg.com/core-js/-/core-js-1.2.7.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, chmod  '/home/cameron/.cache/yarn/v4/npm-core-js-1.2.7-652294c14651db28fa93bd2d5ff2983a4f08c636/node_modules/core-js/library/modules/es6.reflect.apply.js'"

Variations

  • ENOENT: no such file or directory, chmod
  • ENOENT: no such file or directory, stat
  • ENOENT: no such file or directory, open
  • EEXIST: file already exists, mkdir

Other messages

info There appears to be trouble with your network connection. Retrying...
commented

Indeed, I've just tried to reproduce using your package.json and error showed up on first try.

commented

What layer does the WSL filesystem interact with the NTFS fs that's already in place?

Are you folks seeing this error in a mounted drive (/c or /mnt/c for common examples), or outside one of those mounts? Mind testing aan alternate (~/. for example) and reporting any difference?

My intuition is nagging at me, but I may be confusing issues with my docker experiences and I need to check this independently.

commented

[2/4] Fetching packages...
error https://registry.yarnpkg.com/smartwrap/-/smartwrap-1.0.10.tgz: Extracting tar content of undefined failed, the file appears to be co
rrupt: "ENOENT: no such file or directory, open 'C:\Users\Administrator\AppData\Local\Yarn\Cache\v4\npm-smartwrap-1.0.10-873ef350d
4ee1262fed4a80a55634d86ae1faf48\node_modules\smartwrap\ejq'"
info Visit https://yarnpkg.com/en/docs/cli/global for documentation about this command.

commented

Are you folks seeing this error in a mounted drive (/c or /mnt/c for common examples), or outside one of those mounts? Mind testing aan alternate (~/. for example) and reporting any difference?

Is there a consistently reproducible case where this is happening? It's been pretty random for me, but I do all of my yarn add-ing on a mounted drive and this occurs frequently.

I was able to reproduce #2629 (comment) in both a mounted drive and in ~.

I also tried to reproduce #2629 (comment) but it kept failing to fetching the last package, which I'm pretty sure is unrelated.

commented

I had the same issue in the last few hours. yarn was randomly failing to install various packages, showing the errors that were mentioned above.

I tried reseting yarn cache and re-installing and running with network-concurrency 1, neither of which had worked.

What solved the problem for me, was switching to a different network (just used my phone AP instead of WiFi) and everything worked like magic.

I have a hunch that this issue might be related to mishandled recovery for some very specific network errors. Will look into it later.

commented

I can confirm the previous comment. Setting network-concurrency doesn't help. Switching to Phone hotspot fixed the issue for me. My environment: Windows 10 (Linux subsystem - Ubuntu)

commented

I'm on WSL and had been seeing this issue with the geo-tz package which has a (slightly odd) deeply nested folder structure. I tried some --network-timeout and --network-concurrency things but got nowhere. However, when I enabled long paths on Windows (see this SuperUser post) it's now working fine. Perhaps this might help some other people on WSL. Edit it looks like I spoke too soon. It was working, and linking dependencies is faster, but now I'm seeing the same error again.

commented

Still breaking CI....

commented

Ditto - This is still an issue, even in 1.14

Arguments: 
  /home/jeff/n/bin/node /usr/share/yarn/bin/yarn.js install

PATH: 
  /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/c/Windows/System32:/mnt/c/Windows:/mnt/c/Windows/System32/wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/Windows/System32:/mnt/c/Windows:/mnt/c/Windows/System32/wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/Program Files/NVIDIA Corporation/NVIDIA NvDLISR:/mnt/c/Program Files/Git/cmd:/mnt/c/Users/jkono/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/jkono/AppData/Local/hyper/app-2.1.2/resources/bin:/mnt/c/Users/jkono/AppData/Local/Programs/Microsoft VS Code/bin:/home/jeff/n/bin

Yarn version: 
  1.14.0

Node version: 
  10.15.1

Platform: 
  linux x64

Trace: 
  Error: ENOENT: no such file or directory, scandir '/mnt/c/Users/jkono/dev/PROJECT/node_modules/@storybook/addon-links/src'

Also:

➜  yarn cache dir
/mnt/c/Users/jkono/home/.cache/yarn/v4
commented

This is pretty annoying, we see this daily on local and ci machines.

commented

Confirming this is happening all the time in CI for us as well

commented

Hi, confirming that this issue happens on our CI.
This is line with problem.

error https://registry.yarnpkg.com/core-js/-/core-js-1.2.7.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, chmod '/usr/local/share/.cache/yarn/v4/npm-core-js-1.2.7-652294c14651db28fa93bd2d5ff2983a4f08c636/node_modules/core-js/es7/regexp.js'"
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

commented

The same issue started happening today to one of our open source projects.

You can see a failing build here:
https://travis-ci.com/quid/refraction/builds/103692106

And one that succeeds (with --network-concurrency 1) here:
https://travis-ci.com/quid/refraction/builds/103693682

I hope it can help to diagnose the issue!

The source code of the repository is at:
https://github.com/quid/refraction