`dart run <package>` can fail if a `dart pub get` operation is running concurrently

Question

`dart run <package>` can fail if a `dart pub get` operation is running concurrently

andrewkolos opened this issue 2 months ago · comments

As documented in Running a program that's a package, dart run can be used to invoke a dart program that exists in a package dependency. However, this can fail with an filesystem-related error if dart pub get is running concurrently (this can be happen frequently, as the Dart VSCode extension watches for changes on pubspec.yaml and will run dart pub get automatically).

Minimal repro

To demonstrate what I mean, here is a bash script that can repro this issue:

dart create myproject
cd myproject
dart pub add vector_graphics_compiler
dart pub get & dart run vector_graphics_compiler

If this doesn't repro, try running dart pub get & dart run vector_graphics_compiler once more. There might be a race-condition here. Regardless, here is the kind of error this should print out:

Unhandled exception:
PathNotFoundException: Cannot open file, path = '/Users/andrewkolos/Desktop/deleteme/pubgetrun/.dart_tool/pub/bin/vector_graphics_compiler/tmpXaiYq6/vector_graphics_compiler.dart-3.4.0-190.0.dev.snapshot.incremental.temp' (OS Error: No such file or directory, errno = 2)
#0      _checkForErrorResponse (dart:io/common.dart:55)
dart-lang/sdk#1      _File.open.<anonymous closure> (dart:io/file_impl.dart:381)
<asynchronous suspension>
dart-lang/sdk#2      _FileStreamConsumer.addStream.<anonymous closure> (dart:io/file_impl.dart:205)
<asynchronous suspension>

Compilation did not produce any result. Expected file at `/Users/andrewkolos/Desktop/deleteme/pubgetrun/.dart_tool/pub/bin/vector_graphics_compiler/tmpXaiYq6/vector_graphics_compiler.dart-3.4.0-190.0.dev.snapshot.incremental.temp`
Built vector_graphics_compiler:vector_graphics_compiler.
Cannot rename file to '/Users/andrewkolos/Desktop/deleteme/pubgetrun/.dart_tool/pub/bin/vector_graphics_compiler/vector_graphics_compiler.dart-3.4.0-190.0.dev.snapshot', path = '/Users/andrewkolos/Desktop/deleteme/pubgetrun/.dart_tool/pub/bin/vector_graphics_compiler/tmpXaiYq6/vector_graphics_compiler.dart-3.4.0-190.0.dev.snapshot.incremental.temp' (OS Error: No such file or directory, errno = 2)
PathNotFoundException: Cannot rename file to '/Users/andrewkolos/Desktop/deleteme/pubgetrun/.dart_tool/pub/bin/vector_graphics_compiler/vector_graphics_compiler.dart-3.4.0-190.0.dev.snapshot', path = '/Users/andrewkolos/Desktop/deleteme/pubgetrun/.dart_tool/pub/bin/vector_graphics_compiler/tmpXaiYq6/vector_graphics_compiler.dart-3.4.0-190.0.dev.snapshot.incremental.temp' (OS Error: No such file or directory, errno = 2)

`dart info` output

#### General info

- Dart 3.4.0-190.0.dev (dev) (Thu Feb 29 08:14:33 2024 -0800) on "macos_arm64"
- on macos / Version 14.3.1 (Build 23D60)
- locale is en

#### Project info

- sdk constraint: '>=3.0.0 <4.0.0'
- dependencies: cupertino_icons, flutter, vector_graphics, vector_graphics_compiler
- dev_dependencies: flutter_lints, flutter_test
- elided dependencies: 4

#### Process info

| Memory |  CPU | Elapsed time | Command line                                                                    |
| -----: | ---: | -----------: | ------------------------------------------------------------------------------- |
|   4 MB | 0.0% |  06-03:09:32 | dart devtools --machine --allow-embedding                                       |
|   7 MB | 0.0% |  06-01:39:55 | dart devtools --machine --allow-embedding                                       |
|   6 MB | 0.0% |  06-01:38:20 | dart devtools --machine --allow-embedding                                       |
|   6 MB | 0.0% |  06-03:08:18 | dart devtools --machine --allow-embedding                                       |
|   4 MB | 0.0% |  06-03:07:08 | dart devtools --machine --allow-embedding                                       |
|   4 MB | 0.0% |  06-03:05:38 | dart devtools --machine --allow-embedding                                       |
|   4 MB | 0.0% |  06-03:02:59 | dart devtools --machine --allow-embedding                                       |
|   4 MB | 0.0% |  06-02:59:53 | dart devtools --machine --allow-embedding                                       |
|   6 MB | 0.0% |  06-02:52:31 | dart devtools --machine --allow-embedding                                       |
|   4 MB | 0.0% |  06-02:47:30 | dart devtools --machine --allow-embedding                                       |
|  20 MB | 0.0% |     01:07:20 | dart devtools --machine --allow-embedding --dtd-uri ws:<path>/                  |
|  20 MB | 0.0% |     01:07:20 | dart devtools --machine --allow-embedding --dtd-uri ws:<path>/                  |
|   5 MB | 0.0% |  01-22:35:28 | dart devtools --no-launch-browser                                               |
|   5 MB | 0.0% |  01-22:35:27 | dart devtools --no-launch-browser                                               |
|  40 MB | 0.0% |     01:07:21 | dart language-server --protocol=lsp --client-id=VS-Code --client-version=3.84.0 |
| 121 MB | 0.0% |     01:07:20 | dart language-server --protocol=lsp --client-id=VS-Code --client-version=3.84.0 |
|  18 MB | 0.0% |     01:07:21 | dart tooling-daemon --machine                                                   |
|  18 MB | 0.0% |     01:07:20 | dart tooling-daemon --machine                                                   |
|   6 MB | 0.0% |  02-01:34:17 | dart tooling-daemon --machine                                                   |
|  34 MB | 0.0% |     01:07:21 | flutter_tools.snapshot daemon                                                   |
|  34 MB | 0.0% |     01:07:20 | flutter_tools.snapshot daemon                                                   |

Impact/Use case (added March 6)

This was encountered while working on a new feature for Flutter: flutter/flutter#143348. This issue is preventing this new feature from being compatible with hot reload.

The idea of this feature is that Flutter app developers can specify pub packages to be used to transform assets as they are copied to the device.

Here is how a user would could run into this issue:

In VSCode, the user edits their pubspec.yaml to include a new asset that is configured to have a package transform it:

flutter:
  # ...
  assets:
    - path: my-new-asset.svg # new
      transformers:
        - package: vector_graphics_compiler

The user saves their file, VSCode automatically runs a flutter pub get
Within the new few seconds, the user initiates a hot reload
The flutter tool invokes dart run vector_graphics_compiler --input=<path to my-new-asset.svg> --output=<path to output assets dir>. If the pub get from step 2 has not already completed, then the issue will be reproduced here.

Christopher Fujino · Answer 1 · Wed Mar 06 2024 09:36:48 GMT+0800 (China Standard Time)

cc @bkonyi & @jonasfj who would have the context on how this works? I was surprised to see vector_graphics_compiler.dart-3.4.0-190.0.dev.snapshot.incremental.temp in the FileSystemException.

Lasse R.H. Nielsen · Answer 2 · Wed Mar 06 2024 16:47:49 GMT+0800 (China Standard Time)

Move to https://github.com/dart-lang/pub if it's entirely a pub issue, or keep here if it should be treated as an interaction between dart run and dart pub.

Jonas Finnemann Jensen · Answer 3 · Wed Mar 06 2024 18:30:42 GMT+0800 (China Standard Time)

Transferring since snapshots inside .dart_tool/pub/bin/ are created by pub still.

For context:

It's not necessary to run pub get after pub add -- pub add will resolved dependencies
It's not necessary to run pub get before dart run -- dart run will run pub get if necessary.

That said: I think we should aim to avoid a race condition like this.

@sigurdm my two cents is that perhaps pub get deletes .dart_tool/pub/bin/ or deletes temporary files in there? Thus, renaming it becomes a problem.

Christopher Fujino · Answer 4 · Thu Mar 07 2024 03:56:57 GMT+0800 (China Standard Time)

For context:

It's not necessary to run pub get after pub add -- pub add will resolved dependencies

It's not necessary to run pub get before dart run -- dart run will run pub get if necessary.

Right, this is just a minimal repro, the actual issue Andrew's hitting is when saving pubspec.yaml and Dart-Code auto-triggers a flutter pub get on it, and then concurrently he triggers a hot reload, which now will invoke dart run <ASSET_TRANSFORMER> as part of the build for assets that specify one in their pubspec.yaml: flutter/flutter#143348

Sigurd Meldgaard · Answer 5 · Thu Mar 07 2024 17:39:55 GMT+0800 (China Standard Time)

We probably delete the .dart_tool/pub/bin/ folder as a cleanup, while we have the temp-folder in there in the other process.

Perhaps it is enough to create the temp-folder at a higher directory level, so that it doesn't get removed in the clean-up process.

Andrew Kolos · Answer 6 · Tue Mar 12 2024 04:29:46 GMT+0800 (China Standard Time)

Is there any planned work for this issue? This will soon (within X days) be blocking flutter/flutter#143348 barring a hacky workaround

Sigurd Meldgaard · Answer 7 · Tue Mar 12 2024 18:42:10 GMT+0800 (China Standard Time)

Sorry for stalling on this.

#4170 should fix this.