njsmith / posy

I just tried downloading and using cargo run. This is on Windows 11, 64 bit, with Rust 1.66.0. I got a bunch of errors:

error[E0433]: failed to resolve: could not find `unix` in `os`
  --> src\main.rs:95:22
   |
95 |         use std::os::unix::process::CommandExt;
   |                      ^^^^ could not find `unix` in `os`

error[E0600]: cannot apply unary operator `-` to type `usize`
  --> src\platform_tags\windows.rs:43:13
   |
43 |             -1 as *const std::ffi::c_void,
   |             ^^
   |             |
   |             cannot apply unary operator `-`
   |             help: you may have meant the maximum value of `usize`: `usize::MAX`
   |
   = note: unsigned values cannot be negated

error[E0308]: mismatched types
  --> src\platform_tags\windows.rs:63:36
   |
61 | fn map(machine: u16) -> Result<&'static str> {
   |                         -------------------- expected `std::result::Result<&'static str, ErrReport>` because of return type
62 |     match machine {
63 |         IMAGE_FILE_MACHINE_I386 => "win32",
   |                                    ^^^^^^^ expected enum `std::result::Result`, found `&str`
   |
   = note:   expected enum `std::result::Result<&'static str, ErrReport>`
           found reference `&'static str`
help: try wrapping the expression in `Ok`
   |
63 |         IMAGE_FILE_MACHINE_I386 => Ok("win32"),
   |                                    +++       +

error[E0308]: `?` operator has incompatible types
  --> src\platform_tags\windows.rs:74:15
   |
74 |     tags.push(map(native)?);
   |               ^^^^^^^^^^^^- help: try using a conversion method: `.to_string()`
   |               |
   |               expected struct `std::string::String`, found `&str`
   |
   = note: `?` operator cannot convert from `&str` to `std::string::String`

error[E0277]: can't compare `&u16` with `u16`
  --> src\platform_tags\windows.rs:77:20
   |
77 |         if machine != native && is_wow64_guest_machine_supported(machine)? {
   |                    ^^ no implementation for `&u16 == u16`
   |
   = help: the trait `PartialEq<u16>` is not implemented for `&u16`
   = help: the following other types implement trait `PartialEq<Rhs>`:
             <f32 as PartialEq<serde_json::Value>>
             <f32 as PartialEq>
             <f64 as PartialEq<serde_json::Value>>
             <f64 as PartialEq>
             <i128 as PartialEq>
             <i16 as PartialEq<serde_json::Value>>
             <i16 as PartialEq>
             <i32 as PartialEq<serde_json::Value>>
           and 20 others

error[E0308]: mismatched types
  --> src\platform_tags\windows.rs:77:66
   |
77 |         if machine != native && is_wow64_guest_machine_supported(machine)? {
   |                                 -------------------------------- ^^^^^^^ expected `u16`, found `&u16`
   |                                 |
   |                                 arguments to this function are incorrect
   |
note: function defined here
  --> src\platform_tags\windows.rs:26:4
   |
26 | fn is_wow64_guest_machine_supported(machine: u16) -> Result<bool> {
   |    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ------------
help: consider dereferencing the borrow
   |
77 |         if machine != native && is_wow64_guest_machine_supported(*machine)? {
   |                                                                  +

error[E0308]: mismatched types
  --> src\platform_tags\windows.rs:78:27
   |
78 |             tags.push(map(machine)?);
   |                       --- ^^^^^^^ expected `u16`, found `&u16`
   |                       |
   |                       arguments to this function are incorrect
   |
note: function defined here
  --> src\platform_tags\windows.rs:61:4
   |
61 | fn map(machine: u16) -> Result<&'static str> {
   |    ^^^ ------------
help: consider dereferencing the borrow
   |
78 |             tags.push(map(*machine)?);
   |                           +

error[E0308]: `?` operator has incompatible types
  --> src\platform_tags\windows.rs:78:23
   |
78 |             tags.push(map(machine)?);
   |                       ^^^^^^^^^^^^^- help: try using a conversion method: `.to_string()`
   |                       |
   |                       expected struct `std::string::String`, found `&str`
   |
   = note: `?` operator cannot convert from `&str` to `std::string::String`

warning: unreachable expression
   --> src\tree.rs:269:9
    |
267 |             bail!("symlinks not supported on this platform");
    |             ------------------------------------------------ any code following this expression is unreachable
268 |         }
269 |         Ok(())
    |         ^^^^^^ unreachable expression
    |
    = note: `#[warn(unreachable_code)]` on by default

error[E0599]: no method named `exec` found for struct `std::process::Command` in the current scope
  --> src\main.rs:96:17
   |
96 |         Err(cmd.exec())?;
   |                 ^^^^ method not found in `std::process::Command`

Some errors have detailed explanations: E0277, E0308, E0433, E0599, E0600.
For more information about an error, try `rustc --explain E0277`.
warning: `posy` (bin "posy") generated 1 warning
error: could not compile `posy` due to 9 previous errors; 1 warning emitted

I didn't find any specific instructions on how to build on Windows, so this was a bit of a "let's see how it goes" exercise. If the answer is that getting things working on Windows is still a work in progress, that's fine.

Whoops. Those all look like pretty trivial errors, but that's worse than I expected. I thought I built on Windows at some point! Oh well, just gotta fix them.

Ok, I fixed the build at least. Now it's crashing with some sort of permission-denied error when I try to take a file lock -- that's as far as I got tonight.

Confirmed, I get the same as you describe. Sorry, I've no immediate idea what the permission error might mean. I'll dig a bit if I get the chance.

Thanks for looking at this so quickly. I'll try to keep checking the windows builds, but just as a validation that things are OK. If there are issues, fix them at your own pace, don't feel that I'm pushing for a quick resolution.

This looks like the lock error: danburkert/fs2-rs#26

I switched to enabling write permission instead of append and it gets past that, but then I get another access denied error from somewhere inside pubgrub::solver::resolve (returned as an ErrorChoosingPackageVersion).¹

Is it possible we're opening the locked file a second time by name? Locks on Windows are typically per file handle, rather than per process (despite the... ambiguous... documentation), but maybe that's the same elsewhere? It's also possible that the calculation of the length of the file (a.k.a. the range of the lock) is wrong in a way that matters, but I'm really spitballing here.

"Append" on Windows is a creation option, while "write" is both an access mode and a sharing mode. "Locked" is yet another mode, and so getting them all aligned can be nontrivial. ↩

Nice catch -- the append -> write part definitely helps.

I also found an issue that only comes up on the second run, where Posy has a warm HTTP cache: we store HTTP cache data in individual files, so each request does:

Lock cache entry
Read existing file
(if it's changed) Write out new file to temporary
Move temporary over existing file, using the Rust tempfile crate, which ultimately does MoveFileExW(old_path, new_path, MOVEFILE_REPLACE_EXISTING):

https://github.com/Stebalien/tempfile/blob/1c6149d7ce33b2ad8b20a38a8ddce0fc29f68f0e/src/file/imp/windows.rs#L97

However, it turned out that this was giving an Access Denied, because we weren't closing the existing file before calling MoveFileExW. Sigh. Easy fix is to close the old file (a0282ca), but it's not a real fix, because we can't assume no-one else has the file open. I added a test demonstrating the issue here: aff2f7e. It's currently marked to be skipped on Windows, because the part labeled TODO fails.

@zooba I was under the impression that Windows was less strict about this these days... is tempfile calling the wrong function to do the rename, like it should be ReplaceFile or using FILE_RENAME_FLAG_POSIX_SEMANTICS or something?

The other failure looks like it's happening in the PEP 517 code when trying to build peewee from an sdist, and is showing up two different ways... still trying to work out what's happening here.

Huh. I figured out one small part of the problem... it's getting thrown for a loop by the existence of cffi-1.0.2-2.tar.gz. Because the heuristic for parsing sdist names is to take everything up to the last - as the name, to handle scikit-learn-1.2.tar.gz, so it's parsing that as a package named cffi-1.0.2, version 2. And this only happens on Windows, because Trio has a conditional dependency that only pulls in cffi when running on Windows. So: #9

One of the "Access denied" errors when building from source is here:

posy/src/package_db/build_wheel.rs

Line 349 in aff2f7e

fs::rename(&tempdir.into_path(), &*handle)?;

I'm quite puzzled by this one. handle is a path into a temporary directory, that we're holding a lock on. We check that !handle.exists(). And then if it doesn't exist, we create it by making another temp dir, filling it in, and then renaming the temp dir -> handle. And the rename fails with an access violation, even though we know handle doesn't exist. Is there something weird like you can't rename a directory while anyone is holding open handles inside it?

Is there something weird like you can't rename a directory while anyone is holding open handles inside it?

Yeah, this is one of the not-quite-like-POSIX semantics that throws people.

Windows (possibly just NTFS, but that's essentially universal) doesn't remove names until it removes the file/directory object, which doesn't happen until all handles are closed. POSIX [implementations] remove the entry immediately, and keep the data around until the handles are closed, but you can reuse the name for a new file.

Most likely the best way to handle this on Windows is to lock on a file other than the ones you're manipulating. (Or even better, a named mutex, but that's probably too divergent from the POSIX implementation to be worth the effort.)

we store HTTP cache data in individual files, so each request does: ...

This is going to be best done with file share options, basically, open it with GENERIC_READ | GENERIC_WRITE, FILE_SHARE_NONE, check the contents, then overwrite it. (You could first open with GENERIC_READ, FILE_SHARE_READ to do double-checked locking, depending on how many parallel readers you expect.) Just another place where the POSIX patterns don't translate well, and unfortunately the APIs may not even support this easily.

A more compatible algorithm might be to open for read/write and allow others to open for read/write (which should be the typical default for open(..., 'w')), and then use file locking. Start with a shared lock over the whole file while you read, since the lock operation should block until the lock is acquired (and as long as only shared locks exist, it'll be instant). When you need to update the file, take an exclusive lock, which should block until all existing shared locks are released and then will prevent new shared locks from being acquired. "Full" "details" at this page, but there's a lot left unstated.¹

Some of these components would definitely make sense as a library just to hide the platform-specific algorithms. You don't want this stuff bleeding up into your main app, though of course it's impossible to know the right API to put in place to prevent the abstraction leaking. Still, correct-and-slow is better than fast-and-wrong, so if you decide to just throw a massive lock around the whole thing on Windows, I won't judge you :)

For example, if an exclusive lock is waiting and a new shared lock is requested, does it block? Or are locks FIFO? Probably just needs to be tested, but as it's undocumented, it shouldn't be relied upon. ↩

You can get more control by dropping down to NTCreateFile if needed. My fs_at crate does that - I'm not sure you need its API for what you're doing, but if you wanted to use NTCreateFile, I think its a decent example of the complexity involved in using it.

Not working on Windows

Footnotes

Footnotes