winnan
winnan
- (verb) to struggle, suffer, contend
Because everything about managing files on Windows is terrible.
Installation
$ python -m pip install --upgrade winnan
Usage
Use the winnan.open()
function exactly like using Python's built-in open()
function. It aims
to be a drop-in replacement.
import winnan
with winnan.open("myfile") as fileobj:
pass
The file descriptor underlying the returned file
object has the following two properties
on both POSIX and Windows systems:
The file descriptor is non-inheritable.
The file can be scheduled for deletion while the file descriptor is still open.
Saying "scheduled for deletion" rather than "deleted" is to be pedantic about how the file is put into a "delete pending" state on Windows until the last file handle is closed. See Mercurial's developer documentation as a reference for the semantics of certain operations on the file while it is in a "delete pending" state.
Motivation
Unsurprisingly, the complications around managing files involve dealing with other processes. Depending on what the Python application is doing, there may be a need to deal with not only processes spawned by the Python application but also other processes running on the machine.
As documented by PEP-446, opening a file concurrently with spawning a subprocess may lead to the file descriptor being inherited unintentionally.
The process cannot access the file because it is being used by another process.
is perhaps the most classic representation of this issue on Windows. In modern versions of Python, the
O_NOINHERIT
flag is set by default when opening file descriptors on Windows. Setting it on the opened file descriptor prevents it from being inherited by a child process. The equivalentO_CLOEXEC
flag is also set by default in modern versions of Python when opening file descriptors on POSIX systems.It is worth mentioning that due to limitations with being able to set
close_fds=True
when redirectingstdin
,stdout
, orstderr
in older versions of Python, setting theO_NOINHERIT
flag isn't sufficient for preventing files descriptors from being leaked when spawning subprocesses concurrently. Consider guarding all calls tosubprocess.Popen
with athreading.Lock
instance to avoid this as an additional issue in older versions of Python.On POSIX systems, it is possible to
unlink()
a file while it is still open in another thread or process. On Windows, in all versions of Python, non-O_TEMPORARY
files are opened withFILE_SHARE_READ | FILE_SHARE_WRITE
as their sharing mode. OmittingFILE_SHARE_DELETE
prevents another thread or process from attempting to delete a file while it is still open.
The purpose of this library is to mask these behavioral differences across different platforms and in older versions of Python.
References
There have been multiple attempts to address the FILE_SHARE_DELETE
issue within CPython itself
but they unfortunately never succeeded in being integrated.
- bpo-12939:
tempfile.NamedTemporaryFile
not particularly useful on Windows - bpo-14243: Support for opening files with
FILE_SHARE_DELETE
on Windows - bpo-15244: Add new
io.FileIO
using the native Windows API
Starting in Python 3.7, close_fds=True
is able to be set even when redirecting stdin
,
stdout
, or stderr
.