Universal Pathlib is a python library that aims to extend Python's built-in pathlib.Path
api to use a variety of backend filesystems using fsspec
python -m pip install universal_pathlib
conda install -c conda-forge universal_pathlib
# pip install universal_pathlib s3fs
>>> from upath import UPath
>>>
>>> s3path = UPath("s3://test_bucket") / "example.txt"
>>> s3path.name
example.txt
>>> s3path.stem
example
>>> s3path.suffix
.txt
>>> s3path.exists()
True
>>> s3path.read_text()
'Hello World'
For more examples, see the example notebook here
file:
Local filessystemmemory:
Ephemeral filesystem in RAMaz:
,adl:
,abfs:
andabfss:
Azure Storage (requiresadlfs
to be installed)http:
andhttps:
HTTP(S)-based filesystemhdfs:
Hadoop distributed filesystemgs:
andgcs:
Google Cloud Storage (requiresgcsfs
to be installed)s3:
ands3a:
AWS S3 (requiress3fs
to be installed)webdav+http:
andwebdav+https:
WebDAV-based filesystem on top of HTTP(S) (requireswebdav4[fsspec]
to be installed)
Other fsspec-compatible filesystems may also work, but are not supported and tested. Contributions for new filesystems are welcome!
The individual UPath
subclasses relate in the following way with pathlib
classes:
flowchart TB
subgraph s0[pathlib]
A---> B
A--> AP
A--> AW
B--> BP
AP---> BP
B--> BW
AW---> BW
end
subgraph s1[upath]
B ---> U
U --> UP
U --> UW
BP --> UP
BW --> UW
U --> UL
U --> US3
U --> UH
U -.-> UO
end
A(PurePath)
AP(PurePosixPath)
AW(PureWindowsPath)
B(Path)
BP(PosixPath)
BW(WindowsPath)
U(UPath)
UP(PosixUPath)
UW(WindowsUPath)
UL(LocalPath)
US3(S3Path)
UH(HttpPath)
UO(...Path)
classDef np fill:#f7f7f7,stroke:#2166ac,stroke-width:2px,color:#333
classDef nu fill:#f7f7f7,stroke:#b2182b,stroke-width:2px,color:#333
class A,AP,AW,B,BP,BW,UP,UW np
class U,UL,US3,UH,UO nu
style UO stroke-dasharray: 3 3
style s0 fill:none,stroke:#0571b0,stroke-width:3px,stroke-dasharray: 3 3,color:#0571b0
style s1 fill:none,stroke:#ca0020,stroke-width:3px,stroke-dasharray: 3 3,color:#ca0020
When instantiating UPath
the returned instance type depends on the path that was provided to the constructor.
For "URI"-style paths, UPath
returns a subclass instance corresponding to the supported fsppec
protocol, defined
by the URI-scheme. If there is no specialized subclass implementation available, UPath
with return a UPath
instance
and raise a warning that the protocol is currently not being tested in the test-suite, and correct behavior is not
guaranteed.
If a local path is provided, UPath
will return a PosixUPath
or WindowsUPath
instance.
These two subclasses are 100% compatible with the PosixPath
and WindowsPath
classes of their
specific Python version, and are tested against all relevant tests of the CPython pathlib test-suite.
Contributions are very welcome. To learn more, see the Contributor Guide.
Distributed under the terms of the MIT license, universal_pathlib is free and open source software.
If you encounter any problems, please file an issue along with a detailed description.