You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
How to use UPath and fsspec for file operations on multiple backends? #309
Not sure if this is the right place to ask this question, and I debated if it should go in the community Q&A?
Anyway, I want to leverage UPath and fsspec to work with multiple storage backends, for now limited to local disk and s3. One of the main use cases is to copy files and folders from one to the other.
So triggered by the answer, I feel I may have some misunderstanding of how to use the library effectively. I have read some of the documentation, but I may have missed something obvious.
Here is an explanation for how all of this works together. Some of it skips over details, so I really recommend to read the filesystem_spec docs thoroughly and look at the fsspec.spec.AbstractFileSystem implementation in filesystem_spec.
fsspec
All the filesystem abstractions and filesystem operations are implemented and defined in filesystem_spec. The base class of all these filesystems is in fsspec.spec.AbstractFileSystem.
To use a filesystem registered in fsspec, you instantiate the specific subclass of AbstractFileSystem you want to use S3FileSystem for example, by providing storage options to the class constructor.
The instantiated filesystem then takes paths (or paths prefixed with the same protocol) as arguments in all its methods.
When you want to reference an object on a filesystem, you need 3 pieces of information:
which filesystem? this is provided by the protocol, since subclasses of AbstractFileSystem can register their supported protocols in fsspec.registry. (for example: "s3")
what filesystem configuration? this is provided by the storage_options. They are the constructor parameters for building the filesystem instance. (this could be your aws credentials)
what object this is provided by the path. this is a string that references an object for the filesystem instance. (for example: "mybucket/my/special/key")
If you have all three pieces of information, you can provide access to the information stored in the object you're referencing.
Some filesystems in fsspec support combining the 3 pieces of information into a single string urlpath, usually of the form: protocol://path?storage_option1=1&storage_option=2
upath
The UPath class does 2 things:
it provides a way to store all these 3 pieces of information in one object and makes it convenient to pass the information along.
it provides the pathlib.PurePath and pathlib.Path interface for modifying the path and for reading/writing to the path, this allows you to easily add support for arbitrary filesystems if your existing code uses the pathlib.Path interface. (Note there will be a minor, but significant change with Inherit from PathBase instead of Path #193, which will basically remove the __fspath__ method from the UPath interface)
The copy functionality between filesystems will be available in UPath once #227 is completed, which relies on #193. And once that interface exists, the internal implementation will likely be based on the generic filesystem, but that's tbd.
when to use what
UPath simplifies passing around paths in your code and it's a convenient tool for building uris for supported protocols. It's not performance optimized, and adds (in some cases a lot of) overhead to operations.
If your only concern is performance: use fsspec exclusively.
If you like the pathlib interface: use UPath
For cases where it's you need to move between them: UPath provides you with an easy way to move to fsspec:
UPath().protocol# the protocol string used by fsspecUPath().storage_options# the storage_options required to create the fsspec filesystem instanceUPath().path# a path string that can be used by the fsspec instanceUPath().fs# a convenience helper to do fsspec.filesystem(pth.protocol, **pth.storage_options)
Regarding your question
It feels (at least) the part dest.fs.put(str(src), str(dest)) is not the right way for my use case, is there a better way
changed the title [-]Question on using UPath and fs copy/put[/-][+]How to use UPath and fsspec for file operations on multiple backends?[/+]on May 9, 2025
Activity
ap-- commentedon Nov 8, 2024
Hello @scholtalbers
Here is an explanation for how all of this works together. Some of it skips over details, so I really recommend to read the filesystem_spec docs thoroughly and look at the
fsspec.spec.AbstractFileSystemimplementation in filesystem_spec.fsspec
All the filesystem abstractions and filesystem operations are implemented and defined in filesystem_spec. The base class of all these filesystems is in
fsspec.spec.AbstractFileSystem.To use a filesystem registered in fsspec, you instantiate the specific subclass of
AbstractFileSystemyou want to useS3FileSystemfor example, by providing storage options to the class constructor.The instantiated filesystem then takes paths (or paths prefixed with the same protocol) as arguments in all its methods.
When you want to reference an object on a filesystem, you need 3 pieces of information:
AbstractFileSystemcan register their supported protocols infsspec.registry. (for example: "s3")"mybucket/my/special/key")If you have all three pieces of information, you can provide access to the information stored in the object you're referencing.
Some filesystems in fsspec support combining the 3 pieces of information into a single string urlpath, usually of the form:
protocol://path?storage_option1=1&storage_option=2upath
The
UPathclass does 2 things:pathlib.PurePathandpathlib.Pathinterface for modifying the path and for reading/writing to the path, this allows you to easily add support for arbitrary filesystems if your existing code uses thepathlib.Pathinterface. (Note there will be a minor, but significant change with Inherit fromPathBaseinstead ofPath#193, which will basically remove the__fspath__method from the UPath interface)The copy functionality between filesystems will be available in UPath once #227 is completed, which relies on #193. And once that interface exists, the internal implementation will likely be based on the generic filesystem, but that's tbd.
when to use what
UPathsimplifies passing around paths in your code and it's a convenient tool for building uris for supported protocols. It's not performance optimized, and adds (in some cases a lot of) overhead to operations.For cases where it's you need to move between them:
UPathprovides you with an easy way to move to fsspec:Regarding your question
As mentioned above: not yet. But hopefully soon.
Let me know if this helped,
Cheers,
Andreas 😃
scholtalbers commentedon Nov 14, 2024
Yes thanks a lot Andreas, this clarifies a few things for me and I'm looking forward for the upcoming features!
[-]Question on using UPath and fs copy/put[/-][+]How to use UPath and fsspec for file operations on multiple backends?[/+]