Repository navigation
Description
This issue starts collecting ideas how symlink support could be added to universal-pathlib and filesystems that don't support symlinks.
Ideas
For all cases below, it holds that the read only case is fine, but a read write case will require careful locking. And avoiding race conditions here might be complicated / impossible.
(1) storing all symlink targets per directory in file with a specialized name
So we could make UPath special case a name, let's say .upath.symlinks that contains a path to another resource and the name of the symlink as well as its type.
# .upath.symlinks
name=abc,efg
type=file,dir
target=foo/bar/baz,foo/bar
The problem with this is of course that we'd have to always read this file, and there would have to be logic built around the case where the name collides with an actual file.
(2) storing the symlink target in a file with its actual symlink name
# bucket/abc
type=file
target=foo/bar/baz
This is the worst. Because for every single file we'd have to read it. And on object store we wouldn't be able to make directories anyways.
(3) having a global setting for each universal-pathlib implementation.
We could provide a {symlink: target} mapping that would be class scoped for each UPath implementation.
There would have to be some mechanism that would allow the user to load the symlinks they want for their UPath class, and a way to serialize it back to disk.
from upath.symlinks import set_protocol_symlinks
set_protocol_symlinks("s3", {"bucket/foo": "bucket/abc"})
UPath("s3://bucket/foo")This mapping could of course be stored on the remote filesystem serialized to json too.
(4) implementing a chainable fsspec filesystem for symlink support
This would work similar to (3), just that we write a new fsspec filesystem, that basically does pass-through, but does the translation for each requested path, and returns the symlinks as needed when ls or glob are run.
UPath("symlinks::s3://bucket/foo", symlinks={"mapping": {"bucket/foo": "bucket/abc"}, }, s3={"anon": True})This would remove any of these details from UPath, and we'd move symlink support basically to the fsspec layer.
(5) other options ???
@sdvillal I remember discussing this with you years ago, did you have any other idea how remote filesystem symlink support could be implemented?
Activity
sdvillal commentedon Feb 8, 2026
Hard one.
I guess the use case for keeping what is a symlink and what is not in file systems that do not support them is to be able to:
I think for goal 1, your proposed solutions would make it and I would go for one or the other depending on what contracts you want to enforce. I like most solution 1 and solution 4.
For goal 3, it is a tough one. Some stores hide from you how they avoid redundancy, but they do under the hood (e.g., I suspect cloud services efficiently handle copies of files with the same hash).
Goal 2 is, to my mind, the really complicated one to tackle - unless you can ensure every operation will be done using universal_pathlib.
It is a pity that the need for saying "these two files are the same even if appearing in different places" is not universally available, everyone should stop using systems that do not support it! ;-)