Skip to content

Support symlinks for filesystems that don't support symlinks #500

Open

Description

@ap--

This issue starts collecting ideas how symlink support could be added to universal-pathlib and filesystems that don't support symlinks.

Ideas

For all cases below, it holds that the read only case is fine, but a read write case will require careful locking. And avoiding race conditions here might be complicated / impossible.

(1) storing all symlink targets per directory in file with a specialized name

So we could make UPath special case a name, let's say .upath.symlinks that contains a path to another resource and the name of the symlink as well as its type.

# .upath.symlinks
name=abc,efg
type=file,dir
target=foo/bar/baz,foo/bar

The problem with this is of course that we'd have to always read this file, and there would have to be logic built around the case where the name collides with an actual file.

(2) storing the symlink target in a file with its actual symlink name

# bucket/abc
type=file
target=foo/bar/baz

This is the worst. Because for every single file we'd have to read it. And on object store we wouldn't be able to make directories anyways.

(3) having a global setting for each universal-pathlib implementation.

We could provide a {symlink: target} mapping that would be class scoped for each UPath implementation.
There would have to be some mechanism that would allow the user to load the symlinks they want for their UPath class, and a way to serialize it back to disk.

from upath.symlinks import set_protocol_symlinks

set_protocol_symlinks("s3", {"bucket/foo": "bucket/abc"})

UPath("s3://bucket/foo")

This mapping could of course be stored on the remote filesystem serialized to json too.

(4) implementing a chainable fsspec filesystem for symlink support

This would work similar to (3), just that we write a new fsspec filesystem, that basically does pass-through, but does the translation for each requested path, and returns the symlinks as needed when ls or glob are run.

UPath("symlinks::s3://bucket/foo", symlinks={"mapping": {"bucket/foo": "bucket/abc"}, }, s3={"anon": True})

This would remove any of these details from UPath, and we'd move symlink support basically to the fsspec layer.

(5) other options ???

@sdvillal I remember discussing this with you years ago, did you have any other idea how remote filesystem symlink support could be implemented?

Activity

  1. sdvillal commented on Feb 8, 2026

    @sdvillal

    Hard one.

    I guess the use case for keeping what is a symlink and what is not in file systems that do not support them is to be able to:

    • Goal.1: sync to/from symlinked structure in file-systems that do support the concept
    • Goal.2: keep the contract that symlinked files and dirs do change jointly - so changing one should change the other
    • Goal.3: use tricks go most efficiently avoid duplicating data when providing different organizations of files

    I think for goal 1, your proposed solutions would make it and I would go for one or the other depending on what contracts you want to enforce. I like most solution 1 and solution 4.

    For goal 3, it is a tough one. Some stores hide from you how they avoid redundancy, but they do under the hood (e.g., I suspect cloud services efficiently handle copies of files with the same hash).

    Goal 2 is, to my mind, the really complicated one to tackle - unless you can ensure every operation will be done using universal_pathlib.

    It is a pity that the need for saying "these two files are the same even if appearing in different places" is not universally available, everyone should stop using systems that do not support it! ;-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions