Skip to content

How to migrate to 0.3 for APIs that expect PathLike #460

Open

Description

@Zan-L

As an example, Polars used to work perfectly with UPath < 0.3:

path = UPath('s3a://...')
df = pl.read_parquet(path)

But after 0.3 it stops working. The problem is, none of the 3 solutions works from the migration guide. Any suggestions?

Activity

  1. ap-- commented on Oct 27, 2025

    @ap--
    Collaborator

    Hi @Zan-L

    cross-referencing: pola-rs/polars#24921

    In the example you provided, they converted remote UPaths to a string via indirectly calling os.fspath() via os.path.expanduser() see here: https://github.com/pola-rs/polars/blob/77254fce44d7b848aa9aa8bb3a0f01f5d2e63d1c/py-polars/src/polars/_utils/various.py#L235
    It's better to be explicit about how you're calling this API, instead of relying on the UPath being cast to string (and losing its storage options).

    Your example code is effectively equivalent to:

    path = UPath('s3a://...')
    df = pl.read_parquet(path.as_uri())

    Warning

    But be aware !! you're dropping all storage_options in the general case (in your example there are none).

    I would recommend to use it like this, if you want to use fsspec and s3fs as the read backend (probably slower than polars default):

    import upath
    import polars
    
    p = upath.UPath("s3://bucket/file", anon=True)
    
    polars.scan_parquet(p.open('rb'))

    Or if you want to use the polars rust backends

    import upath
    import polars
    
    p = upath.UPath("s3://bucket/file", anon=True)
    
    object_store_storage_options = ...  # convert your fsspec p.storage_options to the object store style
    
    polars.scan_parquet(p.as_uri(), storage_options=object_store_storage_options)

    Warning

    Because polars uses object-store instead of fsspec and s3fs, the storage options have different names and you have to convert: https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html

    I'll add this to the migration guide when I find some time.

    Cheers,
    Andreas

  2. cas-- commented on Dec 10, 2025

    @cas--

    Ah this explains why I am encountering new errors upgrading to latest version of UPath passed to pandas.read_parquet and pyarrow.

    Including the error here to help anyone else encountering it:

      File "pyarrow/_dataset_parquet.pyx", line 271, in pyarrow._dataset_parquet.ParquetFileFormat.make_fragment
      File "pyarrow/_dataset.pyx", line 1379, in pyarrow._dataset.FileFormat.make_fragment
      File "pyarrow/_dataset.pyx", line 137, in pyarrow._dataset._make_file_source
    TypeError: cannot construct a FileSource from s3://bucket/test.parquet

    Considering that pyarrow is fsspec compatible I assume this would be the recommended approach:

    df = pd.read_parquet(path=path.as_uri(), filesystem=path.fs)

    Unless it's a bug and pyarrow should handle UPaths?

    Incidentally I thought path.as_uri() was same as str(path) but looking at the code there is a guard for relative paths which is probably worth documenting too

    def as_uri(self) -> str:
    """Return the string representation of the path as a URI."""
    if self._relative_base is not None:
    raise ValueError(
    f"relative path can't be expressed as a {self.protocol} URI"
    )
    return str(self)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    question ❓Further information is requested

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions