Skip to content

Regression in is_file for HTTP path #535

Open

Description

@delucchi-cmu

As of v0.3.8, http directories are incorrectly identified as a regular file.

Reproducing snippet

import upath

known_dir = upath.UPath("https://data.lsdb.io/hats/")
f"as of {upath.__version__}, known dir {'IS' if known_dir.is_dir() else 'is NOT'} is a directory, known dir {'IS' if known_dir.is_file() else 'is NOT'} is a regular file"

And outputs from running under two most recent versions:

'as of 0.3.7, known dir IS is a directory, known dir is NOT is a regular file'
'as of 0.3.8, known dir IS is a directory, known dir IS is a regular file'

Activity

  1. ap-- commented on Jan 12, 2026

    @ap--
    Collaborator

    Thank you for reporting @delucchi-cmu 🙏

    This is actually correct behavior for the http filesystem and was incorrect in earlier versions. If you check this with plain fsspec:

    >>> import fsspec
    >>> fs = fsspec.filesystem('http')
    >>> fs.isdir('https://data.lsdb.io/hats')
    True
    >>> fs.isdir('https://data.lsdb.io/hats/')
    True
    >>> fs.isfile('https://data.lsdb.io/hats')
    True
    >>> fs.isfile('https://data.lsdb.io/hats/')
    True
    >>> fs.cat_file('https://data.lsdb.io/hats/')
    b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<html>\n <head>\n  <title>Index of /hats</title>\n </head>\n <body>\n<h1>Index of /hats</h1>\n  <table>\n   <tr><th valign="top"><img src="/icons/blank.gif" alt="[ICO]"></th><th><a href="?C=N;O=D">Name</a></th><th><a href="?C=M;O=A">Last modified</a></th><th><a href="?C=S;O=A">Size</a></th><th><a href="?C=D;O=A">Description</a></th></tr>\n   
    [... omitted ...]
    align="right">2025-07-18 11:15  </td><td align="right">  - </td><td>Zwicky Transient Facility DR22</td></tr>\n   <tr><th colspan="5"><hr></th></tr>\n</table>\n</body></html>\n'

    The reason it behaves like this is, that directories on http filesystems are html files with <a> link tags in them, that are relative or share a common prefix.

    I know this is confusing when thinking about it in terms of the local filesystem, but on many remote filesystems is_file() and is_dir() are not mutually exclusive.

    Let me know if you need a recommendation if this bug fix broke some of your code. I am happy to look over it to recommend a more robust way to check this in a compatible manner for remote filesystems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentation 📘Improvements or additions to documentation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions