Vortex should support reading from the Hugging Face Datasets API

For example, the following should Just Work.

import vortex as vx
url = "hf://datasets/danking00/statpopgen-benchmark/10000/vortex-file-compressed/gnomad.genomes.v3.1.2.hgdp_tgp.chr21.vortex"

f = vx.open(url)
arrays = list(f.scan())

array = vx.io.read_url(url)

This works fine for Parquet files:

import pyarrow.dataset as ds
import pyarrow.parquet as pq

table = pq.read_table("hf://datasets/danking00/statpopgen-benchmark/10000/parquet/gnomad.genomes.v3.1.2.hgdp_tgp.chr21.parquet")

dataset = ds.dataset(
    "hf://datasets/danking00/statpopgen-benchmark/10000/parquet/gnomad.genomes.v3.1.2.hgdp_tgp.chr21.parquet",
    format="parquet",
)
scanned = dataset.to_table()

Navigation Menu

Repository navigation

Vortex should support reading from the Hugging Face Datasets API #5379

See also

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Vortex should support reading from the Hugging Face Datasets API #5379

Description

See also

Activity

danking commented on Nov 18, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions