Repository navigation
Let PointsModel be a dask or pandas dataframe #153
Description
For many use cases (like xenium) points can be handled completley in memory without issue. Given that, and all the reasons the first "best practice" in the dask dataframes documentation is "use pandas", I think it's worth supporting pandas dataframes.
They have matched APIs, so it's possible many functions would not need modification to support both. For functions where we require one or the other, we can use a sklearn.utils.check_array style utility to do coercion.
Reacted by Giovanni Palla
Activity
LucaMarconato commentedon Feb 26, 2023
While I see the argument in favor, I think we should keep only
DaskDataFramefor the moment, for the following reasons:read_zarr()returns immediately. Most of the times the user may not need points.DataFrameforGeoDataFrameread_zarr(load_images_in_memory=True, load_points_in_memory=True)etc., or a function `sdata.load_element_in_memory('element_type', 'element_name'), but we need to understand the implications of this with the io. Since the workflow with io is still to be discussed (as we saw the in this pr Io/hot fix #138), I would wait until we have that clear before making the change described in this issueDeveloping this point more, I think that the best would be to have everything lazy by default, even the shapes and the table, and then loading things in-memory if asked explicitly by the user, or if needed by an algorithm.
LucaMarconato commentedon Feb 26, 2023
@gtca in
postdataeverything is lazy by default right? If I remember correctly if the user load something then it persists in memory, is it correct?ivirshup commentedon Mar 3, 2023
This doesn't fit my experience. The
_get_pointsfunction fromspatialdata-iois fairly slow, butread_parquetseems quite fast.I don't think this is a realistic option unless GeoDataFrame starts supporting 3d. Which is not on any roadmap as far as I can tell.
This is what we're trying to do with
read_dispatchedin anndata. Could be useful to link spatial data up with the anndata IO system at some point.AnnData has
.to_memoryas well as an equivalent internalsingledispatchedto_memoryfunction which we could export.LucaMarconato commentedon Mar 3, 2023
I see the speed improvements but I think that 1.88 seconds is too slow, especially with datasets that can have many more points (like MERSCOPE, see zulip message; there are 10x more points in general for a single dataset) or in the future when people will have multiple Xenium slices. I think we want the
SpatialaDataobject to load immediately (lazy loading).The
.to_memory()function would be great, it would be worth try an implementation soon with that, so the user can load large portions of the data before long processing computations.beng1290 commentedon Jun 5, 2025
The entire wall time for this is in PointsModel.parse() here:
spatialdata/src/spatialdata/models/models.py
Line 871 in 872f8c2
would suggest allowing for a mapping of column names to categories as input to PointsModel.parse() and reading the unique values with:
which took about 2 seconds on my machine. its not lazy but its a very minor change and saves about ~13 seconds per categorical column when passing any dask dataframe to PointsModel.parse(). Basically just allowing the user to precompute this if they want.
In a somewhat related note... the pandas dataframe input protocol for PointsModel.parse() could also be dramatically improved by implementing all the formatting that takes place in pandas before converting to a dask dataframe.... most of the wall time there is spent adding columns back to a predefined dask dataframe.