Match streaming reproject output dtype to the other backends (#3093)#3111
Conversation
The streaming fallback allocated its assembled output as float64 regardless of source dtype, so integer rasters lost their dtype on the one path built for memory-bound inputs. It also allocated 2-D, which crashed on 3-D (y, x, band) sources. Use the same integer-round-trip rule as _reproject_dask and carry the band axis through, in both the local and dask.bag branches.
brendancol
left a comment
There was a problem hiding this comment.
PR Review: Match streaming reproject output dtype to the other backends (#3093)
Blockers (must fix before merge)
None.
Suggestions (should fix, not blocking)
- The dask.bag distributed branch (
xrspatial/reproject/__init__.py:1633) gets the same allocation fix but no test exercises it; the new tests all run the local ThreadPoolExecutor branch. A test that spins up adask.distributedLocalCluster would cover it, though that pullsdistributedinto the test deps. If that's not worth it, a comment in the test class noting the distributed branch is verified only by inspection would do.
Nits (optional improvements)
-
_reproject_streaming's docstring (xrspatial/reproject/__init__.py:1565) still says nothing about dtype. The dask paths document the integer round-trip rule in comments; one line in the streaming docstring would save the next reader a trip to_reproject_dask.
What looks good
- The dtype rule is copied from
_reproject_dask/_reproject_dask_cupyverbatim, so the parity claim is structural, not coincidental. - Both allocation sites (local and distributed) are fixed, not just the tested one.
- Empty/non-overlapping tiles return float64 sentinel fills from
_reproject_chunk_numpy; assignment into an integer result casts exactly because_detect_nodataguarantees integer-valued, in-range sentinels for integer dtypes. - The value-parity test compares the streaming output against the in-memory
reproject()result rather than just checking dtype, and the 3-D test pins the previously-crashing case.
Checklist
- Algorithm matches reference (dtype rule identical to
_reproject_dask) - Backend parity restored (streaming now matches numpy/cupy/dask/dask+cupy)
- NaN handling unchanged (float sources keep NaN sentinel semantics)
- Edge cases covered (int16, uint8, float64, 3-D band stack, multi-tile)
- Distributed dask.bag branch has no direct test (see suggestion)
- No premature materialization introduced
- Benchmark not needed (allocation-only change)
- README matrix unchanged (no new function, no backend support change)
- Docstrings: see nit
brendancol
left a comment
There was a problem hiding this comment.
Follow-up review after 784bdd3.
Both findings from the first pass are addressed:
- The distributed dask.bag branch now has direct coverage:
test_streaming_distributed_branch_preserves_dtyperuns_reproject_streamingunder an in-processLocalCluster(16 tiles vs 1 worker, so the bag branch is actually taken) and checks dtype plus exact value parity against the local-branch result. It skips cleanly whendistributedis not installed. _reproject_streaming's docstring now states the dtype rule.
No new issues found. The diff is still confined to the two allocation sites, one docstring, and tests.
…eproject-2026-06-09-01 # Conflicts: # .claude/sweep-metadata-state.csv
The projection kernels in _projections.py are parallel=True, and numba's default workqueue threading layer terminates the process when two host threads enter a parallel region at once. The streaming tile pool added test coverage for exactly that pattern, and the macos-3.14 job died with SIGABRT inside try_numba_transform. dask's threaded scheduler launches the same kernels from worker threads, so the lock lives in _projections.py: try_numba_transform and transform_points now take a module-level lock before launching kernels. Regression test hammers both entry points from 4 threads in a subprocess with NUMBA_THREADING_LAYER=workqueue forced.
|
CI follow-up on the macos-3.14 SIGABRT (run 27243383958): The new streaming tests exercise So the streaming path has had this crash latent in production since it was written; the new coverage just made CI roll the dice. The dask threaded scheduler launches the same kernels from worker threads too. Fix in 72a0f17: The windows-3.14 "failure" was a fail-fast cancellation after macos died, and the Read the Docs failure is "Build terminated due to time out" (RTD resource limit, no docs in this diff; recent unrelated PRs show the same flap). |
…) (#3181) #3111 fixed the 2-D output buffer crash in _reproject_streaming (#3100) two minutes before #3118 merged a strict xfail pinning that same crash. On merged main the test body passes, so the strict marker fails CI with XPASS. Drop the xfail and let the test assert streaming vs in-memory parity directly.
Closes #3093
_reproject_streaming()(the fallbackreproject()uses when dask is not installed and the in-memory source exceeds 512 MB) allocated its assembled output asnp.full(out_shape, nodata, dtype=np.float64)in both the local and dask.bag branches._reproject_dask/_reproject_dask_cupy(reproject: dask+cupy fast path drops integer dtype to float64 #2505 parity). The per-tile worker already cast each tile back to the source dtype; the float64 allocation was throwing that away.(y, x, band)sources now get a(*out_shape, n_bands)allocation instead of crashing withcould not broadcast input array from shape (H,W,B) into shape (H,W).Backend coverage: this only touches the streaming numpy fallback. The numpy, cupy, dask+numpy, and dask+cupy paths are unchanged and their dtype behaviour is already pinned by
TestDaskDtypeParity/TestDaskCupyDtypeParity.Test plan:
TestStreamingDtypeParity: int16/uint8 dtype round-trip, float64 stays float64, value+dtype parity against the in-memory path, 3-D band-axis assembly (the helper had no coverage at all before)