Make dask float polygonize chunk-invariant for rtol#2675
Merged
Conversation
Dedupe duplicate module rows (last-write-wins by last_inspected) and collapse multi-line notes to single physical lines. The notes had embedded newlines, which the merge=union .gitattributes strategy splits record-by-record, corrupting the file into a 156-column phantom row on parallel-agent appends. One line per record keeps union merges safe.
numpy CCL tests adjacent pixels with abs(value - reference) <= atol + rtol*abs(reference), where reference is the higher-ij pixel in scan order, so the predicate is asymmetric in rtol. The dask cross-chunk merge compared unordered (min, max) value ranges, so its merge decision flipped with chunk order: [[10.0, 11.05]] at atol=0, rtol=0.1 merged under numpy but split under dask, and reversing the row flipped the failure. Each chunk-boundary polygon now carries the values of its pixels on internal chunk edges, keyed by global (col, row). The cross-chunk merge applies numpy's higher-ij-is-reference orientation at the exact pixel pair straddling a boundary, so a chunked float raster produces the same polygon partition as the unchunked input. Integer rasters keep strict equality (empty cell map, range fallback). The remaining 8-connectivity divergence at large rtol is a pre-existing ring-merge topology limitation, not the rtol orientation bug, and is left untouched.
brendancol
commented
May 29, 2026
brendancol
left a comment
Contributor
Author
There was a problem hiding this comment.
PR Review: Make dask float polygonize chunk-invariant for rtol
Blockers
None.
Suggestions
- 8-connectivity at large rtol still diverges between numpy and dask (a dense 4x4 float array with rtol=0.1 gives different polygon counts). This is pre-existing on main, not introduced here. Tracing shows it is a ring-merge topology issue, not a close-value orientation issue: it fails identically across chunk shapes that do not even split the relevant pixels, so the cross-chunk boundary logic is not the cause. The PR scopes around it correctly. Worth a follow-up issue so the limitation is tracked outside the PR body.
Nits
_ranges_close(polygonize.py:295) and its interval-overlap fallback are now reached only for integer rasters and the defensive empty-cell-map case. For integers the check reduces to equality, so the overlap branch no longer fires in practice. Harmless and pre-existing, but a one-line comment marking it as a fallback path would help the next reader.
What looks good
- The fix targets the root cause: numpy CCL uses the higher-ij pixel as the rtol reference, and the merge now replicates that orientation at the exact boundary pixel pair instead of comparing unordered ranges.
- Boundary-cell maps are O(chunk perimeter), keyed by global coords, so the dask path gains no full-array materialization.
- Integer rasters keep strict equality through the empty cell map and range fallback.
- NaN boundary cells are skipped during extraction, so they never merge, matching numpy.
- Tests cover the repro, its reverse, column/row orientation, disconnected close-valued pixels, several chunkings, the default tolerance path, integer invariance, and dask+cupy.
Checklist
- Algorithm matches numpy CCL scan-order orientation
- Implemented backends consistent (4-conn float verified; integer unchanged)
- NaN handling correct
- Edge cases covered by tests
- Dask chunk boundaries handled correctly
- No premature materialization or unnecessary copies
- Benchmark not needed (bug fix)
- README/docs unchanged (no public API change)
- Docstrings updated for the new tuple field and helper
…2666) - Document that _ranges_close is now the fallback close-value test (integers and the empty-cell defensive case), since float merging goes through the direction-aware _cells_close_directed. - Add dask+cupy parity tests for the repro and its reverse, guarded by the cuda/cupy availability marker. Follow-up issue #2677 tracks the pre-existing 8-connectivity divergence at large rtol, which is a ring-merge topology problem out of scope here.
brendancol
commented
May 29, 2026
brendancol
left a comment
Contributor
Author
There was a problem hiding this comment.
Follow-up review (after review commits)
Re-reviewed the two follow-up commits.
Disposition of prior findings
- Suggestion (8-conn divergence at large rtol): deferred to follow-up issue #2677. The fix is a separate ring-merge topology problem, pre-existing on main, and would expand this PR well beyond the rtol orientation bug.
- Nit (
_ranges_closenow a fallback path): fixed. Its docstring now states it is the fallback for integers and the empty-cell defensive case, with float merging routed through_cells_close_directed.
New changes
- Added dask+cupy parity tests for the repro and its reverse, guarded by the cuda/cupy marker. They skip locally without a GPU.
No new blockers, suggestions, or nits. The 4-connectivity rtol parity is complete and the existing suite stays green (47 passed, 3 skipped on the polygonize parity files).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2666.
What
abs(value - reference) <= atol + rtol*abs(reference), wherereferenceis the higher-ijpixel in scan order, so the test is asymmetric inrtol. The dask cross-chunk merge compared unordered(min, max)ranges, so its merge decision flipped with chunk order.(col, row). The cross-chunk merge applies numpy's higher-ij-is-reference orientation at the exact pixel pair straddling a boundary, so a chunked float raster produces the same polygon partition as the unchunked input.Backends
Float path: numpy / cupy (float routes through numpy), dask+numpy, dask+cupy. The merge change lives in the shared dask path used by both dask+numpy and dask+cupy. Integer behaviour is unchanged.
Known limitation
8-connectivity at large
rtolstill diverges between numpy and dask. That is a pre-existing ring-merge topology limitation (present onmainbefore this change), not thertolorientation bug, so it is left for a separate fix.Test plan
pytest xrspatial/tests/test_polygonize_issue_2666.py(29 passed)test_polygonize.py,_issue_2583,_issue_2172, coverage files): 291 passed, 16 skippedrtol: 4-connectivity now matches numpy in every case (was 26 failures onmain)