Skip to content

flow_accumulation_dinf: no memory guard on H*W working arrays #1322

@brendancol

Description

@brendancol

Summary

flow_accumulation_dinf has the same unbounded eager-allocation pattern that #1318 reported for flow_accumulation_d8. The numpy and cupy backends allocate H*W working arrays before any memory check runs, so a large raster requests tens of GB before failing.

The d8 fix landed in #1319. The dinf variant was deferred to a follow-up per the one-fix-per-security-PR policy, and this is that follow-up.

Allocations

_flow_accum_dinf_cpu (xrspatial/hydro/flow_accumulation_dinf.py, line 113):

array dtype bytes/pixel
accum float64 8
in_degree int32 4
valid int8 1
queue_r int64 8
queue_c int64 8

CPU peak working set ~29 B/pixel, matching d8.

_flow_accum_dinf_cupy (line 333):

array dtype bytes/pixel
accum float64 8
in_degree int32 4
state int32 4

GPU peak working set ~16 B/pixel, matching d8.

D-inf splits flow between two neighbours by a fractional weight, but those weights are computed inline from the angle and do not need their own H*W buffer, so the per-pixel cost is the same as d8.

Worked example

A 50000x50000 float64 dinf flow-direction raster on the numpy backend:

  • flow_dir already in RAM: 50000 * 50000 * 8 = 20 GB
  • working set: 50000 * 50000 * 29 ~ 72.5 GB

A box with 64 GB of RAM raises an unhelpful allocator error (or starts swapping) instead of a clean MemoryError that points at the dask alternative.

Fix

Mirror #1319 with dinf-specific guards: _check_memory (29 B/pixel) on the numpy path and _check_gpu_memory (16 B/pixel) on the cupy path, both wired into flow_accumulation_dinf before backend dispatch. Dask paths skip the guard because per-tile allocations are bounded by chunk size.

The mfd variant has the same shape and will be handled in a separate follow-up PR.

Related: #1318, #1319.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions