Skip to content

flow_accumulation_mfd: no memory guard on H*W working arrays #1321

@brendancol

Description

@brendancol

Problem

flow_accumulation_mfd (xrspatial/hydro/flow_accumulation_mfd.py) is the MFD sibling of flow_accumulation_d8 and has the same unbounded eager-allocation issue that #1318 reported and #1319 fixed. The numpy and cupy backends each allocate several H*W working buffers without checking projected memory against what the host or device has free, so a too-big raster takes the process to OOM before any user-visible error.

Allocations

CPU kernel _flow_accum_mfd_cpu:

array dtype bytes/px
accum float64 8
in_degree int32 4
valid int8 1
queue_r int64 8
queue_c int64 8
total 29

GPU kernel _flow_accum_mfd_cupy:

array dtype bytes/px
accum float64 8
in_degree int32 4
state int32 4
total 16

Working-buffer totals match the d8 figures from #1319, so the same constants apply.

MFD also takes a (8, H, W) float64 input. The public dispatch casts it via data.astype(np.float64), which materialises another ~64 B/px on top of the 29 B/px working set. The d8 PR documented the same convention (caller-side input not double-counted), so the guard accounts for kernel working memory only and the input cast is left to be sized by the caller.

Worked example

50000 x 50000 raster:

  • CPU working memory: 50000 * 50000 * 29 = ~72.5 GB
  • GPU working memory: 50000 * 50000 * 16 = ~40 GB
  • Input cast on top: 50000 * 50000 * 64 = ~160 GB

A 64 GB workstation OOMs on the working set alone before the BFS even starts.

Fix

Port the _check_memory / _check_gpu_memory helpers from #1319 (29 B/px CPU, 16 B/px GPU, 50% threshold) into flow_accumulation_mfd.py and call them from the public dispatch on the numpy and cupy branches. Dask backends stay un-guarded because per-tile allocations are bounded by chunk size.

Tracking under the same family as #1318 / #1319, alongside the parallel flow_accumulation_dinf work.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions