Skip to content

ThreadPoolExecutor leak on mid-stream failure in tiled writes #2276

@brendancol

Description

@brendancol

Describe the bug

_write_streaming in xrspatial/geotiff/_writer.py creates a ThreadPoolExecutor for per-tile compression on the streaming tiled-write path. The pool is constructed at line 1026 and shut down with tile_pool.shutdown(wait=True) only after the tile-row loop finishes (line 1131).

If anything raises in between (a compression error, a dask compute failure, a disk write failure), the shutdown call is skipped. The outer try/except BaseException one frame up cleans up the temp file but never touches the pool. Worker threads survive the failure path and the executor leaks.

Fix: wrap the executor lifetime in try/finally, or use it as a context manager (with ThreadPoolExecutor(...) as executor:) so shutdown is guaranteed. In-flight futures should be cancelled/awaited cleanly on the error path.

Expected behavior

On any exception raised during the streaming tiled write, the ThreadPoolExecutor is shut down before the exception propagates. No worker thread should outlive the failed call.

Repro sketch

  • Call _write_streaming with a dask-backed input that compresses (e.g. compression='deflate') and a streaming_buffer_bytes small enough to force multiple horizontal segments.
  • Inject a failure in the per-tile work (monkey-patch _compress_block to raise on the second tile, or patch dask compute to raise after the first segment).
  • Confirm that no ThreadPoolExecutor worker threads remain alive after the call returns.

Additional context

test_streaming_write_parallel.py covers the happy path but does not exercise the failure path.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinginfrastructureCI, benchmarks, and tooling

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions