Skip to content

[unsupervised AI] Reschedule P2P merges when an output shuffle closes - #9386

Draft
Minghan2005 wants to merge 1 commit into
dask:mainfrom
Minghan2005:codex/fix-p2p-merge-unpack-retry-8660
Draft

Minghan2005 wants to merge 1 commit into
dask:mainfrom
Minghan2005:codex/fix-p2p-merge-unpack-retry-8660

Conversation

@Minghan2005

Copy link
Copy Markdown

Warning

This PR was written autonomously by an AI agent and has not been reviewed
by a human yet. Maintainers should ignore it until the human author has reviewed,
understood, and approved
everything that the AI agent wrote.

Closes #8660

  • Tests added / passed
  • Passes pixi run lint

Change

Use handle_unpack_errors separately for the left and right output fetch in
merge_unpack, as shuffle_unpack already does. A closed shuffle then raises
Reschedule instead of failing the merge task. Other fetch errors retain the
shared handler's policy and the corresponding shuffle ID.

The pandas merge remains outside both contexts. No scheduler logic, retry
configuration, public API, or dependencies change.

Tests

  • Before the production change, the new regression selection had six expected
    failures and eight passes. Both left/right closure integration cases failed.
  • After the change, all 14 new cases pass.
  • Complete affected module, including slow tests:
    python -m pytest distributed/shuffle/tests/test_merge.py --runslow -q:
    42 passed, zero skipped.
  • Related shuffle error coverage: 5 passed.
  • Repository-pinned changed-file pre-commit checks: Ruff check, Ruff format,
    and Mypy passed. Codespell had no matching documentation files.

The integration cases run a real local scheduler, worker, P2P transfers, and
pandas merge. They inject one closed-output fetch on each side, require the
injection to occur, and compare the complete recovered result with pandas,
including duplicate-key multiplicity. Unit cases cover explicit rescheduling,
consistency/disk errors, other fetch errors, and unchanged merge errors.

Verification limits

Tests used Python 3.13.14, Dask 2026.8.0, pandas 3.1.1, and PyArrow 26.0.1 with
this checkout installed editable and the repository's pytest configuration and
plugins intact. The frozen Pixi environment setup timed out after 180 seconds;
no Pixi or pixi run lint success is claimed. The changed-file pre-commit run
passed instead. No full-repository or cross-platform test pass is claimed.

The deterministic one-shot fault injection verifies the merge task's recovery
contract; it does not simulate every worker-loss or closed-run lifecycle.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    40 files  ±  0      40 suites  ±0   14h 4m 59s ⏱️ + 13m 21s
 4 174 tests + 14   3 995 ✅ + 14    178 💤 ±0  1 ❌ ±0 
81 212 runs  +253  76 971 ✅ +251  4 240 💤 +2  1 ❌ ±0 

For more details on these failures, see this check.

Results for commit d861cdf. ± Comparison against base commit dc182bd.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P2P] merge_unpack does not reschedule on ShuffleClosedError

1 participant