[0z5a]

Description

Closes #3215.

When CUDA_VISIBLE_DEVICES exposes two GPUs on an eight-GPU host, simpleP2P uses NVML's system-wide count of eight and fails while constructing Device(2). Count CUDA-visible devices with Device.get_all_devices() in simpleP2P, simpleMultiGpu, and multiGPUGradientAverage; update the relevant README API descriptions. system.get_num_devices() retains its NVML semantics.

The fix was written with Claude Opus 5.5 and reviewed by the contributor. Related PR #3220 covers the same issue; this branch is based on the fix prepared before that PR.

Testing

On B300 (8 GPUs, physical GPUs 4 and 5 visible, CUDA 13.4.2, driver 610.57.04, Python 3.12, base 2da9c65a), the sample test runner reported:

Sample	Before	After
simpleP2P	failed: device count 8, Device(2) out of range	passed
simpleMultiGpu	passed	passed
multiGPUGradientAverage	skipped: mpi4py missing	skipped: mpi4py missing

The source change was tested there before rebasing. The current head has passed Ruff, syntax compilation and git diff --check; GPU and MPI tests have not been rerun on this head. The CUDA-capable device count label in the script matches the README expected output.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: NVIDIA/cuda-python
• Branch: main
• File: .coderabbit.yaml
• Commit: d4c71ef
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: eead044a-2557-4a7a-b311-0c12dff67353

Commits

Reviewing files that changed from the base of the PR and between d4c71ef and 1e42667.

Files selected for processing: 5

• samples/cuda_core/multiGPUGradientAverage/multiGPUGradientAverage.py
• samples/cuda_core/simpleMultiGpu/README.md
• samples/cuda_core/simpleMultiGpu/simpleMultiGpu.py
• samples/cuda_core/simpleP2P/README.md
• samples/cuda_core/simpleP2P/simpleP2P.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

📝 Summary

Summary by CodeRabbit

• Updates
  • Multi-GPU samples now determine the number of available CUDA devices by listing devices visible to the process. Existing checks and behavior when fewer than two devices are available remain unchanged.
• Documentation
  • Updated sample guidance to describe listing visible CUDA devices when checking device availability.

The three multi-GPU samples now count devices with Device.get_all_devices(). The README descriptions use the same API to identify devices visible to the process.

Changes: Visible device counts

Layer / File(s)	Summary
Use process-visible device counts samples/cuda_core/multiGPUGradientAverage/multiGPUGradientAverage.py, samples/cuda_core/simpleMultiGpu/simpleMultiGpu.py, samples/cuda_core/simpleP2P/simpleP2P.py, samples/cuda_core/simpleMultiGpu/README.md, samples/cuda_core/simpleP2P/README.md	The three samples replace system.get_num_devices() with len(Device.get_all_devices()). The existing minimum-device checks and device selection remain in place. The README descriptions now name Device.get_all_devices().

Assessment against linked issues

Objective	Addressed	Explanation
Count process-visible devices in the three samples and update the related README descriptions [#3215]	✅

Possibly related PRs

• NVIDIA/cuda-python#3220: Both PRs update the same samples and README descriptions to use Device.get_all_devices().

Suggested reviewers: danielfrg

Priority: Normal

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: Minimal · up to 1e426

The samples now count CUDA-visible devices, so they should work when only a subset of GPUs is exposed. No merge-blocking risk was found. GPU and MPI runs were not repeated on the final head.

Finishing Touches: 💡 1

🛠️ Fix failing CI checks: 💡

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[0z5a]

Reran the samples on the current head (1e42667f) and its base (d4c71ef1) on B300.

Setup: 8-GPU host with physical GPUs 4 and 5 visible (CUDA_VISIBLE_DEVICES=4,5), driver 610.57.04 (CUDA 13.3), NVRTC 13.3.1, Python 3.12.3, cuda-core 1.2.2, cuda-bindings 13.4.4, cupy 14.2.0, mpi4py 4.1.2 built against Open MPI 4.1.9. In this setup system.get_num_devices() returns 8 and len(Device.get_all_devices()) returns 2.

cuda_core/tests/example_tests/run_samples.py --filter simpleP2P --filter simpleMultiGpu --filter multiGPUGradientAverage:

Sample	Base d4c71ef1	Head 1e42667f
simpleP2P	failed: device count 8, ValueError: device_id must be within [0, 2), got 2	passed
simpleMultiGpu	passed	passed
multiGPUGradientAverage (mpirun -np 2)	passed	passed

With only GPU 4 visible, head's simpleP2P and simpleMultiGpu both report one device and waive (exit code 2).

multiGPUGradientAverage passes on the base too: with -np 2, ranks 0 and 1 select devices 0 and 1, which are valid. So this run shows no regression for that sample, but does not reproduce the bug there.

[Andy-Jost]

/ok to test 1e42667

[github-actions]

Doc Preview CI
Preview removed because the pull request was closed or merged.