Skip to content

fix(ci): use numba-cuda-mlir wheel-only skips - #2733

Open
rwgk wants to merge 3 commits into
NVIDIA:mainfrom
rwgk:fix_nightly_issue2726
Open

fix(ci): use numba-cuda-mlir wheel-only skips#2733
rwgk wants to merge 3 commits into
NVIDIA:mainfrom
rwgk:fix_nightly_issue2726

Conversation

@rwgk

@rwgk rwgk commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Description

closes #2726

Explanation of the failure reported in #2726

The numba-cuda-mlir nightly jobs install the latest compatible release and then check out that release's matching test suite. The failing nightly resolved numba-cuda-mlir v0.5.0.

All ten numba-cuda-mlir --deselect entries were guarded by NUMBA_CUDA_MLIR_VER <= 0.4.1. Because v0.5.0 did not satisfy that condition, none of the deselects were passed to pytest. That was correct for nine of the entries, which worked around an old-version bug, but the remaining entry addressed a wheel-only environment constraint instead of a version-specific failure.

test_nvjitlink_jit_with_linkable_code_lto_dump_assembly_warn exercises some linkable inputs that require cuobjdump. The wheel-only Linux runner does not provide that executable. CUDA Python also did not set NUMBA_CUDA_MLIR_TEST_WHEEL_ONLY, so the test's existing wheel-only skip guard was not activated. The affected inputs therefore attempted to invoke the missing executable and failed.

Why none of the --deselect entries are needed anymore

The ten entries fall into two classes:

Nine version-gated deselects

Nine array-slicing and CUDA Array Interface tests were deselected to work around NVIDIA/numba-cuda-mlir#135 in numba-cuda-mlir v0.4.1 and earlier. The fix shipped in v0.4.2.

Current CI installs the latest compatible numba-cuda-mlir release, currently v0.5.0, and no configured nightly row resolves v0.4.1 or earlier. These nine deselects therefore no longer protect any supported CI configuration.

One cuobjdump-related deselect

The remaining entry deselected the complete nvJitLink test to avoid requiring cuobjdump. The test already calls self.skipTest() for only the linkable inputs in require_cuobjdump when NUMBA_CUDA_MLIR_TEST_WHEEL_ONLY is set.

This PR sets NUMBA_CUDA_MLIR_TEST_WHEEL_ONLY=1 for wheel-only (LOCAL_CTK != 1) numba-cuda-mlir jobs on Linux and Windows, matching numba-cuda-mlir's own wheel-test contract. The fatbin and object-file inputs that require cuobjdump are skipped, while the archive, cubin, and PTX inputs continue to run. The whole-test deselect is therefore unnecessary.

The existing benchmark and documentation-example collection ignores remain unchanged because they address unrelated imports of the numba package.

Testing

  • pre-commit run --all-files

Checklist

  • Existing upstream tests cover these changes.
  • Documentation is not affected by this CI-only change.

@rwgk rwgk added this to the cuda.core 1.2.0 milestone Aug 29, 2026
@rwgk rwgk added bug Something isn't working P2 Low priority - Nice to have CI/CD CI/CD infrastructure labels Aug 29, 2026
@rwgk rwgk self-assigned this Aug 29, 2026
@copy-pr-bot

copy-pr-bot Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@rwgk rwgk changed the title fix(ci): keep cuobjdump deselect environment-gated fix(ci): use numba-cuda-mlir wheel-only skips Aug 29, 2026
@rwgk rwgk added P0 High priority - Must do! and removed P2 Low priority - Nice to have labels Aug 30, 2026
@rwgk
rwgk marked this pull request as ready for review August 30, 2026 04:16
@github-actions

Copy link
Copy Markdown

@rwgk

rwgk commented Aug 30, 2026

Copy link
Copy Markdown
Contributor Author

The targeted validation passed: PR #2733 fixes issue #2726. The workflow remains red only because of unrelated failures.

Linux CUDA 13.3 MLIR Original run Validation run
Passed 4615 4615
Skipped 62 64
Failed 2 0
Xfailed 301 301

In the original job, two subtests failed because cuobjdump was missing. In the new job:

  • NUMBA_CUDA_MLIR_TEST_WHEEL_ONLY=1 was exported.
  • Pytest ran without any deselects.
  • The exact two failures became two skips: wheel-only environments do not have cuobjdump.
  • Every other test count remained identical.

That is unusually clean confirmation of the fix. The new run also used PR workflow SHA 35071a635... and artifacts from successful CI run 33292118985, built from that same SHA.

Corrections To Issue #2726

The issue’s central diagnosis is right, but two details are inaccurate:

  • The old workflow had ten deselects—nine for numba-cuda-mlir test #135 and one for nvJitLink—not twelve.
  • The old CUDA 12.9 row also installed numba-cuda-mlir 0.5.0. It passed because the affected tests skipped earlier for missing NVCC test binaries, not because it resolved version 0.4.1 or earlier.

Therefore, the evidence actually strengthens the PR’s rationale: every tested row now uses 0.5.0, and none needs the old version-gated deselects.

Remaining Red Jobs

Both runs had 42 jobs: 31 succeeded and 11 failed. The original #2726 failure disappeared but was replaced by one unrelated Windows MLIR flake:

  • Six recurring numba-cuda failures involving NumPy’s removed row_stack.
  • Two recurring cuda-core failures involving missing cuda.bindings._test_helpers.
  • One recurring Windows PyTorch access-violation flake.
  • One new Windows CUDA 12.9 MLIR failure: a device-record assertion comparing identical-looking records containing NaN.

That last test uses an uninitialized np.recarray in test_cuda_devicerecord.py, and it passed in the old run under a different random seed. The new environment variable is referenced only by nvJitLink tests in test_nvjitlink.py, so it cannot cause the device-record failure.

My conclusion: the run is sufficient targeted evidence that #2733 fixes #2726 and is ready for review.

@rwgk rwgk added the PR review get-together Mark PRs you'd like the team to review at the weekly PR review get-together. label Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working CI/CD CI/CD infrastructure P0 High priority - Must do! PR review get-together Mark PRs you'd like the team to review at the weekly PR review get-together.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: numba-cuda-mlir cu13 nightly — version gate drops the cuobjdump deselect along with the fixed #135 ones

1 participant