cuda.core: graph slot table for node attachment lifetimes

[Andy-Jost]

Summary

CUDA graph nodes frequently reference Python-owned resources — kernel-argument buffers, host-callback functions and their user data, and memcpy/memset operands — but the driver does not keep those resources alive. The driver copies argument values into a node at add time; it does not retain the Python wrappers or the device allocations they point at. If the owning object is collected (or a Buffer is closed) before the graph is instantiated or launched, the node is left referencing freed memory.

This PR introduces a per-graph slot table that binds the lifetime of these attachments to the lifetime of the CUgraph itself. Each node gets a small, fixed set of slots; an owner placed in a slot is held until the graph is destroyed. The table is stored on the graph as a CUDA user object, so destroying (or cloning) the graph propagates ownership correctly with no per-node bookkeeping in Python.

It also exposes GraphBuilder.graph_definition, completing step 3 of #1330 by giving users an explicit GraphDefinition view of a captured graph.

Design

• OpaqueHandle: a type-erased owning handle with two factories — make_opaque_py retains a PyObject* (incref/decref), and make_opaque_malloc owns a malloc'd buffer. A slot can therefore hold any kind of owner uniformly.
• Per-graph slot table: a map from CUgraphNode to a fixed-size array of OpaqueHandle slots. It is created lazily on first attachment and retained on the CUgraph as a user object (cuUserObjectCreate + cuGraphRetainUserObject with MOVE). When the graph is destroyed, the user object's destructor frees the table and releases every owner it holds. Conditional-branch body graphs (ref handles) get their own table the same way.
• Attachment API: graph_set_slot(graph, node, slot, owner) installs an owner into a node slot and returns CUresult for HANDLE_RETURN-style error handling. This replaces the previous approach of attaching an ad-hoc CUDA user object per resource at each node, along with its per-type heap-copy deleters.

Changes

• Slot-table infrastructure in resource_handles.{hpp,cpp} (OpaqueHandle, the slot table, lazy creation, graph_set_slot) with its Cython surface in _resource_handles.{pxd,pyx}.
• Graph nodes in _graph_node.pyx — kernel, event-record, event-wait, host-callback, memcpy, and memset — now store their owning handles in node slots. Stream-captured callbacks recover the just-captured host node from cuStreamGetCaptureInfo and use the same path; forked builders share the primary builder's graph handle, so their attachments land in the same table.
• Kernel nodes retain the kernel-argument tuple (slot 1) so the Python objects backing the arguments — notably device Buffers — outlive the graph. This is the slot-table port of the user-object fix from cuda.core: keep kernel-argument objects alive in graph kernel nodes #2041.
• GraphNode.memcpy/memset (and the GraphDefinition pass-throughs) now accept either a Buffer or a raw int address for each operand. A Buffer operand is retained at the allocation level (its DevicePtrHandle), so close()/reset cannot free memory the graph still references; a raw int behaves exactly as before (caller owns the lifetime), keeping the change backward compatible. Keyword-only dst_owner/src_owner arguments let callers attach an arbitrary owner to a raw-pointer operand; combining an owner with a Buffer operand is rejected.
• graph/_utils is renamed to graph/_host_callback now that it holds only host-callback machinery, and a shared _attach_host_callback_owners unifies the eager (GN_callback) and capture (add_callback) attachment paths.
• GraphBuilder.graph_definition exposes the captured graph as a GraphDefinition that shares ownership of the underlying CUgraph. State-guard rules: valid on a primary builder only after end_building(); valid on a conditional body both before begin_building() and after end_building(); never valid on a forked builder (access through the primary instead).
• A test-only _utils/_weak_handles module provides weak_handle() for deterministic, refcount-free lifetime assertions.

Stream-capture lifetime contract

Operations recorded during stream capture reference caller-owned memory and are not retained, unlike explicit GraphDefinition construction. Host callbacks are the one exception: they are retained on both the capture and explicit paths. This contract is documented on GraphBuilder.

Test coverage

• Slot-table lifetime tests for Buffer memcpy/memset operands (including clone), and for capture host callbacks retained after dropping their Python references.
• dst_owner/src_owner retention verified with weakrefs, plus rejection of Buffer+owner combinations.
• Device-allocation lifetime tests using weak_handle() to confirm an allocation survives reset/close while a graph still references it.
• graph_definition tests: happy path, both hybrid conditional-body flows (populate-via-explicit-API and capture-then-augment), the three error states (forked, capturing, primary pre-capture), and the shared-ownership guarantee (the GraphDefinition survives the builder's close()).

Related work

• Builds on the handle-layer plumbing from cuda.core: Cythonize GraphBuilder and Graph with handle-layer cleanup #2008.
• Ports and generalizes the kernel-argument user-object fix from cuda.core: keep kernel-argument objects alive in graph kernel nodes #2041.
• Completes step 3 of CUDA graph phase N - graph updates #1330.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[Andy-Jost]

/ok to test

[github-actions]

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-2280/
https://nvidia.github.io/cuda-python/pr-preview/pr-2280/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-2280/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-2280/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

[Andy-Jost]

/ok to test

[rwgk]

I'm trying to help a little bit using gpt-5.5 on my side.

It'd be great to get into the habit of letting our agents add the authorship markers, at least to agent-generated tests.

Findings

• P2: memset is no longer backward-compatible for existing positional height/pitch calls. Both GraphDefinition.memset(...) and GraphNode.memset(...) now insert * before height, so calls like g.memset(dst, value, width, height, pitch) start raising TypeError. That is a public API break unless intentional and documented. See cuda_core/cuda/core/graph/_graph_definition.pyx:158 and cuda_core/cuda/core/graph/_graph_node.pyx:285.
• P2: GraphBuilder.graph_definition returns a GraphDefinition wrapping an empty/reset graph handle after gb.close(). close() resets _h_graph and sets CLOSED, but the property does not call GB_check_open() or check CLOSED, so later nodes(), instantiate(), etc. hit CUDA with a null graph instead of raising a clear builder-closed error. This only affects accessing the property after close; a view obtained before close is intended to remain valid. See cuda_core/cuda/core/graph/_graph_builder.pyx:280 and cuda_core/cuda/core/graph/_graph_builder.pyx:339.
• P3: The newly added tests lack the repo’s explicit authorship markers. pytest.ini registers agent_authored, human_reviewed, and human_authored, and the repo guidance says newly added unit tests should carry one marker immediately above each test. This PR adds many tests without one, starting at cuda_core/tests/graph/test_graph_builder.py:295, cuda_core/tests/graph/test_graph_builder.py:444, and cuda_core/tests/graph/test_graph_definition_lifetime.py:634.