[FEA] cuda.core: support stream re-capture into an existing graph (CUDA 13.3)

[leofang]

Summary

CUDA 13.3 adds
cuStreamBeginRecaptureToGraph(CUstream hStream, CUstreamCaptureMode mode, CUgraph hGraph, CUgraphRecaptureCallback callbackFunc, void* userData):
begin a stream capture that re-captures into an existing graph instead of creating a new
one. The registered callback receives, per recaptured node,
(void* data, CUgraphNode node, const CUgraphNodeParams* originalParams, const CUgraphNodeParams* recaptureParams, CUgraphRecaptureStatus status)
with status CU_GRAPH_RECAPTURE_{ELIGIBLE_FOR_UPDATE,INELIGIBLE_FOR_UPDATE,ERROR} (exact
invocation semantics to be confirmed against the 13.3 driver docs during design). Failures
surface as the new CUDA_ERROR_GRAPH_RECAPTURE_FAILURE.

This is update-by-recapture: rerun the capture-producing code and let the driver map it onto
the existing graph — complementing the setter-based node updates of #2352 / #2354 and the
general graph-updates task #1330. cuda.core's graph support is capture-based
(GraphBuilder), so this slots in naturally.

Underlying C APIs to cover

Symbol	Purpose
cuStreamBeginRecaptureToGraph(stream, mode, graph, callback, userData)	begin re-capture into an existing CUgraph
CUgraphRecaptureCallback	per-node hook comparing original vs. recaptured CUgraphNodeParams
CUgraphRecaptureStatus	ELIGIBLE_FOR_UPDATE / INELIGIBLE_FOR_UPDATE / ERROR
CUDA_ERROR_GRAPH_RECAPTURE_FAILURE	new error code to map in cuda.core error handling

Design sketch (draft — needs design-meeting review)

Important

Starting point only, not a settled design — review in the cuda.core design meeting.

gb = graph.recapture(stream, mode="global")   # names TBD; wraps cuStreamBeginRecaptureToGraph
launch(stream, config, kernel, new_args)      # replay the capture region
gb.end()                                      # graph updated in place

Graph construction is the one flow where cuda.core deliberately uses a builder, so returning a
(re)capture-scoped GraphBuilder mirrors the existing API shape.

Open questions for the meeting:

1. The C callback fires from within capture — exposing an optional Python hook means calling
   back into Python mid-capture (GIL/perf story). Verify whether a NULL callback is accepted
   and make the Python hook opt-in.
2. Should cuda.core collect and report per-node recapture statuses (updated vs. ineligible)
   after end()?
3. Interaction with GraphBuilder's existing capture-state guards (is_…capturing,
   conditional handles).
4. Version gating: 13.3+ only; actionable error otherwise.

References

• Driver docs (Graph Management): https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__GRAPH.html
• Found during the CUDA 12.8 → 13.3 bindings vs. cuda.core gap sweep (2026-07-14)

-- Leo's bot

[Andy-Jost]

Updated design sketch

CUDA 13.3 added stream recapture for an existing CUgraph. Recapture updates the graph definition while it preserves node identity.

This feature fits the existing GraphBuilder state machine. It does not update an executable graph. The user must call Graph.update(builder) separately.

State machine

• Add a repeatable CAPTURE_ENDED -> RECAPTURING -> CAPTURE_ENDED transition.
• Add begin_rebuilding(mode="relaxed") to support recapture from the CAPTURE_ENDED state.
• Use the existing end_building() method to end recapture.
• is_building returns True during capture and recapture.
• Do not expose the distinction between capture and recapture.
• Treat RECAPTURING as an active capture during cleanup.
• Do not adopt the graph returned by end_building(), because it is the existing graph.

Failure handling

• A recapture failure destroys the target CUgraph.
• Invalidate the shared graph state to prevent a second destroy operation.
• Add an invalid state that is distinct from CLOSED.
• Preserve the first recapture error and report it from end_building().

Attachments

• Recapture releases the CUDA user objects attached to the graph.
• Clear the matching cuda.core attachment records when recapture starts.
• Rebuild attachments as each operation is captured again.

Scope

• Existing GraphNode objects remain valid after a successful recapture.
• The operation order, node types, and dependency sets must match the original capture.
• The driver rejects child graphs, conditional nodes, and several explicit node types.
• Reject embed, conditional methods, and split during recapture.
• Add split support in a follow-up change.
• Changes to allocation or free nodes can cause a hard failure.
• Explicit changes through builder.graph_definition can make recapture unsupported.

API

• Reuse the stream that belongs to the builder.
• Allow recapture only on a completed primary builder.
• Require CUDA driver and binding version 13.3 or later.
• Keep Graph.update(builder) as a separate operation.
• Do not add recapture support to GraphDefinition.

Open question: callback support

The CUDA callback reports whether each changed node is eligible for executable-graph update. It can also stop recapture, but it cannot change node parameters.

We need user requirements before we select a Python API. Possible designs include:

• Do not expose callback information.
• Use a private C callback and return a status report after recapture.
• Use a private C callback to raise an exception at the operation that cannot be updated.
• Expose a public callback for advanced users.

A public Python callback can reenter the CUDA driver from inside a driver call. This creates an undocumented lock risk. Any public callback design must address this risk.

[carterbox]

@Andy-Jost asked in the meeting today whether this method should be unique or if we should just reuse the (begin|end)_building() methods.

The node creation order while recapturing the graph must be identical to the original graph. The recapture will fail immediately for: * Topology mismatches between the existing graph and the recaptured graph * Parameter mismatches for memory allocation or free nodes

Because the semantics of this method are to only update an existing graph with new parameters and trying to change the graph structure causes an error, I think it makes sense to have a separate method because creating a new thing and updating an existing thing are conceptually different.