Skip to content

interpreter,cel/async: recover panics in async function goroutines - #1640

Open
itzPranshul wants to merge 1 commit into
cel-expr:masterfrom
itzPranshul:fix/async-panic-recovery
Open

itzPranshul wants to merge 1 commit into
cel-expr:masterfrom
itzPranshul:fix/async-panic-recovery

Conversation

@itzPranshul

Copy link
Copy Markdown

Summary

Fixes #1356 for asynchronous functions by recovering panics in framework-managed async goroutines and converting them into CEL types.Err values.

Previously, a panic inside an AsyncBinding or async.TimeoutBinding occurred on a spawned goroutine without a recover() block, crashing the Go process (outside ConcurrentEval's evaluator-goroutine recover()).

Key Changes

  • interpreter/async.go (asyncCallStateTracker.launch): Recover panics in the launched goroutine and record types.NewErr() before invoking observer.OnCallFinished and gate.Complete.
  • cel/async/async.go (Timeout): Recover panics in Timeout's inner goroutine and send types.NewErr() on resCh.
  • Tests: Added panic-recovery and graceful-degradation (||, &&) tests.

@TristonianJones

Copy link
Copy Markdown
Collaborator

/gcbrun

@TristonianJones TristonianJones left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fixes! Could you also add that a panic in an async function becomes a CEL error in the async doc comments? Currently panics in sync functions halt the program so it's a semantic difference that's worth mentioning.

Comment thread cel/async/async.go
go func() {
defer func() {
if r := recover(); r != nil {
resCh <- types.NewErr("internal error: %v", r)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you feel it would be useful to capture the panic stack trace or give more details about which call panicked?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Timeout only wraps a BlockingAsyncOp so it doesn't know the function/overload name here, though the interpreter already attaches the AST node ID via LabelErrNode and passes the function/overload to AsyncObserver.OnCallFinished

I skipped debug.Stack() on purpose so this stays consistent with Eval and ConcurrentEval

plus, types.Err values are often surfaced to end users in API/validation errors, where multi-KB stack traces bloat responses and expose internal file paths.

Comment thread cel/async/async_test.go

func TestTimeoutBindingPanicRecovery(t *testing.T) {
op := buildZeroArgAsync(t, async.TimeoutBinding(func(ctx context.Context, args ...ref.Val) ref.Val {
panic("timeout boom")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great for panics before the deadline. Could you add a test for a panic after the deadline as well? e.g. late timeout boom?

Comment thread cel/program_async_test.go
}, time.Second),
),
)
slowTrueOpt := cel.Function("slow_true",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider upgrading these tests to use synctest.Test to avoid flakiness in execution by using a fake clock.

Comment thread cel/program_async_test.go
),
)

tests := []struct {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure to run gofmt-clean to catch the indentation issue in the struct.

Convert panics in asyncCallStateTracker.launch and async.Timeout worker goroutines into types.Err results instead of crashing the process. This allows logical operators to degrade gracefully while ensuring observers and concurrency gates
still complete cleanly.
@itzPranshul
itzPranshul force-pushed the fix/async-panic-recovery branch from 2b68943 to 7ff6699 Compare November 14, 2026 05:29

@itzPranshul itzPranshul left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TristonianJones Thanks for the detailed review! I've addressed all your nits and updated the doc comments as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Graceful degradation on extension function panics

2 participants