Repository navigation
interpreter,cel/async: recover panics in async function goroutines - #1640
itzPranshul wants to merge 1 commit into
Conversation
|
/gcbrun |
TristonianJones
left a comment
There was a problem hiding this comment.
Thanks for the fixes! Could you also add that a panic in an async function becomes a CEL error in the async doc comments? Currently panics in sync functions halt the program so it's a semantic difference that's worth mentioning.
| go func() { | ||
| defer func() { | ||
| if r := recover(); r != nil { | ||
| resCh <- types.NewErr("internal error: %v", r) |
There was a problem hiding this comment.
Do you feel it would be useful to capture the panic stack trace or give more details about which call panicked?
There was a problem hiding this comment.
Timeout only wraps a BlockingAsyncOp so it doesn't know the function/overload name here, though the interpreter already attaches the AST node ID via LabelErrNode and passes the function/overload to AsyncObserver.OnCallFinished
I skipped debug.Stack() on purpose so this stays consistent with Eval and ConcurrentEval
plus, types.Err values are often surfaced to end users in API/validation errors, where multi-KB stack traces bloat responses and expose internal file paths.
|
|
||
| func TestTimeoutBindingPanicRecovery(t *testing.T) { | ||
| op := buildZeroArgAsync(t, async.TimeoutBinding(func(ctx context.Context, args ...ref.Val) ref.Val { | ||
| panic("timeout boom") |
There was a problem hiding this comment.
This is great for panics before the deadline. Could you add a test for a panic after the deadline as well? e.g. late timeout boom?
| }, time.Second), | ||
| ), | ||
| ) | ||
| slowTrueOpt := cel.Function("slow_true", |
There was a problem hiding this comment.
Consider upgrading these tests to use synctest.Test to avoid flakiness in execution by using a fake clock.
| ), | ||
| ) | ||
|
|
||
| tests := []struct { |
There was a problem hiding this comment.
Make sure to run gofmt-clean to catch the indentation issue in the struct.
Convert panics in asyncCallStateTracker.launch and async.Timeout worker goroutines into types.Err results instead of crashing the process. This allows logical operators to degrade gracefully while ensuring observers and concurrency gates still complete cleanly.
2b68943 to
7ff6699
Compare
itzPranshul
left a comment
There was a problem hiding this comment.
@TristonianJones Thanks for the detailed review! I've addressed all your nits and updated the doc comments as well.
Summary
Fixes #1356 for asynchronous functions by recovering panics in framework-managed async goroutines and converting them into CEL
types.Errvalues.Previously, a panic inside an
AsyncBindingorasync.TimeoutBindingoccurred on a spawned goroutine without arecover()block, crashing the Go process (outsideConcurrentEval's evaluator-goroutinerecover()).Key Changes
interpreter/async.go(asyncCallStateTracker.launch): Recover panics in the launched goroutine and recordtypes.NewErr()before invokingobserver.OnCallFinishedandgate.Complete.cel/async/async.go(Timeout): Recover panics inTimeout's inner goroutine and sendtypes.NewErr()onresCh.||,&&) tests.