fix(bigtable): report the real error behind session-path UNKNOWNs - #14349
fix(bigtable): report the real error behind session-path UNKNOWNs#14349mutianf wants to merge 4 commits into
Conversation
DivertingUnaryCallable.translateException is the only place the session path converts a failure into the caller's exception, and it defaulted to UNKNOWN for anything that was not a StatusException/StatusRuntimeException. A production incident surfaced application-visible UNKNOWN errors with no matching UNKNOWN in CSM or on the server, and the bare status code left nothing to diagnose from. Three changes: - Walk the full cause chain when looking for a gRPC status. Unwrapping previously stopped at CompletionException/ExecutionException, so a valid StatusRuntimeException wrapped in any other type lost its code and was reported as UNKNOWN. Bounded at depth 32 with a self-cycle guard. - Map CancellationException to CANCELLED, matching csm.attributes.Util#extractStatus. The two mappings disagreed, so the same failure could be CANCELLED in metrics and UNKNOWN to the caller. - When the fall-through to UNKNOWN is genuine, name the throwable. The message now carries the cause-chain class names and the underlying message, and the first occurrence per callable is logged at WARNING with a full stack (FINE thereafter, since a storm is exactly when this fires most and an unconditional WARNING would flood the log). Adds DivertingUnaryCallableTest covering the status mapping, and SessionPathErrorEscapeTest, which injects faults at the seam between SessionPoolMap.apply and the session machinery to establish which throw sites escape synchronously (metrics.newTableTracer does) and which are converted to a status inside the op chain first (SessionPool.newCall is caught by RetryingVRpc.start and becomes CANCELLED).
There was a problem hiding this comment.
Code Review
This pull request enhances exception translation in DivertingUnaryCallable by walking the cause chain of throwables to extract gRPC status codes, logging unrecognized errors with their cause chains to aid diagnostics. It also introduces comprehensive tests in SessionPathErrorEscapeTest and DivertingUnaryCallableTest to validate error propagation. The review feedback suggests moving the CancellationException check inside the cause-walking loop to correctly identify wrapped cancellations, and adding a null check in describeUnrecognized to prevent a potential NullPointerException.
| private static Status.Code findStatusCode(Throwable t) { | ||
| if (t instanceof CancellationException) { | ||
| return Status.Code.CANCELLED; | ||
| } | ||
| if (cause instanceof StatusException) { | ||
| code = ((StatusException) cause).getStatus().getCode(); | ||
| Throwable current = t; | ||
| for (int depth = 0; current != null && depth < MAX_CAUSE_DEPTH; depth++) { | ||
| if (current instanceof StatusRuntimeException) { | ||
| return ((StatusRuntimeException) current).getStatus().getCode(); | ||
| } | ||
| if (current instanceof StatusException) { | ||
| return ((StatusException) current).getStatus().getCode(); | ||
| } | ||
| Throwable next = current.getCause(); | ||
| if (next == current) { | ||
| break; // self-referential chain | ||
| } | ||
| current = next; | ||
| } | ||
| return null; | ||
| } |
There was a problem hiding this comment.
The current implementation of findStatusCode only checks if the top-level throwable t is an instance of CancellationException. If a CancellationException is wrapped inside another exception (such as a RuntimeException or custom wrapper), it will not be detected, and the method will return null (which ultimately maps to UNKNOWN instead of CANCELLED).
Moving the CancellationException check inside the cause-walking loop ensures that wrapped cancellations are correctly identified and mapped to CANCELLED, matching the behavior of StatusRuntimeException and StatusException.
| private static Status.Code findStatusCode(Throwable t) { | |
| if (t instanceof CancellationException) { | |
| return Status.Code.CANCELLED; | |
| } | |
| if (cause instanceof StatusException) { | |
| code = ((StatusException) cause).getStatus().getCode(); | |
| Throwable current = t; | |
| for (int depth = 0; current != null && depth < MAX_CAUSE_DEPTH; depth++) { | |
| if (current instanceof StatusRuntimeException) { | |
| return ((StatusRuntimeException) current).getStatus().getCode(); | |
| } | |
| if (current instanceof StatusException) { | |
| return ((StatusException) current).getStatus().getCode(); | |
| } | |
| Throwable next = current.getCause(); | |
| if (next == current) { | |
| break; // self-referential chain | |
| } | |
| current = next; | |
| } | |
| return null; | |
| } | |
| private static Status.Code findStatusCode(Throwable t) { | |
| Throwable current = t; | |
| for (int depth = 0; current != null && depth < MAX_CAUSE_DEPTH; depth++) { | |
| if (current instanceof StatusRuntimeException) { | |
| return ((StatusRuntimeException) current).getStatus().getCode(); | |
| } | |
| if (current instanceof StatusException) { | |
| return ((StatusException) current).getStatus().getCode(); | |
| } | |
| if (current instanceof CancellationException) { | |
| return Status.Code.CANCELLED; | |
| } | |
| Throwable next = current.getCause(); | |
| if (next == current) { | |
| break; // self-referential chain | |
| } | |
| current = next; | |
| } | |
| return null; | |
| } |
| private static String describeUnrecognized(Throwable cause) { | ||
| StringBuilder chain = new StringBuilder(); | ||
| Throwable current = cause; |
There was a problem hiding this comment.
To prevent a potential NullPointerException if cause is null, we should add a null check at the beginning of describeUnrecognized before calling cause.getMessage().
private static String describeUnrecognized(@Nullable Throwable cause) {
if (cause == null) {
return "Session operation failed with a null error; reporting UNKNOWN.";
}
StringBuilder chain = new StringBuilder();
Throwable current = cause;Review feedback on googleapis#14349: - findStatusCode checked CancellationException only at the top level, so a cancellation wrapped in anything other than Completion/ExecutionException fell through to UNKNOWN -- the same defect the chain walk exists to fix. Moved into the loop, after the Status checks so outermost still wins. - describeUnrecognized dereferenced a possibly-null cause. Unreachable today, but an NPE raised while building the error message would destroy exactly the diagnostic the message carries.
Review feedback on googleapis#14349: - IllegalStateException now maps to INTERNAL and RejectedExecutionException to RESOURCE_EXHAUSTED, rather than both falling through to UNKNOWN. UNKNOWN is left for types that genuinely say nothing. A carried Status still wins over an inferred code; within each category the outermost match wins. - MAX_CAUSE_DEPTH 32 -> 8. - Corrected the CSM claims in the comments. CSM takes its session-path status from VRpcResult, never from translateException's ApiException, so nothing here changes what CSM records -- and for the throwables that reach this code CSM has either no record of the operation or a recorded success. - Dropped tracerConstructionThrow_escapesReadRowSynchronously. - Removed section banners, gave every test a "// Verifies ..." lead, and dropped the references to production. - Documented that the SessionList race is injected rather than raced for, and that a missing row does not reach UnaryResponseFuture's OK-without-message branch (the server sends a response with `row` unset, which ReadRowShim turns into a null row), so throwing there is right. - sessionPoolNewCallThrow_isConvertedToCancelled now also asserts the status CSM recorded, not just that it recorded something.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request enhances exception translation in DivertingUnaryCallable by walking the cause chain of throwables to extract or infer gRPC status codes, and adds comprehensive tests to validate these mappings. The review feedback suggests a performance optimization to pre-compute the statusless error message once, avoiding redundant cause-chain traversals during logging and exception creation.
| Status.Code inferred = classifyStatuslessCause(cause); | ||
| reportStatusless(cause, inferred); | ||
| return ApiExceptionFactory.createException( | ||
| describeStatusless(cause, inferred), e, GrpcStatusCode.of(inferred), false); |
There was a problem hiding this comment.
The describeStatusless method is called multiple times for the same exception (once for logging and once for creating the exception). We can avoid redundant string construction and cause-chain traversal by computing the message once and passing it to reportStatusless.
Status.Code inferred = classifyStatuslessCause(cause);
String message = describeStatusless(cause, inferred);
reportStatusless(message, cause);
return ApiExceptionFactory.createException(
message, e, GrpcStatusCode.of(inferred), false);| private void reportStatusless(@Nullable Throwable cause, Status.Code reported) { | ||
| if (loggedStatusless.compareAndSet(false, true)) { | ||
| LOGGER.log(Level.WARNING, describeStatusless(cause, reported), cause); | ||
| } else if (LOGGER.isLoggable(Level.FINE)) { | ||
| LOGGER.log(Level.FINE, describeStatusless(cause, reported), cause); | ||
| } | ||
| } |
There was a problem hiding this comment.
Update reportStatusless to accept the pre-computed message string directly, avoiding redundant calls to describeStatusless.
| private void reportStatusless(@Nullable Throwable cause, Status.Code reported) { | |
| if (loggedStatusless.compareAndSet(false, true)) { | |
| LOGGER.log(Level.WARNING, describeStatusless(cause, reported), cause); | |
| } else if (LOGGER.isLoggable(Level.FINE)) { | |
| LOGGER.log(Level.FINE, describeStatusless(cause, reported), cause); | |
| } | |
| } | |
| private void reportStatusless(String message, @Nullable Throwable cause) { | |
| if (loggedStatusless.compareAndSet(false, true)) { | |
| LOGGER.log(Level.WARNING, message, cause); | |
| } else if (LOGGER.isLoggable(Level.FINE)) { | |
| LOGGER.log(Level.FINE, message, cause); | |
| } | |
| } |
Problem
DivertingUnaryCallable.translateExceptionis the only place the session path converts a failure into the caller's exception, and it defaulted toUNKNOWNfor anything that is not aStatusException/StatusRuntimeException.A production incident surfaced application-visible
UNKNOWNerrors with no matching UNKNOWN in CSM and none on the server. Because the default discarded the status and said nothing about the throwable, the bare code left nothing to diagnose from — the client's last chance to report what actually failed was spent emitting three uninformative characters.Changes
Walk the full cause chain when looking for a gRPC status. Unwrapping stopped at
CompletionException/ExecutionException, so a validStatusRuntimeExceptionwrapped in any other type lost its code and was reported asUNKNOWN. Bounded at depth 32 with a self-cycle guard.Map
CancellationExceptiontoCANCELLED, matchingcsm.attributes.Util#extractStatus. The two mappings disagreed, so the same failure could beCANCELLEDin metrics andUNKNOWNto the caller.When the fall-through to
UNKNOWNis genuine, name the throwable:The first occurrence per callable is logged at
WARNINGwith a full stack,FINEthereafter — a storm is exactly when this fires most, and an unconditionalWARNINGwould flood the log at the moment an operator can least afford it.Reviewer notes
This changes observable status codes, not just messages: failures that were
UNKNOWNonly because their status sat one wrapper too deep now report their real code, and cancellations becomeCANCELLED. That is the intent, but it is user-visible. Blast radius is limited —ApiExceptionFactory.createException(..., false)hard-codesretryable=false, so gax retry behavior does not key off this.Deliberately not included: no counter was added to
SessionPoolMap.apply, since every throw it catches funnels throughtranslateExceptionanyway and the logging covers it without threading aMetricsdependency through a new layer.UnaryResponseFuture's OK-without-messageIllegalStateExceptionis also left alone — the new message names it exactly if it ever fires, and changing a status on a defensive "can't happen" branch is a separate decision.Tests
DivertingUnaryCallableTest(14 tests) covers the mapping: which throwables becomeUNKNOWN, that Status-bearing ones keep their code through arbitrary wrappers, theCancellationExceptioncase, the self-referential cause chain, and the message content — including the null-messageNullPointerException, where a barecause.getMessage()would have given an operator literally nothing.SessionPathErrorEscapeTest(4 tests) injects faults at the seam betweenSessionPoolMap.applyand the session machinery to establish which throw sites can produce an applicationUNKNOWNat all:metrics.newTableTracerthrowsTableBase.readRowsynchronously; listener never notified; zero CSM recordsSessionPool.newCallthrowsRetryingVRpc.startconverts it toCANCELLED, recorded in CSMIllegalStateExceptionwhile CSM and the server both record OKThe second row is worth calling out: it rules the
SessionListclose/drain race out as a source of applicationUNKNOWN, since it would appear asCANCELLEDin both places.Full
data.v2.internalsuite passes (1021 tests).