Repository navigation
Make a best effort rebuild Apache client/pool after java.lang.Error - #7395
Draft
alextwoods wants to merge 3 commits into
Draft
alextwoods wants to merge 3 commits into
alextwoods wants to merge 3 commits into
Conversation
alextwoods
marked this pull request as draft
September 24, 2026 17:32
alextwoods
force-pushed
the
alexwoo/apache_panic
branch
from
October 15, 2026 21:50
7c9dffe to
49c6f57
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Apache deliberately shuts its connection manager down when a java.lang.Error escapes a request (HTTPCLIENT-1924), which left the SDK's Apache clients permanently unusable. An OutOfMemoryError raised by unrelated application code while a request thread was inside the HTTP client took the shared pool down for the whole process: the application recovered, but every later SDK call failed with IllegalStateException: Connection pool shut down until the JVM was restarted. Apache 5.x behaves identically, so migrating to apache5-client did not help.
RecreatingHttpClientConnectionManager now wraps the pool in both modules. It honours Apache's intent by discarding the affected pool rather than resurrecting it, then builds a replacement on the next request. Throwing the pool away also disposes of the connection leases Apache's Error path never released, so the replacement starts at full capacity. An intentional close() goes through closePermanently() instead, so closing a client stays permanent.
Connections leased from a replaced pool are safe to release into the replacement: connect and routeComplete are entry-local, and AbstractConnPool.release (StrictConnPool.release in 5.x) is guarded by leased.remove(entry) and ignores an entry it never leased.
Pool metrics now report through the wrapper so they follow the current pool. Apache 5.x declares closeIdle on the concrete pooling manager rather than the interface, so IdleConnectionReaper reaps through a new IdleConnectionCloser interface, letting it hold one stable registration for the client's lifetime even when the pool underneath is replaced.
Verified with a repro harness that drives a real OutOfMemoryError through Apache's handler: previously 6/13 (apache) and 2/4 (apache5) runs left a dead pool, now 0, with no Connection pool shut down occurrences and leases correctly returned.
Summary
When a
java.lang.Errorsuch as anOutOfMemoryErrorescapes a request, Apache'sMainClientExeccatches it and destroys the shared connection manager (connManager.shutdown()in 4.x,close(CloseMode.IMMEDIATE)in 5.x) before rethrowing — which used to leave the SDK's client permanently dead, failing every later call withIllegalStateException: Connection pool shut downeven after the application itself recovered.ApacheHttpClient/Apache5HttpClientnow hand Apache aRecreatingHttpClientConnectionManager(one per module, ininternal.conn) instead of thePoolingHttpClientConnectionManagerdirectly, built from a supplier so it can make more pools. It treats Apache's shutdown as "discard and rebuild later": it honors the destruction (post-Errorconnection state is untrustworthy, and throwing the pool away also disposes of the leases Apache'sErrorpath never released), flips to aNEEDS_RECREATEstate, and lazily builds a fresh pool on the next lease — so an idle client is never resurrected, and metrics and theIdleConnectionReaperfollow the current pool without rebuilding one. An intentionalclose()routes through a separateclosePermanently()into a terminalCLOSEDstate, which is what keeps closing a client permanent even though Apache and the SDK otherwise call the same method. Two edges matter: a request that read the old pool moments before it died retries once on the replacement (theIllegalStateExceptionApache raises there is not retryable by the SDK), and in 5.x a connection returned to a replaced pool is rejected rather than silently ignored as in 4.x, so that specific exception is swallowed once a rebuild has happened. Requests genuinely in flight when theErrorlands still fail, but with retryableIOExceptions rather than a permanently broken client.Motivation and Context
Modifications
Testing
Screenshots (if appropriate)
Types of changes
Checklist
mvn installsucceedsscripts/new-changescript and following the instructions. Commit the new file created by the script in.changes/next-releasewith your changes.License