RestCatalog: concurrent HTTP requests can crash in libcurl #1089

HttpClient sends every request through one cpr::ConnectionPool (http_client.cc#L201). That pool wraps a libcurl share handle that shares the connection cache, and libcurl doesn't support using a shared connection cache from several threads at once. When two requests overlap, the process can crash.

A RestCatalog sends all its requests through one HttpClient, so requests can overlap even when the application only calls the catalog from one thread:

• Metrics reports are posted from the executor passed to RestCatalog::Make (rest_metrics_reporter.cc#L73-L74).
• OAuth2 tokens are refreshed on the auth_executor passed to RestCatalog::Make (auth_session_internal.h#L284).
• Vended S3 credentials are refreshed through LoadCredentials when the AWS SDK signs a request. The refresh runs on whichever thread is doing the S3 I/O (arrow_s3_file_io.cc#L701-L708, rest_catalog.cc#L91).

Applications that share one catalog between threads run into it directly.

The shared pool was introduced in #530, which replaced a mutex-guarded cpr::Session.

Reproduction

Run this against the REST fixture that rest_catalog_integration_test uses:

HttpClient client;
auto session = auth::AuthSession::MakeDefault({});
std::vector<std::thread> threads;
for (int i = 0; i < 32; ++i) {
  threads.emplace_back([&] {
    for (int j = 0; j < 200; ++j) {
      (void)client.Get("http://localhost:8181/v1/config", {}, {},
                       *DefaultErrorHandler::Instance(), *session);
    }
  });
}
for (auto& thread : threads) thread.join();

With libcurl 8.22.0, this aborted in 6 of 6 runs with heap corruption (free(): corrupted unsorted chunks or double free or corruption). Sharing one RestCatalog between 32 threads that each call LoadTable also crashes, with both 8.22.0 and 8.5.0. Neither reproduces with 8.18.0, the version CI uses, so CI doesn't catch it.