feat(io): refresh vended storage credentials per prefix

[plusplusjiajia]

Vended storage credentials can expire while an ArrowS3FileIO instance and its
file handles are still in use. This change refreshes them on demand and keeps
the refresh state scoped to each storage prefix.

The implementation:

• Keeps one CredentialStore for credential snapshots, per-prefix slots,
  expiry state, refresh coordination, backoff, and generation checks.
• Resolves each credential once into its normalized prefix, effective client
  properties, client settings, and AWS credentials.
• Refreshes credentials independently by prefix. A response may update the
  prefixes that are still active; invalid, expired, missing, or incompatible
  entries keep their previous state.
• Preserves S3-compatible schemes during routing. For example, s3:// and
  oss:// remain separate prefixes so credentials for one endpoint cannot be
  sent to another.
• Lets existing AWS SDK clients pick up refreshed credentials through
  AWSCredentialsProvider, while keeping clients and handles safe when the
  FileIO or a credential generation is replaced.
• Uses the REST client connection and request timeout properties for all HTTP
  methods. Transport failures during table creation or commit are reported as
  CommitStateUnknown because the server may already have applied the write.

The change removes the previous global expiry/layout bookkeeping and duplicate
credential-property parsing, making the refresh path easier to maintain and
extend while preserving the existing StorageCredentialProvider API.

Tests cover prefix routing, refresh timing and concurrency, failed and invalid
refreshes, reordered responses, expiry handling, REST client timeouts, and
real S3-compatible round trips.

[wgtmac]

Thanks for adding this feature! I think it is too large to review which may incur a long delay. Perhaps let's split it into smaller ones so we can review it one by one?

[plusplusjiajia]

Thanks for adding this feature! I think it is too large to review which may incur a long delay. Perhaps let's split it into smaller ones so we can review it one by one?

@wgtmac Thanks — split into a stack, smallest first:

1. test(io): stop probing the EC2 metadata service in S3 tests #897 — S3 tests stop probing the EC2 metadata service (54s → 0.3s).
2. refactor(io): hand out S3 delegates by shared_ptr and build them off the lock #898 — ArrowS3FileIO gets refactor(io): make FileIO resolution registry-driven #889's concurrency treatment. No behavior change.
3. feat(rest): load table credentials from the LoadCredentials endpoint #899 (draft until refactor(io): hand out S3 delegates by shared_ptr and build them off the lock #898 merges) — LoadCredentials plumbing, not yet invoked.
4. This PR shrinks to just the refresh policy.

Each builds and passes the full suite standalone; the diff here narrows as each one merges.

[wgtmac]

Is this ready for review?

[plusplusjiajia]

Is this ready for review?

@wgtmac Thanks! Not yet. #899 is ready for review now, and this one is stacked on it, so I'll mark it ready once #899 lands.

[plusplusjiajia]

@wgtmac Thanks for the provider refactor in #899! Rebased on main and adapted the refresh policy and tests to StorageCredentialProvider and InitializeStorageCredentials. Ready for review.

[wgtmac]

Could you rebase onto main and resolve the conflicts?

[plusplusjiajia]

@wgtmac Thanks! Rebased onto main. The first commit is the previous one rebased; the second addresses this round. Since #783 made Get virtual, the timeout is a parameter without a default, so callers and the two mocks pass it. A refresh now only rotates keys, which keeps it simple and safe to run from a request.

[wgtmac]

Thanks for the continued optimization on this PR. I pushed one cleanup commit after another review of the implementation against the Java behavior and REST spec.
The goal of this change is to make the design simpler and easier to maintain. In particular, I:

• consolidated credential resolution and per-prefix refresh state in CredentialStore;
• removed the duplicated layout and global expiry tracking;
• kept different S3-compatible schemes routed independently;
• reused the REST client timeout settings for all HTTP methods;
• handled transport failures during commits as CommitStateUnknown.

These changes are intended to preserve the current behavior while reducing duplicated state and making future extensions safer. I also cross-checked the design with several AI-assisted review suggestions, Java implementation details, and the REST spec. Thanks again for continuously refining the PR!