[chrismccord]

Summary

With existing: true, start_child/3 and ensure_started_child/3 read the child's stored state before starting it, and turned any read error into {:error, :not_found}:

case StorageBackend.get_object(config.storage_backend, storage_key, consistent: existing) do
  {:ok, %{body: body, etag: etag}} -> ...
  {:error, _} -> {nil, nil}        # a 503, a timeout, a dropped connection...
end

if existing and stored_object == nil, do: {:error, :not_found}

check_existing/4 (the start_child path) did the same. The object-store backend tells these apart: a 404 is :not_found, every other failed response is :bad_gateway, and transport failures come back as their own reasons. But the supervisor collapsed them all. So a throttled or failed read told the caller the child doesn't exist.

get_server_info/2 did the same, and reads eventually consistent, so it can also miss state written moments ago. A caller that acts on that miss, for example by treating the server as gone, acts on a transient error.

Sprites calls ensure_started_child(..., existing: true) for every operation on a sprite whose owner isn't registered: after it idles down, after a deploy re-homes it, while a new owner's registration propagates. A 503 SlowDown there became a 404 "not found" for a sprite that exists. Clients treat that as "gone" rather than retrying.

Change

• One stored-object read, read_stored_object/4,5, used by both paths.
• Only the store saying the object doesn't exist is :not_found.
• A read error that may pass is retried: :bad_gateway (which covers 408/429/5xx), :timeout, :closed, :econnrefused, :econnreset, and Req/Mint transport errors. There are 3 attempts in all, with a 100–200 ms then 200–300 ms backoff, and a retry happens only if it fits in the caller's remaining :timeout. These retries sit above Req's own default GET retries.
• With existing: true, a read that still fails returns {:error, {:storage_unavailable, reason}} and starts nothing. An error that isn't retryable, such as an undecodable object, returns that at once.
• Without existing: true, an unreadable stored state is retried the same way and then starts as before: the child's init reads and claims its state itself, and the lock claim keeps a fresh start from replacing stored state.
• get_server_info/3 (was /2):
  • uses the same retries;
  • returns {:error, {:storage_unavailable, reason}} when the state can't be read, keeping {:error, :not_found} for a store miss;
  • reads the store's consistent view by default, since callers act on what it says and an eventually consistent read can miss state written moments ago;
  • takes consistent: false to opt back into an eventually consistent read. stream_all_server_info/1 uses that for its per-entry reads, because the listing behind it is eventually consistent already.
• :existing is now documented on both functions, and there's a changelog entry.

Testing

New test/durable_server/existing_read_error_test.exs uses an in-memory backend that fails reads on cue. It runs these cases for both start_child and ensure_started_child with existing: true:

• one :bad_gateway read is retried, and the stored server starts with its stored count;
• reads that keep failing with a transport timeout return {:error, {:storage_unavailable, reason}} after more than one read, and nothing is registered for the key;
• a non-retryable error returns storage_unavailable after exactly one read;
• a definite miss is :not_found after exactly one read.

For get_server_info, four more tests cover:

• a retried :bad_gateway;
• storage_unavailable after reads that keep failing;
• a miss read once;
• consistent by default, consistent: false on request, and eventually consistent reads from stream_all_server_info/1.

The error-path tests fail on main with {:error, :not_found}, and the option test fails because get_server_info/3 is undefined. The miss tests pass before and after.

mix test: 476 passed, 37 excluded. mix format --check-formatted and mix compile --warnings-as-errors are clean.

Callers

{:error, {:storage_unavailable, reason}} is a new return value from start_child/3, ensure_started_child/3 and get_server_info. Callers that match only {:ok, _} and {:error, :not_found} on get_server_info will raise CaseClauseError until they handle it, so update them when bumping.