Fix issue with file sync post container restart

.changeset/retry-computerd-restarts.md

@@ -0,0 +1,5 @@
+---
+"@cloudflare/computer": patch
+---
+
+`container-shell` operations now reconnect after computerd restarts when retrying is safe, and process-local execution handles return `EEXEC_LOST` after container replacement. See [container connection recovery](https://github.com/cloudflare/computer/blob/main/docs/05_runtime_interface.md#command-synchronization).

docs/02_sync_protocol.md

@@ -267,11 +267,8 @@ edited file) shows up exactly once on the wire. See
 
 ## Failure handling
 
-- **Container restart mid-exec.** The DO's connection detects the
-  closed WebSocket and self-destructs. The next call transparently
-  rebuilds against the still-running `computerd` (or restarts it if needed).
-  `pushRev` and the fetch cursor mean the catch-up is incremental, modulo
-  whatever the container's deployment chose for its DB lifetime.
+- **Container restart or transport loss.** The Durable Object detects the closed WebSocket, removes and closes the stale backend handle, and reconnects through the normal computerd health gate. `pushOnce` and `pullOnce` get one reconnect retry inside the original logical operation. `pushRev` and the fetch cursor make those retries idempotent: a torn push replays entries whose watermark did not advance, while a torn pull resumes after its last committed batch. A fresh process-lifetime computerd database resets the matching local watermarks before the replacement handle is exposed, so the next push rebuilds it from a rev-0 baseline.
+- **Container restart during command execution.** The pre-exec push must succeed, including its reconnect retry, before `shell.exec` is called. A locally disposed stub proves that a spawn request was never sent and permits one reconnect retry. A generic disconnect after dispatch is ambiguous, so the backend invalidates the handle and reports that the command may have started instead of replaying it. Event-stream failures follow the same no-replay rule. The post-command pull remains safe to retry only against the runtime UUID that ran the command. If reconnect reaches a replacement container, the execution result reports pending sync rather than treating its empty VFS as a successful zero-entry pull. The durable retry intent retains the original runtime UUID. When a scheduled retry confirms that runtime is gone, it reports the sync as lost and clears the unrecoverable intent so later commands on the live runtime can schedule their own pending pulls.
 - **Container crash mid-apply.** `push` is atomic from the DO's
   perspective on the receiver: the server wraps the whole batch in a
   single `db.transactionSync` via the synchronous `applyChangesSync`

docs/05_runtime_interface.md

@@ -98,11 +98,15 @@ push → spawn → events/result → pull
 
 A backend with `sync: "none"`, such as `worker-shell`, shares the host store and reports zero push/pull counts. A Container has its own VFS and synchronizes changes before and after command execution. Fully draining either `result()` or the event stream completes the post-command pull before the stream closes.
 
+The pre-command push is a safety gate, not a best-effort optimization. The push and spawn use the same backend handle. If that handle fails before dispatch, the reconnect retry repeats both steps on the replacement container so the command cannot skip its push. If the push still fails, `exec()` rejects before the spawn request is sent; `pushed: 0` means a successful push found no entries, not that synchronization failed. A failed post-command pull does not change the completed command result. The pull is fenced to the runtime UUID that ran the command, so reconnecting to an empty replacement cannot report a clean zero-entry sync. It reports `sync.status: "pending"` and persists that UUID with the retry intent. `retryPendingSync()` resumes against the original runtime when possible; if that runtime was replaced, it clears the unrecoverable intent and reports `status: "lost"` so a later command can schedule sync for the live runtime.
+
+Container connection failures also get one backend-internal reconnect attempt. Sync calls are safe to repeat. `getExec`, `killExec`, and `disposeExec` are retried only when the new connection reaches the same computerd runtime; a replacement container returns `EEXEC_LOST` instead of applying an old execution id to its new process table. `shell.exec` is different: the backend retries it only when connection setup failed or a locally disposed stub proves that no request was sent. If the transport fails after computerd may have accepted the spawn, the error states that the command may have started and the backend does not replay it. A failure while reading the event stream also invalidates the connection without rerunning the command.
+
 Module backends use host capability calls against the authoritative Workspace and therefore require no push/pull round trip.
 
 ## Lifecycle differences
 
-`container-shell` provides computerd's retained process log, replay, signals, and disposal.
+`container-shell` provides computerd's retained process log, replay, signals, and disposal. Execution handles are scoped to the UUID of the container process that accepted them, so they cannot target a reused execution id after process replacement. The latest UUID owner for each execution ID is stored in Workspace SQLite, with a bounded in-memory LRU cache, so direct by-ID operations preserve the fence across Durable Object incarnation and cache eviction.
 
 `worker-javascript` provides a Workspace-owned execution journal, retained result/events, host cancellation, and explicit disposal. Active Workers cannot be serialized across host restart; orphaned running records are reconciled to failed.
 

docs/07_injected_service.md

@@ -144,12 +144,8 @@ Sharp edges actually present in `cloudflare-container.ts`:
 
 - `#armUpgrade` must be set up *before* `#postConnect`, because `computerd`
   can dial back before the `POST /connect` response returns.
-- A `#monitoring` flag watches container exit and drops the cached
-  handle so the next call rebuilds from scratch.
-- **No transparent reconnect after a mid-session drop.** If the
-  WebSocket dies, the caller is expected to reconstruct the
-  `Workspace` rather than the backend trying to splice a new socket
-  into the existing session.
+- The container host records each monitored generation's exit reason. The dead container closes its WebSocket, and `fetchPort()` also short-circuits later requests with a transport error; either path invalidates the matching Workspace handle.
+- **Reconnect replaces the whole session.** If the WebSocket dies, `Workspace` invalidates and closes the matching backend handle, then calls `CloudflareContainerBackend.connect()` again. The replacement runs the complete start, egress-interception, health, `/connect`, and reverse-WebSocket sequence; the backend never splices a new carrier into the dead capnweb session. Replay-safe sync and process lifecycle operations get one retry. Command spawn is retried only when no request was dispatched.
 
 ## Environment variables
 
@@ -196,17 +192,9 @@ Today:
 
 ## Lifetime
 
-The `computerd` process is long-lived and outlives DO restarts — the
-sandbox container is reaped only when its lifetime policy says so,
-and a fresh DO incarnation reconnects to the same running daemon over
-a new WebSocket (the Cloudflare backend's `#monitoring` flag drops the
-cached handle if the container itself exits, forcing a rebuild).
-
-Caveat: **no on-disk persistence yet** (`packages/computerd/README.md`).
-The "same in-memory VFS across DO restarts" picture only holds while
-the container process is alive. A container restart loses VFS state;
-sync via `UPSTREAM_URL` is what brings state back across container
-restarts.
+The `computerd` process is long-lived and outlives Durable Object restarts — the sandbox container is reaped only when its lifetime policy says so, and a fresh Durable Object incarnation reconnects to the same running daemon over a new WebSocket. The container monitor and transport error classifier drop stale handles so an operation can reconnect through the readiness gate.
+
+Caveat: **no on-disk persistence yet** (`packages/computerd/README.md`). The same in-memory VFS across Durable Object restarts only holds while the container process is alive. A container restart loses VFS state. Watermark reconciliation and the next push rebuild the mirror from Durable Object storage, but reconnect cannot recover container-local files that were never pulled before the process died.
 
 ## Open questions
 

docs/11_lifecycle.md

@@ -84,6 +84,9 @@ an incarnation boundary. What survives is:
   container-side cursor the DO has fetched). These are written via the
   same SQLite transaction as the data they describe, so they cannot
   drift out of sync with the store.
+- The latest container runtime UUID for each execution ID. This lets a
+  reconstructed Workspace reject stale get, kill, or dispose calls before
+  they can reach a replacement container that reused the same ID.
 
 On every new incarnation `Workspace.ready()` re-runs `#connect()`,
 which re-enters the backend's bootstrap sequence. If the container is
@@ -173,14 +176,11 @@ package WebSocket and require the `computerd` process to be live.
 3. **Death.** The WebSocket closes (clean or RST). capnweb errors
    every pending answer. The session is unrecoverable.
 
-The death case today is **not handled** — the `Workspace` keeps its
-`#handle` reference pointing at the dead session, and the next RPC
-call throws. The caller is expected to reconstruct the workspace.
+Session death is handled at the `Workspace` backend boundary. A close event, capnweb `onRpcBroken` callback, container exit, or classified transport error removes and closes the matching handle. The original operation gets one reconnect retry when replay is safe; the replacement handle is not exposed until computerd passes its health check and `reconcileWatermarks()` has compared the two stores.
 
 ### What an in-flight RPC looks like across a transport failure
 
-Because the rev counters drive every operation, a torn RPC is safe
-to retry against a fresh session. Specifically:
+Because the revision counters drive every sync operation, a torn sync RPC is safe to retry against a fresh session. `Workspace` performs one such reconnect retry automatically. Specifically:
 
 - **`pushOnce`.** `pushRev` is written only after
   `assertAppliedPushCursor` succeeds. A torn push leaves `pushRev` at
@@ -192,12 +192,11 @@ to retry against a fresh session. Specifically:
   past that point, including within the same rev. `applyChanges`'s
   `alreadyApplied` check drops any duplicates the resume happens to
   overlap with.
-- **`exec.events`.** Each event carries a monotonic `seq` per exec
-  id. The client reattaches via `getExec({ id, after: seq })`.
+- **`exec` dispatch.** A failed connection setup or a local disposed-stub error happens before dispatch and can be retried once. Other transport failures are ambiguous: computerd may have accepted the command before the response was lost. The backend invalidates the handle and reports the failure without replaying the command.
+- **`exec.events`.** Each event carries a monotonic `seq` per exec ID and callers can reattach with `getExec({ id, after: seq })`. The current automatic recovery boundary does not reattach a torn event stream; it reports the stream failure and leaves the next explicit operation to reconnect.
+- **`getExec`, `killExec`, and `disposeExec`.** These ID-addressed operations get one reconnect retry when the connection still points at the same container runtime UUID. A replacement process has an empty execution registry, so a runtime mismatch returns `EEXEC_LOST` without sending the old execution ID to the replacement.
 
-This is why the sync protocol survives transport failures: every
-operation has a persistent cursor, and every receiver is idempotent.
-capnweb itself is fragile, but the protocol layered on top isn't.
+This is why the sync protocol survives transport failures: every sync operation has a persistent cursor, and every receiver is idempotent. Shell commands require the separate no-replay boundary above because their side effects are not generally idempotent. capnweb itself is fragile, but the protocols layered on top define where recovery is safe.
 
 ### Stub disposal contract
 
@@ -394,9 +393,9 @@ items not yet shipped.
 | DO restart, container alive | New incarnation | Unchanged | Fresh session over fresh socket |
 | DO hibernate (future) | Isolate evicted, socket survives | Unchanged | *Fresh tables on wake; sync resumes from `_vfs_watermark`, exec resumes from `serializeAttachment` seqs* |
 | DO OOM | Killed, new incarnation on next event | Unchanged (until backend rebuilds) | Dies, fresh session on next call |
-| Container SIGTERM | Unchanged until next call | Restarted; in-memory VFS lost | Dies on container exit; *watermark reconcile on next connect repairs the mismatch* |
-| Container OOM/kill | Unchanged until next call | Killed; restarted on next call | Same as SIGTERM |
-| WebSocket idle disconnect | Unchanged | Unchanged | Dies on `close`; *reconnect wrapper rebuilds* |
+| Container SIGTERM | Invalidates the handle and reconnects on the active or next operation | Restarted; in-memory VFS lost | Dies on container exit; watermark reconciliation rebuilds from Durable Object storage |
+| Container OOM/kill | Invalidates the handle and reconnects on the active or next operation | Killed; restarted on reconnect | Same as SIGTERM; container-only unsynced data is lost |
+| WebSocket idle disconnect | Invalidates and closes the handle | Unchanged | Dies on `close`; replay-safe operations reconnect once |
 | Both die (host failure) | New incarnation on next event | New container | Rev-0 baseline from DO store |
 
 The recurring theme: **DO storage is the only durable thing in this

packages/computer/README.md

@@ -476,9 +476,10 @@ fail; the result exposes `sync: { status: "pending", ... }`. Configure a
 `SyncRetryScheduler` on `Workspace` to persist one coalesced retry per
 backend, then call `workspace.retryPendingSync(backend)` from your DO's
 alarm. Retries use bounded exponential backoff and return `"exhausted"`
-after the configured maximum. The library does not own your DO's alarm.
-See `SyncRetryScheduler`, `SyncRetryIntent`, and `SyncRetryOptions` in
-the package exports.
+after the configured maximum. A container replacement returns `"lost"`
+and clears the unrecoverable intent so new work is not blocked. The library
+does not own your DO's alarm. See `SyncRetryScheduler`, `SyncRetryIntent`,
+and `SyncRetryOptions` in the package exports.
 
 ### Observability
 

packages/computer/src/backend.ts

@@ -73,6 +73,11 @@ export interface BackendHandle {
   // The composite WorkspaceRPC stub pointing at the computerd
   // backend produced.
   rpc: WorkspaceRPC;
+  // Durable identity of the runtime process behind this connection.
+  // Reconnecting to the same process preserves it; a replacement
+  // process receives a new id. Backends without process-local state
+  // may omit it.
+  runtimeId?: string;
   // Declares whether this backend pairs with an independent
   // remote store that the Workspace must sync against.
   //