Skip to content

fix(ssh): add EMFILE backoff and exit notification to SSH accept loop - #2705

Merged
sjenning merged 1 commit into
NVIDIA:mainfrom
politerealism:fix/ssh-accept-backoff-and-exit-notification
Aug 25, 2026
Merged

fix(ssh): add EMFILE backoff and exit notification to SSH accept loop#2705
sjenning merged 1 commit into
NVIDIA:mainfrom
politerealism:fix/ssh-accept-backoff-and-exit-notification

Conversation

@politerealism

Copy link
Copy Markdown
Contributor

Summary

Depends on #2370 — this branch is stacked on fix/proxy-exit-detection. Please review #2370 first; once it merges the diff here will collapse to only the SSH-specific changes below.

Applies the same two-layer defense from the proxy accept loop (#2369 / #2370) to the SSH accept loop:

  • EMFILE backoff: classify accept errors as Retry (EMFILE/ENFILE/ENOBUFS — exponential backoff) vs Terminal (≥10 consecutive unknown errors → break). Mirrors classify_accept_error in openshell-supervisor-network.
  • Exit notification: thread a oneshot::Sender<()> drop-guard through run_process so the sandbox is notified when the SSH task ends for any reason. The sandbox selects on ssh_exited in both process_enabled paths and returns an error, causing Kubernetes to restart the container rather than leaving it running without SSH access.

Related Issue

Closes #2372

Changes

File Change
crates/openshell-supervisor-process/src/ssh.rs Replace bare accept().await? with classify-and-retry loop; add SshAcceptAction enum and classify_ssh_accept_error
crates/openshell-supervisor-process/src/run.rs Add ssh_exit_tx: Option<oneshot::Sender<()>> parameter; hold as drop-guard in SSH spawn block
crates/openshell-sandbox/src/lib.rs Create (ssh_exit_tx, ssh_exit_rx) channel when ssh_socket_path is Some, pin ssh_exited future, select on it in both process_enabled paths

Testing

  • cargo clippy -p openshell-sandbox -p openshell-supervisor-process — clean
  • cargo fmt --check — clean
  • Sandbox e2e: verify container exits when SSH accept loop terminates unexpectedly

Checklist

@copy-pr-bot

copy-pr-bot Bot commented Aug 11, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown

All contributors have signed the DCO ✍️ ✅
Posted by the DCO Assistant Lite bot.

@politerealism

Copy link
Copy Markdown
Contributor Author

Once #2370 merges into main, this branch will need to be rebased onto the updated main and force-pushed. The rebase should be clean — the proxy changes and SSH changes don't overlap.

@politerealism
politerealism force-pushed the fix/ssh-accept-backoff-and-exit-notification branch 2 times, most recently from 3e0c305 to 16ecda1 Compare August 17, 2026 15:28
@politerealism
politerealism force-pushed the fix/ssh-accept-backoff-and-exit-notification branch from 16ecda1 to 2079b07 Compare August 25, 2026 19:23
Apply the same two-layer defense from the proxy accept loop (NVIDIA#2369/NVIDIA#2370)
to the SSH accept loop: classify transient vs terminal accept errors with
exponential backoff on EMFILE/resource-exhaustion, and notify the sandbox
when the accept loop exits so the container terminates instead of running
without SSH access.

- Add SshAcceptAction enum and classify_ssh_accept_error in ssh.rs,
  mirroring the proxy pattern (EMFILE/ENFILE/ENOBUFS → Retry with
  backoff, unknown errors → Terminal after 10 consecutive failures)
- Replace the bare accept().await in run_ssh_server with a classify-and-
  retry loop; resets consecutive-error counter on each successful accept
- Thread ssh_exit_tx: Option<oneshot::Sender<()>> through run_process;
  hold it as a drop-guard inside the SSH spawn so the receiver fires when
  the task ends for any reason
- Wire ssh_exited future in lib.rs (created only when ssh_socket_path is
  Some) and select! on it in both process_enabled paths, returning an
  error so the sandbox container restarts

Closes NVIDIA#2372

Signed-off-by: politerealism <burdcat17@gmail.com>
@politerealism
politerealism force-pushed the fix/ssh-accept-backoff-and-exit-notification branch from 2079b07 to 5ed5f7a Compare August 25, 2026 19:54
@sjenning

Copy link
Copy Markdown
Collaborator

/ok to test 5ed5f7a

@sjenning sjenning added the test:e2e Requires end-to-end coverage label Aug 25, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 5ed5f7a. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@sjenning
sjenning enabled auto-merge August 25, 2026 21:06
@sjenning
sjenning added this pull request to the merge queue Aug 25, 2026
Merged via the queue into NVIDIA:main with commit 60a9b4d Aug 25, 2026
84 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: SSH server accept loop exits permanently on transient errors (same class as #2337)

2 participants