fix(ssh): add EMFILE backoff and exit notification to SSH accept loop - #2705
Merged
sjenning merged 1 commit intoAug 25, 2026
Merged
Conversation
politerealism
requested review from
a team,
derekwaynecarr,
maxamillion and
mrunalp
as code owners
August 11, 2026 17:19
|
All contributors have signed the DCO ✍️ ✅ |
Contributor
Author
|
Once #2370 merges into main, this branch will need to be rebased onto the updated main and force-pushed. The rebase should be clean — the proxy changes and SSH changes don't overlap. |
politerealism
force-pushed
the
fix/ssh-accept-backoff-and-exit-notification
branch
2 times, most recently
from
August 17, 2026 15:28
3e0c305 to
16ecda1
Compare
politerealism
force-pushed
the
fix/ssh-accept-backoff-and-exit-notification
branch
from
August 25, 2026 19:23
16ecda1 to
2079b07
Compare
Apply the same two-layer defense from the proxy accept loop (NVIDIA#2369/NVIDIA#2370) to the SSH accept loop: classify transient vs terminal accept errors with exponential backoff on EMFILE/resource-exhaustion, and notify the sandbox when the accept loop exits so the container terminates instead of running without SSH access. - Add SshAcceptAction enum and classify_ssh_accept_error in ssh.rs, mirroring the proxy pattern (EMFILE/ENFILE/ENOBUFS → Retry with backoff, unknown errors → Terminal after 10 consecutive failures) - Replace the bare accept().await in run_ssh_server with a classify-and- retry loop; resets consecutive-error counter on each successful accept - Thread ssh_exit_tx: Option<oneshot::Sender<()>> through run_process; hold it as a drop-guard inside the SSH spawn so the receiver fires when the task ends for any reason - Wire ssh_exited future in lib.rs (created only when ssh_socket_path is Some) and select! on it in both process_enabled paths, returning an error so the sandbox container restarts Closes NVIDIA#2372 Signed-off-by: politerealism <burdcat17@gmail.com>
politerealism
force-pushed
the
fix/ssh-accept-backoff-and-exit-notification
branch
from
August 25, 2026 19:54
2079b07 to
5ed5f7a
Compare
Collaborator
|
/ok to test 5ed5f7a |
|
Label |
sjenning
enabled auto-merge
August 25, 2026 21:06
sjenning
approved these changes
Aug 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Applies the same two-layer defense from the proxy accept loop (#2369 / #2370) to the SSH accept loop:
Retry(EMFILE/ENFILE/ENOBUFS — exponential backoff) vsTerminal(≥10 consecutive unknown errors → break). Mirrorsclassify_accept_errorinopenshell-supervisor-network.oneshot::Sender<()>drop-guard throughrun_processso the sandbox is notified when the SSH task ends for any reason. The sandbox selects onssh_exitedin bothprocess_enabledpaths and returns an error, causing Kubernetes to restart the container rather than leaving it running without SSH access.Related Issue
Closes #2372
Changes
crates/openshell-supervisor-process/src/ssh.rsaccept().await?with classify-and-retry loop; addSshAcceptActionenum andclassify_ssh_accept_errorcrates/openshell-supervisor-process/src/run.rsssh_exit_tx: Option<oneshot::Sender<()>>parameter; hold as drop-guard in SSH spawn blockcrates/openshell-sandbox/src/lib.rs(ssh_exit_tx, ssh_exit_rx)channel whenssh_socket_pathisSome, pinssh_exitedfuture, select on it in bothprocess_enabledpathsTesting
cargo clippy -p openshell-sandbox -p openshell-supervisor-process— cleancargo fmt --check— cleanChecklist
Signed-off-bypresent