bug: gateway does not reload TLS certificate after cert-manager renewal #1836

Agent Diagnostic

• Skills loaded: debug-openshell-cluster — followed the Kubernetes Helm gateway path (Step 5) to investigate TLS certificate loading in the gateway server crate.
• What was found:
  • TlsAcceptor::from_files() in crates/openshell-server/src/tls.rs:39 reads the cert/key PEM files once and builds an Arc<rustls::ServerConfig>, which is immutable after construction. The acceptor is created in crates/openshell-server/src/lib.rs:417 before any listeners start and is never replaced.
  • A search across the entire crates/openshell-server/src/ tree for watch, notify, inotify, reload, hot_reload, or file_watch returned zero results related to TLS certificate reloading.
  • The TlsConfig struct in crates/openshell-core/src/config.rs:398 only stores file paths (cert_path, key_path, client_ca_path) — no reload interval, watch flag, or any mechanism to trigger re-reading.
  • The Kubernetes Secret mount (/etc/openshell-tls/server/) and kubelet symlink update work correctly — the renewed cert is on disk inside the pod after cert-manager rotates it. The gateway simply never re-reads the file.
  • The cert-manager integration in deploy/helm/openshell/templates/cert-manager-pki.yaml and the certgen hook JWT-only fallback in deploy/helm/openshell/templates/certgen.yaml:98 are correctly wired on the Kubernetes side.
  • Root cause: The certificate is loaded exactly once at startup and held in an immutable Arc<ServerConfig> for the lifetime of the process. There is no file-watch, no periodic poll, and no atomic acceptor swap mechanism.
• What was tried:
  • Searched the full openshell-server source for any file-watch or reload mechanism — confirmed none exists for TLS.
  • Verified rustls::ServerConfig immutability and confirmed tokio-rustls provides no built-in cert reload facility.
  • Traced the full Kubernetes-side chain (cert-manager Certificate → Secret → kubelet symlink sync → pod filesystem) — confirmed it works correctly up to the point where the gateway must re-read the files.

Description

Actual behavior: When the OpenShell gateway is deployed with certManager.enabled=true, cert-manager automatically renews TLS certificates before expiry and updates the Kubernetes Secret. The kubelet syncs the new cert/key files into the pod's mounted volume. However, the gateway never picks up the renewed certificate, it continues serving the old certificate until the pod is manually restarted. Once the old certificate expires, all TLS connections from CLI clients, sandbox supervisors, and mTLS peers break. No error is logged, and there is no automatic recovery.

Expected behavior: The gateway should detect when its TLS certificate files change on disk (via the mounted Secret volume) and reload them without requiring a pod restart. After cert-manager renews the certificate and kubelet syncs the updated Secret, the gateway should begin serving the new certificate within a reasonable interval.

Reproduction Steps

1. Create a local cluster:

   k3d cluster create openshell-repro \
     --servers 1 --agents 0 \
     --k3s-arg '--disable=traefik@server:0' \
     --wait

2. Install cert-manager:

   helm repo add jetstack https://charts.jetstack.io
   helm repo update
   helm upgrade --install cert-manager jetstack/cert-manager \
     --namespace cert-manager --create-namespace \
     --set crds.enabled=true --wait --timeout 5m

3. Deploy OpenShell with cert-manager enabled and short-lived certificates (so renewal can be tested without waiting days):

   cat > /tmp/openshell-values.yaml << 'EOF'
   server:
     disableTls: false
   certManager:
     enabled: true
     certificateDuration: "2h"
     certificateRenewBefore: "1h"
   postgres:
     enabled: false
   EOF
   
   helm upgrade --install openshell \
     oci://ghcr.io/nvidia/openshell/helm-chart \
     --namespace openshell --create-namespace \
     --values /tmp/openshell-values.yaml --wait --timeout 5m

4. Record the certificate fingerprint the gateway is currently serving (call this OLD_FP):

   kubectl -n openshell port-forward pod/openshell-0 18443:8080 &
   sleep 3
   
   echo "" | openssl s_client -connect 127.0.0.1:18443 \
     -servername openshell 2>/dev/null \
     | openssl x509 -fingerprint -sha256 -noout
   # Example: sha256 Fingerprint=1C:3D:8E:E0:F9:33:92:E5:...
   
   echo "" | openssl s_client -connect 127.0.0.1:18443 \
     -servername openshell 2>/dev/null \
     | openssl x509 -text -noout | grep "Not After"
   # Example: Not After : Jun  9 13:28:23 2026 GMT
   
   kill %1 2>/dev/null

5. Trigger cert-manager renewal and confirm the Secret is updated:

   cmctl renew -n openshell openshell-server
   sleep 5
   kubectl -n openshell get certificate openshell-server
   # Should show READY=True
   
   kubectl -n openshell get secret openshell-server-tls \
     -o jsonpath='{.data.tls\.crt}' | base64 -d \
     | openssl x509 -fingerprint -sha256 -noout
   # This fingerprint (NEW_FP) differs from OLD_FP — cert-manager worked correctly

6. Wait for kubelet to sync the updated Secret into the pod's filesystem (60s), then connect again:

   sleep 60
   
   kubectl -n openshell port-forward pod/openshell-0 28443:8080 &
   sleep 3
   
   echo "" | openssl s_client -connect 127.0.0.1:28443 \
     -servername openshell 2>/dev/null \
     | openssl x509 -fingerprint -sha256 -noout
   
   kill %1 2>/dev/null

7. Observe the bug: The gateway still serves OLD_FP, the original certificate that cert-manager has already replaced in the Secret. The renewed certificate is on disk inside the pod but the gateway never picked it up.

8. Restart the gateway pod — the new certificate is now served (proving restart is the only workaround):

   kubectl -n openshell rollout restart statefulset/openshell
   kubectl -n openshell rollout status statefulset/openshell --timeout=120s
   
   kubectl -n openshell port-forward pod/openshell-0 38443:8080 &
   sleep 3
   
   echo "" | openssl s_client -connect 127.0.0.1:38443 \
     -servername openshell 2>/dev/null \
     | openssl x509 -fingerprint -sha256 -noout
   # Now serves NEW_FP — the renewed certificate
   
   kill %1 2>/dev/null

Environment

• OS: macOS (Darwin)
• OpenShell Helm chart: 0.0.58
• cert-manager: v1.20.2
• Kubernetes (k3d): v1.31.5-k3s1
• cmctl: v2.5.0

Logs

No error or warning logs are emitted by the gateway when the certificate files change on disk. The failure is entirely silent, the gateway continues serving the old certificate with no indication that a newer certificate is available. This lack of observability is part of the issue: operators have no signal that the gateway's served certificate has diverged from what is in the Secret.

Agent-First Checklist

• I pointed my agent at the repo and had it investigate this issue
• I loaded relevant skills (e.g., debug-openshell-cluster, debug-inference, openshell-cli)
• My agent could not resolve this — the diagnostic above explains why