[ConsoleProxy] Performance fixes on concurrency and closing console sessions sockets and introduce reconnection window for console sessions - #13683
Conversation
There was a problem hiding this comment.
Pull request overview
This PR targets ConsoleProxy reliability and resource usage by (1) making the “allowed sessions” tracking safe under concurrent access, and (2) ensuring VNC-related connections are actively closed when a noVNC browser session ends, preventing lingering sockets on KVM hosts.
Changes:
- Introduces explicit close paths for NIO socket resources and noVNC client connections to avoid leaking VNC sockets.
- Makes
ConsoleProxy.allowedSessionsconcurrency-safe and guards one-time session consumption. - Ensances WebSocket close handling to trigger client teardown and make remote-address logging more robust.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| services/console-proxy/server/src/main/java/com/cloud/consoleproxy/vnc/NoVncClient.java | Adds a close() method to shut down NIO, websocket, and tunnel socket resources. |
| services/console-proxy/server/src/main/java/com/cloud/consoleproxy/vnc/network/NioSocketHandlerImpl.java | Stores the underlying NioSocket and exposes close() to release it. |
| services/console-proxy/server/src/main/java/com/cloud/consoleproxy/vnc/network/NioSocketHandler.java | Adds a close() contract to the handler interface. |
| services/console-proxy/server/src/main/java/com/cloud/consoleproxy/vnc/network/NioSocket.java | Adds a close() method to release the socket channel and selectors. |
| services/console-proxy/server/src/main/java/com/cloud/consoleproxy/ConsoleProxyNoVNCHandler.java | Ensures viewer teardown is triggered on WebSocket close and hardens remote-address logging. |
| services/console-proxy/server/src/main/java/com/cloud/consoleproxy/ConsoleProxyNoVncClient.java | Ensures closeClient() closes the underlying NoVncClient before removing the viewer. |
| services/console-proxy/server/src/main/java/com/cloud/consoleproxy/ConsoleProxy.java | Switches allowedSessions to a concurrent set and synchronizes one-time session consumption; uses closeClient() when replacing viewers. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 4.22 #13683 +/- ##
============================================
- Coverage 17.69% 17.69% -0.01%
Complexity 15833 15833
============================================
Files 5925 5925
Lines 533534 533627 +93
Branches 65273 65290 +17
============================================
- Hits 94421 94419 -2
- Misses 428434 428531 +97
+ Partials 10679 10677 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@blueorangutan package |
|
@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18654 |
|
Marked on draft as still working on fixing the issues reported on: #13422 |
…after acquiring a session
|
@blueorangutan package |
|
@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
nvazquez
changed the title
| if (grant.clientIp != null && !grant.clientIp.equals(clientIp)) { | ||
| LOGGER.warn("Rejecting reconnection for session {} as it was requested from IP {} " + | ||
| "but the session was granted to IP {}", sessionUuid, clientIp, grant.clientIp); | ||
| return false; |
There was a problem hiding this comment.
should "return true" only if client IPs match, and "return false" by default ?
There was a problem hiding this comment.
Hi @weizhouapache not only client IPs match but also the reconnection attempt is within the reconnection window/timeframe
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18698 |
|
@blueorangutan test |
|
@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-16643)
|
|
@blueorangutan package |
|
@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19156 |
|
@blueorangutan test |
|
@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-16934)
|
kiranchavala
left a comment
kiranchavala
left a comment
There was a problem hiding this comment.
LGTM tested manually
cmk update configuration name=consoleproxy.session.reconnection.window value=5000 zoneid=
cmk destroy systemvm id=
root@v-3-VM:~# cat /var/cache/cloud/cmdline | tr ' ' '\n' | grep session_reconnection_window
session_reconnection_window=5000
root@v-3-VM:/var/log/cloud# grep -i "sessionReconnectionWindowMs" cloud.out
2026-09-10T05:15:48,486 INFO [cloud.consoleproxy.ConsoleProxy] (Console-Proxy-Main:[]) Setting sessionReconnectionWindowMs=5000
Test1 : lingering VNC sockets
- Open the VM console in the UI, confirm exactly one ESTABLISHED socket from the CPVM IP.
- Close the browser tab → the socket must disappear within a second or two.
- Repeat open/close 10 times → count must stay at 0/1, never accumulate. (Pre-fix you'd see one leftover socket per session.)
- Also test the abrupt paths: kill the browser, and reload the console tab repeatedly.
[root@ref-trl-12349-k-Mol8-kiran-chavala-kvm1 ~]# virsh domdisplay i-2-4-VM
vnc://10.0.35.135:3
[root@ref-trl-12349-k-Mol8-kiran-chavala-kvm1 ~]# watch -n1 "ss -tn state established '( sport >= :5900 and sport <= :5999 )'"
Every 1.0s: ss -tn state established '( sport >= :5900 and sport <= :5999 )' ref-trl-12349-k-Mol8-kiran-chavala-kvm1: Thu Sep 10 06:06:25 2026
Recv-Q Send-Q Local Address:Port Peer Address:Port
0 0 10.0.35.135:5903 10.0.47.40:52320
Test 2 a — reconnection window
With consoleproxy.session.reconnection.window = 5000:
-
Open a console, then break the connection so the proxy declares it dead —
-
Reload the same console URL within 5s of the "Connection with client ... is dead." log line → should reconnect.
logs
/var/log/cloud/cloud.out
2026-09-10T06:07:28,964 INFO [cloud.consoleproxy.ConsoleProxyNoVncClient] (Thread-105:[]) Connection with client [3] [IP: 10.0.3.251] is dead.
2026-09-10T06:13:40,188 INFO [cloud.consoleproxy.ConsoleProxyResourceHandler] (Thread-106:[]) Get resource request for /resource/noVNC/vnc.html
2026-09-10T06:13:40,215 INFO [cloud.consoleproxy.ConsoleProxyResourceHandler] (Thread-106:[]) Sent file /resource/noVNC/vnc.html with content type text/html
2026-09-10T06:13:40,898 INFO [cloud.consoleproxy.ConsoleProxyNoVNCHandler] (qtp100802290-42:[]) Verifying session source IP 10.0.3.251 from WebSocket connection request.
2026-09-10T06:13:40,965 INFO [cloud.consoleproxy.ConsoleProxyNoVNCHandler] (qtp100802290-42:[]) Viewer has been created successfully [session UUID: bc997b0d-5a8b-4f73-a06e-55eb0c3ee010, client IP: 10.0.3.251].
2026-09-10T06:13:40,971 INFO [cloud.consoleproxy.ConsoleProxyNoVncClient] (Thread-107:[]) Connect to VNC server directly. host: 10.0.35.135, port: 5903
2026-09-10T06:13:40,976 INFO [consoleproxy.vnc.NoVncClient] (Thread-107:[]) Connecting to VNC server 10.0.35.135:5903 ...
2026-09-10T06:13:40,982 INFO [cloud.consoleproxy.ConsoleProxyNoVncClient] (Thread-107:[]) Connection to VNC server has been established successfully.
2026-09-10T06:13:41,370 INFO [vnc.security.VncTLSSecurity] (Thread-107:[]) Processing VNC TLS security
2026-09-10T06:13:41,375 INFO [utils.nio.Link] (Thread-107:[]) Conf file found: /usr/local/cloud/systemvm/conf/agent.properties
2026-09-10T06:13:41,622 INFO [vnc.security.VncAuthSecurity] (Thread-107:[]) VNC server requires password authentication
2026-09-10T06:13:41,658 INFO [vnc.security.VncAuthSecurity] (Thread-107:[]) Finished VNCAuth security
Test 2 b — reconnection window
- consoleproxy.session.reconnection.window = 60000, CPVM recreated.
- Copy the console URL from the UI (or cmk createConsoleEndpoint virtualmachineid=) so you can replay the same session UUID.
- Paste the same URL into a new tab within 60s → expect Reconnecting the session after a dropped connection.
CPVM log: Reconnecting the session after a dropped connection.
2026-09-10T06:29:04,211 INFO [cloud.consoleproxy.ConsoleProxy] (qtp2118053489-41:[]) Reconnecting the session 1953d0da-bc1f-4214-b9f9-861dd80b0e2f after a dropped connection
2026-09-10T06:29:04,217 INFO [cloud.consoleproxy.ConsoleProxyNoVNCHandler] (qtp2118053489-41:[]) Viewer has been created successfully [session UUID: 1953d0da-bc1f-4214-b9f9-861dd80b0e2f, client IP: 10.0.3.251].
2026-09-10T06:29:04,225 INFO [cloud.consoleproxy.ConsoleProxyNoVncClient] (Thread-13:[]) Connect to VNC server directly. host: 10.0.35.135, port: 5903
2026-09-10T06:29:04,229 INFO [consoleproxy.vnc.NoVncClient] (Thread-13:[]) Connecting to VNC server 10.0.35.135:5903 ...
2026-09-10T06:29:04,241 INFO [cloud.consoleproxy.ConsoleProxyNoVncClient] (Thread-13:[]) Connection to VNC server has been established successfully.
2026-09-10T06:29:04,590 INFO [vnc.security.VncTLSSecurity] (Thread-13:[]) Processing VNC TLS security
2026-09-10T06:29:04,592 INFO [utils.nio.Link] (Thread-13:[]) Conf file found: /usr/local/cloud/systemvm/conf/agent.properties
2026-09-10T06:29:04,817 INFO [vnc.security.VncAuthSecurity] (Thread-13:[]) VNC server requires password authentication
2026-09-10T06:29:04,839 INFO [vnc.security.VncAuthSecurity] (Thread-13:[]) Finished VNCAuth security
Test 3 — concurrency
Generate many console endpoints in parallel and connect at once:
for i in $(seq 1 4); do
cmk createConsoleEndpoint virtualmachineid=<vm-id> | jq -r '.consoleendpoint.url' &
done; wait
Then hit them concurrently (e.g. xargs -P 30 -n1 curl -sk -o /dev/null -w '%{http_code}\n'). Expect no ConcurrentModificationException in the CPVM LOG, and each URL usable exactly once. Doing this from several browsers/VMs simultaneously is the realistic version of the original bug.
Logs
2026-09-10T06:50:35,262 INFO [cloud.consoleproxy.ConsoleProxyNoVNCHandler] (qtp2118053489-41:[]) Verifying session source IP 10.0.3.251 from WebSocket connection request.
2026-09-10T06:50:35,281 INFO [cloud.consoleproxy.ConsoleProxy] (qtp2118053489-41:[]) Initializing new novnc client and disconnecting existing session
2026-09-10T06:50:35,286 INFO [cloud.consoleproxy.ConsoleProxyNoVncClient] (Thread-110:[]) Connection with client [6] [IP: 10.0.3.251] is dead.
2026-09-10T06:50:35,295 INFO [cloud.consoleproxy.ConsoleProxyNoVNCHandler] (qtp2118053489-41:[]) Viewer has been created successfully [session UUID: f454e5e9-e545-449d-9fb6-63780826bb4d, client IP: 10.0.3.251].
2026-09-10T06:50:35,297 INFO [cloud.consoleproxy.ConsoleProxyNoVncClient] (Thread-112:[]) Connect to VNC server directly. host: 10.0.35.135, port: 5903
2026-09-10T06:50:35,298 INFO [consoleproxy.vnc.NoVncClient] (Thread-112:[]) Connecting to VNC server 10.0.35.135:5903 ...
2026-09-10T06:50:35,301 INFO [cloud.consoleproxy.ConsoleProxyNoVncClient] (Thread-112:[]) Connection to VNC server has been established successfully.
2026-09-10T06:50:35,634 INFO [vnc.security.VncTLSSecurity] (Thread-112:[]) Processing VNC TLS security
2026-09-10T06:50:35,637 INFO [utils.nio.Link] (Thread-112:[]) Conf file found: /usr/local/cloud/systemvm/conf/agent.properties
2026-09-10T06:50:35,771 INFO [vnc.security.VncAuthSecurity] (Thread-112:[]) VNC server requires password authentication
2026-09-10T06:50:35,779 INFO [vnc.security.VncAuthSecurity] (Thread-112:[]) Finished VNCAuth security
Description
This PR addresses 3 issues:
In case of concurrent console proxy requests, 2 different threads could modify the allowedSessions set and cause problems accessing the console sessions\
Partial fix for the issue Intermittent Console Proxy Failure in 4.22.1.0 #13422: in cases in which a session has been properly acquired but after a few seconds declared as dead and then tried to reconnect, the CPVM prevented it as the session has already been acquired.
consoleproxy.session.reconnection.windowwith default value = 0, which sets a reconnection window for the same client IP once a connection is declared as dead.After closing the browser tab for a VM console session, it was observed that the connection remained open on the KVM host's VNC port.
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
agent.zipfile on/usr/share/cloudstack-common/vms/for each KVM host and recreated the CPVMss -tnpcommand for VM's VNC portsconsoleproxy.session.reconnection.window= 0, observe no changes on console sessions reconnectionconsoleproxy.session.reconnection.window= 5000, recreate CPVM. Observe refreshing the browser tab allows the same client to reclaim back/reconnect to the same session. Also, if another session acquires the console session, the previous session could be reclaimed back within the reconnection window timeframeHow did you try to break this feature and the system with this change?