KVM host declared down and then getting stuck in connecting state in ACS

[GerorgeEG]

problem

We experienced issue atleast 2-3 times where KVM host declared down by ACS and then it gets stuck in "Connecting" state in ACS. We have seen this issue in our 2 environments and it is production impacting issue.

KVM cloudstack agent logs

2026-05-18 22:10:49,716 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:) Trying to fetch storage pool 15cf3201-666a-3b51-8c73-233589579245 from libvirt 2026-05-18 22:10:50,133 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:) Trying to fetch storage pool 15cf3201-666a-3b51-8c73-233589579245 from libvirt 2026-05-18 22:10:51,699 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:) Asking libvirt to refresh storage pool 15cf3201-666a-3b51-8c73-233589579245 2026-05-18 22:12:58,808 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool 59139077-6f99-32a5-886b-852149d9bf36 from libvirt 2026-05-18 22:12:58,821 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Asking libvirt to refresh storage pool 59139077-6f99-32a5-886b-852149d9bf36 2026-05-18 22:13:06,637 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-1:[]) (logid:) Trying to fetch storage pool 15cf3201-666a-3b51-8c73-233589579245 from libvirt 2026-05-18 22:17:13,430 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:) Trying to fetch storage pool e0709dcf-1389-36b6-a5ed-4a1de4af2da7 from libvirt 2026-05-18 22:17:16,806 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:) Trying to fetch storage pool e0709dcf-1389-36b6-a5ed-4a1de4af2da7 from libvirt 2026-05-18 22:17:20,213 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:) Trying to fetch storage pool e0709dcf-1389-36b6-a5ed-4a1de4af2da7 from libvirt 2026-05-18 22:17:23,401 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:) Trying to fetch storage pool e0709dcf-1389-36b6-a5ed-4a1de4af2da7 from libvirt 2026-05-18 22:17:30,474 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool a9c72063-59b8-3381-b091-33efea922438 from libvirt 2026-05-18 22:17:30,513 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool a9c72063-59b8-3381-b091-33efea922438 from libvirt 2026-05-18 22:17:30,540 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool a9c72063-59b8-3381-b091-33efea922438 from libvirt 2026-05-18 22:17:30,566 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool a9c72063-59b8-3381-b091-33efea922438 from libvirt 2026-05-18 22:17:30,591 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool a9c72063-59b8-3381-b091-33efea922438 from libvirt 2026-05-18 22:17:30,617 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool a9c72063-59b8-3381-b091-33efea922438 from libvirt 2026-05-18 22:17:30,653 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool a9c72063-59b8-3381-b091-33efea922438 from libvirt 2026-05-18 22:17:30,688 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:) Trying to fetch storage pool a9c72063-59b8-3381-b091-33efea922438 from libvirt 2026-05-18 22:17:39,513 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:) Trying to fetch storage pool 15cf3201-666a-3b51-8c73-233589579245 from libvirt 2026-05-18 22:22:33,008 INFO  [cloud.agent.Agent] (Agent-Handler-4:[]) (logid:8dfd4ff2) Lost connection to host: x.x.x.x. Attempting reconnection while we still have 5 commands in progress. 2026-05-18 22:22:33,009 INFO  [utils.nio.NioClient] (Agent-Handler-4:[]) (logid:8dfd4ff2) NioClient connection closed 2026-05-18 22:22:38,009 INFO  [cloud.agent.Agent] (Agent-Handler-4:[]) (logid:8dfd4ff2) Reconnecting to host: x.x.x.x 2026-05-18 22:22:38,009 INFO  [utils.nio.NioClient] (Agent-Handler-4:[]) (logid:8dfd4ff2) Connecting to x.x.x.x:8250 2026-05-18 22:22:38,010 INFO  [utils.nio.NioClient] (Agent-Handler-4:[]) (logid:8dfd4ff2) Connected to x.x.x.x:8250 2026-05-18 22:22:38,010 INFO  [utils.nio.Link] (Agent-Handler-4:[]) (logid:8dfd4ff2) Conf file found: /etc/cloudstack/agent/agent.properties 2026-05-18 22:22:38,081 INFO  [utils.nio.NioClient] (Agent-Handler-4:[]) (logid:8dfd4ff2) SSL: Handshake done 2026-05-18 22:22:39,384 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-1:[]) (logid:) Asking libvirt to refresh storage pool 15cf3201-666a-3b51-8c73-233589579245 2026-05-18 22:22:39,384 WARN  [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:) Unable to send response: null 2026-05-18 22:22:39,395 WARN  [cloud.agent.Agent] (AgentOutRequest-Handler-5:[]) (logid:) Unable to send request to /x.x.x.x:8250 due to 'null', request: null 2026-05-18 22:22:39,395 WARN  [cloud.agent.Agent] (AgentOutRequest-Handler-1:[]) (logid:) Unable to send request to /x.x.x.x:8250 due to 'null', request: null 2026-05-18 22:22:39,405 WARN  [cloud.agent.Agent] (AgentOutRequest-Handler-2:[]) (logid:) Unable to send request to /x.x.x.x:8250 due to 'null', request: null 2026-05-18 22:22:39,417 WARN  [cloud.agent.Agent] (AgentOutRequest-Handler-3:[]) (logid:) Unable to send request to /x.x.x.x:8250 due to 'null', request: null 2026-05-18 22:22:39,444 WARN  [cloud.agent.Agent] (AgentOutRequest-Handler-4:[]) (logid:) Unable to send request to /x.x.x.x:8250 due to 'null', request: null

versions

ACS version post-upgrade : 4.20.2.0 (upgraded from 4.19.1.2 few weeks before)
Management and DB OS : RHEL 8
KVM : RHEL 8

The steps to reproduce the bug

1. It is generated by ACS on its own not able to regenerate

...

What to do about it?

KVM host gets reconnected on its own after 1 hour.

[DaanHoogland]

I see problems mounting the storage pools in these logs @GerorgeEG . I think the connection to the MS is consequential. you can check if you can reach port 8250 on the host masked as x.x.x.x, but I think the actual connection is not your problem.