Management server msid depends on volatile network interface order, causing agent reconnection failures

[Pearl1594]

problem

The management server identity (msid) is computed at MS startup by iterating NetworkInterface.getNetworkInterfaces(), reversing the list, and selecting the first non-virtual, non-loopback physical interface. This ordering is controlled by the OS/kernel and is not stable across reboots, kernel updates, or configuration changes.

When the selected interface changes (e.g., from ens35 to ens34), the management server boots with a new msid while retaining the same service_ip. This causes:

1. A new row is inserted into cloud.mshost with the new msid and state Up.
2. The old row remains as Down with removed = NULL indefinitely - no automatic cleanup.
3. KVM agents remember the old msid and place the new one on the avoidMSHostList, refusing to reconnect.
4. Manual systemctl restart cloudstack-agent on each host is required.

Similar to : #10915

versions

Observed on 4.22.1.0 - but dates back to when the clustered management server feature was introduced.

The steps to reproduce the bug

...

What to do about it?

• Derive msid from a stable identifier - e.g., persist a UUID on first boot (file-based or DB-backed), not from a volatile MAC address.
• When registering, detect if another mshost row already has the same service_ip and mark it removed=1.
• Provide a config knob (management.server.id) for operators who need full control over the identity