Repository navigation
RvR Services should be provided on VIP instead of VR IP #3409
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
Redundant VRs
CLOUDSTACK VERSION
4.11.2
CONFIGURATION
Advanced Networking, possibly also VPCs
OS / ENVIRONMENT
N/A
SUMMARY
Redundant VRs provide their services on their own internal IP rather than the Virtual IP. This leads to a number of problems, mostly triggered around VRs failing over after a VM has booted:
- DHCP server is run on VR IP, which means VM cannot renew its IP. Once the renew + expire times run out, the VM attempts to discover a new DHCP server (this causes additional problems with secondary IPs being removed).
- Userdata services only listen on VR IP, so accessing this data requires checking the DHCP lease info to find the VR's IP. If the VR has failed over, userdata cannot be accessed.
- Hairpin static NAT connections appear to come from the VR's IP, and since that can change it may confuse end clients
STEPS TO REPRODUCE
Build a redundant advanced network with guest range 10.1.1.0/24, deploy a Linux VM.
- Observe DHCP server
- Check userdata (curl http://VRIP/latest/instance-id)
- Add a static NAT rule
EXPECTED RESULTS
- DHCP server should be 10.1.1.1
- Userdata should be accessible on http://10.1.1.1
- POSTROUTING rules on nat table should use 10.1.1.1 as the source
ACTUAL RESULTS
- DHCP server is the unique VR IP from the internal range
- Userdata is only accessible at VR's IP, not 10.1.1.1
- POSTROUTING rules on nat table (hairpin nat) use the VR IP
Activity
richardlawley commentedon Jun 19, 2019
To elaborate on the DHCP problem, this is related to #3351. The VMs we've found most affected by this are Linux machines with a secondary internal IP. Their primary IP comes from DHCP, but the secondaries are all statically configured.
If the DHCP lease is renewed, no problem occurs. However, if a renew fails and a re-discover takes place, the statically configured secondary IPs are all removed along with the DHCP primary, and not restored when the new lease is acquired.
There are two causes of this:
Problem 2 was addressed in #3351, but without running the DHCP server on the VIP, problem 1 still exists.
We are working around this until this issue is resolved by going back to infinite leases in our custom ISO.
yadvr commentedon Jun 20, 2019
@ustcweizhou can you share the diff/patch or send a PR in case you've solve this issue in your tree?
ustcweizhou commentedon Jun 20, 2019
@rhtyd
I have checked all my code. it seems very difficult to port our changes to 4.11 or 4.13 as there are big difference between our branch and 4.11/4.13.
Anyone who is interested in the fixes please feel free to pick it up.
yadvr commentedon Jun 20, 2019
Okay, is your tree public/accessible somewhere @ustcweizhou ?
ustcweizhou commentedon Jun 22, 2019
@rhtyd
I spent spare time on these issues, they seem to be fixed now.
I will test it and create PR next week.
yadvr commentedon Jun 23, 2019
Thanks @ustcweizhou looking forward to your PR
yadvr commentedon Jul 9, 2019
Ping @ustcweizhou any update? Thanks.
Related: #3179
yadvr commentedon Jul 9, 2019
Thanks @ustcweizhou I'll help review and test
yadvr commentedon Jul 24, 2019
This is a known issue and it's likely that we are unable to fix issue and the PR seems to be adding regressions. I've moved it to 4.13.1.0 milestone for now, if the regression/issues can be fixed we can consider including the PR towards 4.13.0.0.
router: Set up metadata/password/dhcp server on gateway IP instead of…
router: Set up metadata/password/dhcp server on gateway IP instead of…