Repository navigation
VPC Router Corruption when working with large number of networks containing instances with public IP addresses #3025
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
VR
CLOUDSTACK VERSION
4.11.1
CONFIGURATION
N/A
OS / ENVIRONMENT
N/A
SUMMARY
VPC network / Virtual router is in an unstable/corrupted state and requires one or more restarts with "Clean" option
STEPS TO REPRODUCE
We are using CloudStack 4.11.1 running with KVM hosts. To simulate our usecase, we created a small program that calls CloudStack API to
- create VPC network with 20 guest networks, each containing one virtual machine with a public IP address allocated.
- delete the machines and networks one by one.
However, we frequently get a timeout error, sometimes during VM deletion, and sometimes during guest network deletion or even during static NAT disable step. Once the timeout occurs, it seems that the VPC network / Virtual router is in an unstable/corrupted state. We need to restart the Virtual Router with a clean option (sometimes have to try restart several times as it fails to deploy router VM as well). After that, we can continue delete the network remaining environment. Here is the high level steps that we did:
- Create VPC Network
- For each of the 20 "environments"
- Create Guest Network
- Add a VM to the network
- Acquire Public IP
- Associate the Public IP with VM
- For each of the 20 environment
- Disassociate the Public IP
- Delete VM
- Delete Guest network
- Delete VPC
I'm attaching the simple java program which performs all of the above described steps and which allowed us to consistently run into the bug.
To use the application:
java -jar testCloudStack.jar <CloudStack API url: e.g. http://foo:8080/client/api>
Note, that the test application works successfully with CloudStack server 4.9.2 but consistently reproduces the bug with CloudStack server 4.11.1
EXPECTED RESULTS
Network deletion is successful and VPC is in operational state
ACTUAL RESULTS
The hanging / timeout problems could be in any time during environment deletion. The first few deletions could go through successfully, and then fail at some point. The failure could be in any stage. i.e. Disassociate public IP, delete VM or delete guest network. We looked at cloud.log, agent log and management server log but couldn’t get any obvious errors. It seems that management server sends the request to do the deletion, but the VR does not respond and the system/network becomes stuck in an invalid state. Network often gets stuck in “Shutdown” state as a result.
Here are some errors in the management server log:
2018-11-01 01:15:29,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-119:ctx-c14b2ab4 job-29965) (logid:dbe80d4f) Complete async job-29965, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to delete network"}
2018-11-01 01:15:29,245 DEBUG [c.c.a.t.Request] (API-Job-Executor-119:ctx-c14b2ab4 job-29965 ctx-eb2dda94) (logid:dbe80d4f) Seq 4-667095694804259240: Received:
{ Ans: , MgmtId: [7474664765770|tel:7474664765770], via: 4(cehv02.core.jazz.net), Ver: v1, Flags: 110,
{ GroupAnswer }
}
2018-11-01 01:15:29,245 WARN [c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] (API-Job-Executor-119:ctx-c14b2ab4 job-29965 ctx-eb2dda94) (logid:dbe80d4f) Unable to destroy guest network on router VM[DomainRouter|r-3388-VM]
2018-11-01 01:15:29,247 WARN [c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] (API-Job-Executor-119:ctx-c14b2ab4 job-29965 ctx-eb2dda94) (logid:dbe80d4f) Failed to destroy guest network config Ntwk[1122|Guest|12] on router VM[DomainRouter|r-3388-VM]
2018-11-01 01:15:29,247 WARN [c.c.n.e.VpcVirtualRouterElement] (API-Job-Executor-119:ctx-c14b2ab4 job-29965 ctx-eb2dda94) (logid:dbe80d4f) Failed to unplug nic in network Ntwk[1122|Guest|12] for virtual router VM[DomainRouter|r-3388-VM]
2018-11-01 01:15:29,247 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-119:ctx-c14b2ab4 job-29965 ctx-eb2dda94) (logid:dbe80d4f) Unable to complete shutdown of the network elements due to element: VpcVirtualRouter
2018-11-01 01:15:29,255 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-119:ctx-c14b2ab4 job-29965 ctx-eb2dda94) (logid:dbe80d4f) Lock is released for network Ntwk[1122|Guest|12] as a part of network shutdown
2018-11-01 01:15:29,256 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-119:ctx-c14b2ab4 job-29965 ctx-eb2dda94) (logid:dbe80d4f) Network is not not in the correct state to be destroyed: Shutdown
Activity
dubauski commentedon Dec 19, 2018
Any updates from development team?
dubauski commentedon Jan 8, 2019
Any updates?
yadvr commentedon Jun 25, 2019
@dubauski sorry we couldn't get to this earlier, we'll try to reproduce this and get back to you. Could you also consider testing 4.11.2.0 which has several improvements and fixes over 4.11.1.0?
ustcweizhou commentedon Jun 25, 2019
@dubauski are the public ips in same subnet or different subnets ?
yadvr commentedon Jun 28, 2019
Related: #3103
Related: #3104
13 remaining items
weizhouapache commentedon Aug 19, 2021
@dubauski could you test with 4.15.1.0 ?
#4484 fixed some vpc issues.
yadvr commentedon Aug 20, 2021
@weizhouapache out of curiousity, did you try reproducing this?
weizhouapache commentedon Aug 20, 2021
@rhtyd
no.
I wrote 4 component tests in pr #4484 which cover many cases and verify their expected results. The cases are more complicated than what @dubauski described.
test/integration/component/test_multiple_subnets_in_isolated_network.py
test/integration/component/test_multiple_subnets_in_isolated_network_rvr.py
test/integration/component/test_multiple_subnets_in_vpc.py
test/integration/component/test_multiple_subnets_in_vpc_rvr.py
I can run the application @dubauski provided.
yadvr commentedon Aug 31, 2021
Copy @weizhouapache moving to next milestone - pl discuss with @nvazquez and close the issue if it's not reproducible now.
weizhouapache commentedon Sep 3, 2021
@nvazquez
could you please prioritize it ?
yadvr commentedon Sep 8, 2021
Closing this based on @weizhouapache 's comment. @dubauski pl reopen if you're able to reproduce this with latest 4.15.
weizhouapache commentedon Sep 10, 2021
tested several times, all succeeeeeed.
we can conclude this issue has been fixed in 4.16.0.0.
cc @nvazquez @rhtyd
to test it, I made following changes
(1) installed openjdk7 and tested using it (openjdk11 does not work).
(2) hardcoded service offering to use 1 * 1000 cpu, 512MB ram and shared storage (the jar uses 'ALM local system - 2 x 2 GHz CPU, 8 GB RAM')
(3) increased vlan range (the test requires 20 free vlans) in zone
(4) updated some global settings
router.aggregation.command.each.timeout = 120
vpc.max.networks = 50 (from 5)
an example of output