Not able to deploy VM: Insufficient un-allocated capacity on StoragePool #14045
Replies: 9 comments 5 replies
increase this and retry |
|
Although changing this parameter may cause the VM to deploy, it is not the expected result, and besides that, I want the calculated space not to exceed the size of the primary storage disk (x1.0). In think the problem is not the overprovisioning factor, it is how the available space is calculated, when allocating 1.9T root disk from template (details above) on an empty 4 TB primary storage (reporting: Disk Size (in GB): 4095.5 GB (x1.0), 0.05% Used, Why: totalAllocatedSize : (1.8672 TB) 2052994367488 ? is it using "template size" for this calculation instead of the physical size of the qcow2 file which is "Template physical size: 20.81 GB"?
|
|
As a workaround, After setting overprovisioning factor to 2 I was able to deploy the VM. 
|
@mbertolina |
|
I tested with a thin disk service offering and Overprovisioning factor=1 on primary storage and got the same error when allocating the space for the VM |
@mbertolina
In this case, the total allocated size is 200 GB, but only 5 GB is actually used on the physical storage pool. If the storage pool size is 200 GB:
This is the trade-off with overprovisioning: a lower factor provides more protection against the storage pool becoming full, but may leave physical capacity underutilized. A higher factor allows you to make better use of the physical storage, but increases the risk of running out of physical space if the volumes actually consume their allocated capacity. |
@mbertolina this sounds like a way to hit the limit sooner, rather than later. If thin provisioning, it will not occupy the space completely from the start, but it will mark it as allocated. thin provisioning and overprovisioning work together to help you make good use of your disk space. If say half your disks are thin provisioned and they take about 10% of the space you need and the other half are thick provisioned, you would want 0.5*0.1 + 0.5 times the physical space to be able to allocate all. You would then use 5+0.5 times the space you actually have to be allocatable. Rounding down for safety of course. Also keep in mind life-spans of VMs, are they expected to grow to their full size during their life-time or will they be recycled when half way through their space. I wrote this while @weizhouapache replied. We are saying basically the same thing but in slightly different ways. Please ask more questions or if you understand help us improve the documentation at https://github.com/apache/cloudstack-documentation |
|
Hi @DaanHoogland and @weizhouapache , thanks for your quick response. I believe the previous points do not apply to this case. I am deploying a VM with only one ROOT disk, using the following configuration: Hypervisor: KVM On: During the VM deployment, StorageManagerImpl reports the following: maxSize : (3.9995 TB) 4397514358784
What I don't understand is why StorageManagerImpl is calculating askingSize as 1.8672 TB (2052994367488 bytes) and, at the same time, reporting the same value for totalAllocatedSize. This seems inconsistent with the actual physical usage of the storage pool. Just a few milliseconds earlier, the pool was reporting only ~0.05% physical usage:
So the main question is: Why does StorageManagerImpl use 1.8672 TB as both askingSize and totalAllocatedSize when the storage pool has only ~2.17 GB physically used (~0.05%) and the VM being deployed has a single ROOT disk? (considering overprovisioning factor = 1) I am wondering if this value comes from the template's virtual size (1,912 GB) rather than its physical size (20.81 GB)? |
|
Hello @mbertolina, Thank you for the excellent and detailed logs — this really helps pinpoint exactly what's happening. Let me explain why CloudStack is rejecting your storage pools and how to fix it. Root Cause AnalysisYou've hit the "Allocated Capacity Threshold", not the physical usage threshold. Here's the key moment from your logs: What this means: CloudStack maintains two separate capacity metrics for storage:
In your case:
The allocated threshold is set to 0.85 (85%) by default, and your pools are at 93.37%, so CloudStack correctly blocks new deployments to prevent overallocation. SolutionsSolution 1: Increase the Allocated Threshold (Quick Fix)Increase the
Why this works: It allows CloudStack to provision volumes up to 95% or 98% of the pool's allocated capacity, giving you room to deploy VMs. Limitation: Once you exceed the threshold again (e.g., create more volumes), you'll hit the same issue. This is a temporary fix, not a long-term solution. Solution 2: Check for Orphaned Volumes (Recommended)If you have unused volumes, detach or delete them to free up allocated capacity:
Why this works: Each volume you delete reduces the Solution 3: Monitor Allocated Capacity ProactivelyUse this API call to check allocated capacity for a specific pool: curl -k -H "Authorization: <your-auth-header>" \
"http://<ms-ip>:8080/client/api?command=listStoragePools&id=<pool-uuid>"Look for the Solution 4: If You're Using Thin ProvisioningIf your underlying storage supports thin provisioning, you can safely increase the threshold closer to 1.0 since your physical usage is low. However, be cautious — monitoring actual storage usage is critical to avoid real capacity exhaustion. 📝 Summary Table
Conclusion: Increase the allocated threshold to 95%, or clean up unused volumes. Either approach will allow your VM deployment to proceed. Let us know which approach you take, and feel free to share any additional questions! Regards, |
|
Thank you very much for your detailed reply, @RyanP-Testing. I was able to work around the issue by setting the overprovisioning factor to 2. However, I don't think this should be the expected storage allocation calculation for the volume. Based on my testing, the qcow2 file for the template is never expanded. This is what I observed:
Therefore, in both cases, we have:
From what I understand, the maximum possible physical storage consumption in this scenario should be: Template physical size + VM volume requested size For this particular case:
So the expected maximum physical storage requirement should be approximately: 20.81 GB + 1,912.00 GB ≈ 1.93 TB However, CloudStack appears to calculate the required storage as: 2 × Template virtual size = 2 × 1,912.00 GB ≈ 3.82 TB This seems to overestimate the actual possible physical storage consumption by using the template's virtual size rather than its physical size. Is the design intentionally to always use the virtual/provisioned size for the allocated-capacity calculation (regardless of thin/thick and regardless of the actual qcow2 physical size on disk)? Thanks again! |
|
Hi @mbertolina, First off, really appreciate you taking the time to dig into this and share your findings. That's some solid troubleshooting work on your part. The Short AnswerTo answer your core question – yes, this is by design. CloudStack intentionally uses the template's virtual/provisioned size for its allocated capacity calculation. It's not a bug, it's a deliberate decision to prevent storage overallocation. Why CloudStack Does ThisYou're absolutely right that physically, your calculation makes way more sense. On a thin-provisioned NFS share, the actual space consumed should be roughly:
But CloudStack isn't looking at physical disk usage – it's looking at logical allocation. Here's why:
The Fat vs. Thin DifferenceYour observation about fat vs. thin is spot on, but it's a hypervisor-level detail, not something CloudStack's allocation logic cares about. Whether the disk is fat or thin, CloudStack still looks at the virtual size when making its "do we have space?" decision. The Your Overprovisioning Factor WorkaroundSetting the Just keep an eye on the physical usage – that's the one you actually need to monitor now. Overprovisioning is great until the day everyone actually fills up their disks and you hit real storage exhaustion. Set up some monitoring alerts for the NFS share's actual usage just to be safe. My TakeawayThe design is intentional, but I'll be honest – it can be frustrating. CloudStack's allocation calculation works well for thick provisioning and environments where storage is tightly managed, but for thin-provisioned qcow2 on NFS, it often feels unnecessarily restrictive. There's been some discussion in the community about making this calculation smarter, maybe using a configurable percentage or a hybrid model that accounts for both virtual and physical sizes. But for now, overprovisioning factor adjustments are the way to go. Good catch on this, and thanks for sharing your findings – this is exactly the kind of practical insight that helps others in the community avoid the same confusion. Let me know if you run into any other quirks. Always happy to compare notes. Cheers, |
Is there a way to workaround this on CLVM_NG primary storages? Overprovisioning factor is supported on shared storage FS but I does not when using CLVM_NG. |
|
Thanks for sharing the CLVM_NG logs. In this case, increasing the storage overprovisioning factor will not help in the same way it did with the SharedMountPoint storage. From the logs, CloudStack is calculating the storage like this:
Your current pool.storage.allocated.capacity.disablethreshold is set to 0.85, so CloudStack stops the deployment once the calculated allocation goes above 85%. As a workaround, I would suggest increasing this setting to 0.95 for the affected storage pool instead of changing it globally. After changing it, please retry the VM deployment. The calculated 93.36% should then be within the allowed limit. Also, please keep monitoring the actual free space on the CLVM_NG/LVM storage, since increasing the threshold reduces the safety margin before the physical storage becomes full. One more thing: could you please confirm the exact CloudStack version currently running in this environment? Your earlier post mentioned 4.22.1.0, while CLVM_NG is associated with the newer CloudStack releases. Please let me know the result after changing the pool-specific threshold. |
|
We upgraded to Cloudstack 4.23 |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
problem
When trying to deploy a VM from a template I get "Could not find suitable Deployment Destination for this VM under any clusters, returning." because of "Insufficient un-allocated capacity on: StoragePool..."
From template information:
Hypervisor: KVM
Format: qcow2
Template physical size: 20.81 GB
Template Size: 1912.00 GB
StoragePool:
Type: SharedMountPoint
Overprovisioning factor: 1
Disk Size (in GB): 4095.5 GB (x1.0)
0.05% Used
0.00% Allocated
Looking at the logs I see that the actual filesystem/storage usage reported here is only about 0.05%.
But later the logs says:
totalAllocatedSize: 2052994367488
allocated percentage: 0.9337067261132008
That's roughly 93.37% of the pool's capacity allocated/reserved according to CloudStack, not actual bytes currently consumed on the filesystem.
why CloudStack believes each pool has ~93.37% allocated ?
while the underlying storage reports only ~0.05% physically used ?
Does it use the template root disk size instead of qcow2 physical size?
versions
Cloudstack: 4.22.1.0
KVM host: rhel9.8
Primary storage: Shared storage on GFS2
The steps to reproduce the bug
...
What to do about it?
No response
All reactions