solidfire primary capacity is reset to zero bytes

ISSUE TYPE

Bug Report

COMPONENT NAME

Primary iSCSI solidfire

CLOUDSTACK VERSION

4.15.0 (after upgrade from 4.11.3)

CONFIGURATION

zone wide storage

OS / ENVIRONMENT

SUMMARY

a 4.11.3 environment with solidfire zone wide primary storage is upgraded to 4.15.0. After the upgrade the capacity of the storage is set to 0 bytes. After adjusting it it will be reset within a few minutes to 0 bytes again.

STEPS TO REPRODUCE

EXPECTED RESULTS

This happened in 4.11.3 and was expected in 4.15.0 as well

2021-04-12 10:01:25,843 DEBUG c.c.s.StorageManagerImpl (logid:) Found storage pool London Primary Storage of type Iscsi
2021-04-12 10:01:25,843 DEBUG c.c.s.StorageManagerImpl (logid:) Total over provisioned capacity of the pool London Primary Storage id: 2 is 17592186044416
2021-04-12 10:01:25,844 DEBUG c.c.s.StorageManagerImpl (logid:) Successfully set Capacity - 17592186044416 for capacity type - 3 , DataCenterId - 2, HostOrPoolId - 2, PodId null

ACTUAL RESULTS

2021-04-12 09:17:26,370 DEBUG c.c.s.StorageManagerImpl (logid:) Found storage pool London Primary Storage of type Iscsi
2021-04-12 09:17:26,371 DEBUG c.c.s.StorageManagerImpl (logid:) Total over provisioned capacity of the pool London Primary Storage id: 2 is (0 bytes) 0
2021-04-12 09:17:26,372 DEBUG c.c.s.StorageManagerImpl (logid:) Successfully set Capacity - (0 bytes) 0 for capacity type - 3 , DataCenterId - 2, HostOrPoolId - 2, PodId null

@mike-tutkowski

Hi @mike-tutkowski - do you have any guidance for us?

Hi @rhtyd: In the log lines above, it looks like the capacity was already at 0 by the time the createCapacityEntry method (which is where those log lines come from) is invoked.

I don't currently have a CloudStack environment up and running. Would you (or someone else) be able to re-set the capacity to the desired value, wait for it to be set back to 0, and then update this ticket with the log file?

Thanks!

I found this telling bit: probably why further calculation is stopped and 0 is passed to createCapacityEntry

2021-04-12 00:00:33,676 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-b9094408) (logid:fc1ff0ae) Seq 2-8552898642330477068: Received:  { Ans: , MgmtId: 2482650231502, via: 2(pool1-kvm1.sys.lon1.cloudsys.tmcs), Ver: v1, Flags: 10, { Answer } }
2021-04-12 00:00:33,676 DEBUG [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-b9094408) (logid:fc1ff0ae) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: com.cloud.utils.exception.CloudRuntimeException: Wrong format for iScsi path: f7bf6389-cb9a-4360-8116-2f2ed85b6220. It should be formatted as '/targetIQN/LUN'.
	at com.cloud.hypervisor.kvm.storage.IscsiAdmStorageAdaptor.getComponent(IscsiAdmStorageAdaptor.java:272)
	at com.cloud.hypervisor.kvm.storage.IscsiAdmStorageAdaptor.getIqn(IscsiAdmStorageAdaptor.java:257)
	at com.cloud.hypervisor.kvm.storage.IscsiAdmStorageAdaptor.getByPath(IscsiAdmStorageAdaptor.java:217)
	at com.cloud.hypervisor.kvm.storage.IscsiAdmStorageAdaptor.getPhysicalDisk(IscsiAdmStorageAdaptor.java:222)
	at com.cloud.hypervisor.kvm.storage.IscsiAdmStoragePool.getPhysicalDisk(IscsiAdmStoragePool.java:109)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
	at com.cloud.agent.Agent.processRequest(Agent.java:640)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1053)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

I'm not really sure what this createCapacityEntry is exactly supposed to be doing, but does it apply for managed storage?

@swill

cc @swill @skattoju4 - can you have a look at advise?

@DaanHoogland

@DaanHoogland when you say 'adjusting' it, was the storage pool capacity value manually updated using the API http://cloudstack.apache.org/api/apidocs-4.15/apis/updateStoragePool.html ?

Looking into the codebase, it looks if the capacity value is getting reset to 0, it's triggered via a thread, most likely on pool connect/add (when the mgmt server starts or kvm host connects I think) or likely by the stat collector:
https://github.com/apache/cloudstack/blob/4.15/server/src/main/java/com/cloud/server/StatsCollector.java#L1026

If it's the stats collector, then the GetStorageStatsCommand answer has capacity 0 returned. In case of solidfire on KVM, it could be resulting from https://github.com/apache/cloudstack/blob/4.15/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtGetStorageStatsCommandWrapper.java#L39 which calls over few methods/logic:
https://github.com/apache/cloudstack/blob/4.15/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/IscsiAdmStoragePool.java#L63

The code from the above class returns capacity as 0. @mike-tutkowski @swill @skattoju4 can you check/advise?

server: Fix regression in storage pool stat update

I found some changes in code from 4.11.1 to 4.15, the logic in stat collector may end up updating for storage plugin it shouldn't which don't implement the total/used value to be returned. Proposed PR https://github.com/apache/cloudstack/pull/4918/files

@rhtyd it was updated both with the API and directly in the DB.

if a message from the host saying "Wrong format for iScsi path: f7bf6389-cb9a-4360-8116-2f2ed85b6220. It should be formatted as '/targetIQN/LUN'." is the issue, will than not the new value always be 0 and this fix still do the resetting? Retorical question, we can only let this be tested by a user using KVM with the solidfire plugin.

btw, i marked this as CRITICAL , but we might call it a blocker as well for those users.

@rhtyd will take a look. In the past storage used was computed with getUsedBytes which summed up the size of volumes templates and snapshots from the db. The capacity was set manually to a high value since the actual capacity was somewhat dynamic depending on the storage usage.

@skattoju4

Like what @skattoju4 mentioned: It used to be that you would just manually set the capacity bytes for managed storage.

The idea is that you might have a large SAN (like a SolidFire SAN), which does all sorts of fancy things like compressing and de-duplicating data - in addition to thin provisioning. That being the case, whatever the physical storage capacity is of this kind of a SAN, you might get, say, 2 - 3x more out of it because of those data-efficiency techniques (of course, depending on the type of data you're storing).

In any case, to make this easy to use from CloudStack, the admin just sets a capacity number manually. It can be large or small. This number should not automatically be updated at any point: It's just a number that an admin can manually update.

As an example, let's say you pick a (small) number like 100 GB. If you create a volume that has a 10 GB virtual size, then you now only have 90 GB left from this managed storage. If you create a back-end snapshot on this 10-GB volume, that takes another 10 GB off of the 100 GB of the managed storage (even though neither the volume nor the snapshot is likely really taking10 GB of physical capacity on the SAN).

Sorry...I didn't mean to close this ticket. There was something weird going on in the UI. I thought I was cancelling a comment that I was typing, but it ended up closing the ticket. I've re-opened the ticket now.

server: Stat collector solidfire capacity fix (#4918)

Fixed in #4918

Navigation Menu

Repository navigation

solidfire primary capacity is reset to zero bytes #4911

ISSUE TYPE

COMPONENT NAME

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

solidfire primary capacity is reset to zero bytes #4911

Description

ISSUE TYPE

COMPONENT NAME

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

Activity

yadvr commented on Apr 12, 2021

mike-tutkowski commented on Apr 12, 2021

DaanHoogland commented on Apr 13, 2021

mike-tutkowski commented on Apr 13, 2021

yadvr commented on Apr 13, 2021

yadvr commented on Apr 13, 2021

yadvr commented on Apr 13, 2021

DaanHoogland commented on Apr 13, 2021

DaanHoogland commented on Apr 13, 2021

DaanHoogland commented on Apr 13, 2021

skattoju4 commented on Apr 13, 2021

mike-tutkowski commented on Apr 13, 2021

mike-tutkowski commented on Apr 13, 2021

yadvr commented on Apr 21, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions