HPE Storage Users Group

Hi,

So I have a customer who managed to run of space on his 3PAR.

it is 12 disk SSD system with 480GB cMLC disks.

The issue was like this: Customer filled up one of his VV(SSD_R5_TP_01) - when he got the 95% warning mail, he then tried to move the VM to another VV(SSD_R5_TPD_01) in the same CPG

First the CPG failed then the SSD_R5_TPD_01 volume failed because it could not reserve more space from the CPG, we were able to compactcpg just enough to get things going again to move some VMs(all VV are for VMFS) to another storage and run unmap to get things going again.

Now the one thing that bothers me is this: The "SSD_R5_TP_01" stil has almost all reserved space allocated to it(almost 600GB more than it needs) leaving almost nothing to the other VV's

The CPG is using 99% of the RAW 3PAR capacity(just 50GB left):
Is there something you can do to free the reserved user space for the "SSD_R5_TP_01", that would in turn free up space on the CPG ?
I have see that in the past 12 hours the reserved space has shrunk with 20GB.

From what I have heard this is a backround process that you cant control?

I have created a case with HP - but havent had any reasonable response yet.

Output below


showvv -s
----Adm---- ---------Snp---------- -------------Usr-------------
---(MB)---- --(MB)--- -(% VSize)-- -----(MB)------ --(% VSize)-- ------(MB)------- -Capacity Efficiency-
Id Name Prov Type Rsvd Used Rsvd Used Used Wrn Lim Rsvd Used Used Wrn Lim Tot_Rsvd VSize Compaction Dedup
1 .srdata full base 0 0 0 0 0.0 -- -- 61440 61440 100.0 -- -- 61440 61440 -- --
0 admin full base 0 0 0 0 0.0 -- -- 10240 10240 100.0 -- -- 10240 10240 -- --
16 SSD_R5_TP_01 tpvv base 1920 1681 0 0 0.0 -- -- 2622464 1996853 47.6 0 0 2624384 4194304 2.1 --
17 SSD_R5_TP_02 tpvv base 640 404 0 0 0.0 -- -- 637440 618464 14.7 0 0 638080 4194304 6.8 --
11 SSD_R5_TPD_01 tdvv base 33792 33370 512 0 0.0 0 0 474112 443430 10.6 80 0 508416 4194304 8.8 1.6
13 SSD_R10_TP_01 tpvv base 256 3 0 0 0.0 -- -- 2560 18 0.0 0 0 2816 2097152 102848.8 --
-----------------------------------------------------------------------------------------------------------------------------------
6 total 36608 35458 512 0 3808256 3130445 3845376 14751744 4.7 1.6


showcpg -r
--------------------------------------(MB)---------------------------------------
----Volumes---- -Usage- ------------- Usr ------------- --------- Snp --------- ---------- Adm ----------
Id Name Warn% VVs TPVVs TDVVs Usr Snp Total RTotal Used RUsed Total RTotal Used RUsed Total RTotal Used RUsed
0 SSD_r1 - 1 1 0 1 0 2560 5120 2560 5120 9728 19456 0 0 10240 30720 256 768
1 SSD_r5 - 3 2 1 3 1 3734016 4480815 3734016 4480815 13824 16593 512 614 45056 135168 36352 109056
2 SSD_r6 - 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
-------------------------------------------------------------------------------------------------------------------------
3 total 4 1 3736576 4485935 3736576 4485935 23552 36049 512 614 55296 165888 36608 109824


showsys
---------------(MB)----------------
ID --Name--- ----Model---- -Serial- Nodes Master TotalCap AllocCap FreeCap FailedCap
xxxxxx HP_3PAR 7200c 1684737 2 1 5480448 5429248 51200 0

How much space have you freed from SSD_R5_TP_01 ? If you've run unmap then the VV user space should reduce immediately yet it looks like you still have a lot of user space in there. This can't be reclaimed to the CPG until it is freed, so the system will only attempt to return the difference between user and reserved and if that is fragmented it will potentially only return slowly.

Run an unmap or similar e.g sdelete in the guest, because your reserved capacity appears to be your user size plus parity overhead @5+1, which would mean as far as the system is concerned the VV has no unused space in the written data, finally run a compact CPG to return space to the system.

JohnMH wrote:How much space have you freed from SSD_R5_TP_01 ? If you've run unmap then the VV user space should reduce immediately yet it looks like you still have a lot of user space in there. This can't be reclaimed to the CPG until it is freed, so the system will only attempt to return the difference between user and reserved and if that is fragmented it will potentially only return slowly.

Run an unmap or similar e.g sdelete in the guest, because your reserved capacity appears to be your user size plus parity overhead @5+1, which would mean as far as the system is concerned the VV has no unused space in the written data, finally run a compact CPG to return space to the system.

Hi

Thanks for your feedback - we deleted about those 600GB that is the difference, I dont think its the raid overhead? I did not think "showvv -s" incuded the raid overhead.. This is what it shows if i specifiy showvv with showing raw space, we have run the vmware unmap command after storage vmotioning them..

showvv -r
-------------------------------(MB)--------------------------------
-----Adm----- ----Snp----- ------Usr------ ------Tot------
Id Name Prov Type RawRsvd Rsvd RawRsvd Rsvd RawRsvd Rsvd RawRsvd Rsvd VSize
1 .srdata full base 0 0 0 0 73728 61440 73728 61440 61440
0 admin full base 0 0 0 0 20480 10240 20480 10240 10240
16 SSD_R5_TP_01 tpvv base 5760 1920 0 0 3146954 2622464 3152714 2624384 4194304
17 SSD_R5_TP_02 tpvv base 1920 640 0 0 764927 637440 766847 638080 4194304
11 SSD_R5_TPD_01 tdvv base 101376 33792 614 512 568934 474112 670924 508416 4194304
13 SSD_R10_TP_01 tpvv base 768 256 0 0 5120 2560 5888 2816 2097152
----------------------------------------------------------------------------------------------
6 total 109824 36608 614 512 4580143 3808256 4690581 3845376 14751744

I was looking at the showvv -r rather showvv -s hence the raid comment.

Looking at the showvv -s
Usr Rvsd is 2622464, Usr Used is 1996853 so that's your 610GB reclaimed to the VV by the unmap, however in order to return those 610GB's of 16KB pages to the CPG they must be 128MB contiguous regions, so a defrag may need to happen before the LD's can be truncated and the space returned to the CPG for a subsequent compaction.

If you had space then you could force the defrag by converting the TPVV to a FPVV and then back to a TPVV. This forces a defrag as part of the volume creation, removal but without the space for this you'll have to wait, unless support has a quicker method maybe at the LD level.

Not really a great situation to get into.

You could try and run compactld and see if that helps

First check with a dry run to see how much you can reclaim.

compactld -dr -pat *

You might also look into doing it with the -cons option too , but then you need to do it on all lds to one CPG at the time.

You can run the showld command to see what the name is on the system.

But then again if the system is almost full i would first check with HP support to do it the correct way.

Hi seritlj,

Were you in Barcelone this week ? Because during HP presentation on Peer Persistence there was exactly this question ^^