HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 5 posts ] 
Author Message
 Post subject: Vvol use >100% and Space allocation failed write protect
PostPosted: Tue Jan 19, 2021 6:52 pm 

Joined: Sat Nov 24, 2018 4:18 pm
Posts: 35
Hello, today I experienced problem with one vvol. We use KVM virtualization on Oracle Linux 7. VM gets into pused state and in dmesg I found this "Space allocation failed write protect" on exported lun slaves.

I resumed VM and it is working fine, but I investigate futher and found that 3PR reports that vvol used more that 100% space!

Code:
3PAR8450 cli% showvv -s feldcloud.one.89.vv
                                              ----------Snp----------- ------------Usr------------ -----------Total-----------                 
                                              ---(MiB)--- -(% VSize)-- ----(MiB)---- --(% VSize)-- -----------(MiB)----------- ---Efficiency---
 Id Name                Prov Compr Dedup Type  Rsvd  Used Used Wrn Lim   Rsvd   Used  Used Wrn Lim   Rsvd   Used HostWr  VSize Compact Compress
393 feldcloud.one.89.vv tdvv Yes   Yes   base 16384 13375  3.7   0   0 366720 358775 100.1   0   0 383104 372149 249942 358400    1.93     1.14
-----------------------------------------------------------------------------------------------------------------------------------------------
  1 total                                     16384 13375              366720 358775               383104 372149 249942 358400   


I also tried to issue "fstrim -av" command in the VM, but only just "Host written" field dropped to real value.

Any advice on this? Thank you!


Top
 Profile  
Reply with quote  
 Post subject: Re: Vvol use >100% and Space allocation failed write protect
PostPosted: Wed Jan 20, 2021 1:25 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
I'm making an educated guess that you are using tdvv2.

For tdvv2, the usr used and usr rsvd are weighted number based on data stored in DDS and data stored in DDC.

Based on those numbers I assume that your DDS is quite big and you're getting bad dedupe.

With tdvv3, those number only contain the DDC numbers.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: Vvol use >100% and Space allocation failed write protect
PostPosted: Wed Jan 20, 2021 5:17 am 

Joined: Sat Nov 24, 2018 4:18 pm
Posts: 35
Hello, thank you for reply. It is 8450 system deployed to the production with 3.3.1 MU3, so it should be tdvv v3? How I can check?

EDIT: CPG is using version 3

Code:
3PAR8450 cli% showcpg -d
                        ----Volumes---- -Usage- ------------(MiB)------------- --LD--- -RC_Usage- -Shared-
Id Name           Warn% VVs TPVVs TDVVs Usr Snp    Base     Snp   Free   Total Usr Snp  Usr   Snp  Version
 0 SSD_r1             -   0     0     0   0   0       0       0      0       0   0   0    0     0        -
 1 SSD_r5             -   0     0     0   0   0       0       0      0       0   0   0    0     0        -
 2 SSD_r6             - 569     0   568 569 568 7840256 1165952  62336 9068544   0  37    0     0        3
 4 SSD_r6_windows     -   6     0     5   6   5   91136   10240  46080  147456   0   4    0     0        3
----------------------------------------------------------------------------------------------------------
 4 total                                575 573 7931392 1176192 108416 9216000   0  41    0     0         


EDIT 2:

Total used raw capacity is 75%, it is a problem? I think that there is still 25% free for allocations. All other VVs are without problems, only this one. Same problem hapens today also.

EDIT 3:

Something happens in 3PAR and space allocations are diferent regardless of real usage in the VM

Code:
3PAR8450 cli% showvv -s feldcloud.one.89.vv
                                              ----------Snp----------- -----------Usr------------ -----------Total-----------                 
                                              ---(MiB)--- -(% VSize)-- ----(MiB)---- -(% VSize)-- -----------(MiB)----------- ---Efficiency---
 Id Name                Prov Compr Dedup Type  Rsvd  Used Used Wrn Lim   Rsvd   Used Used Wrn Lim   Rsvd   Used HostWr  VSize Compact Compress
393 feldcloud.one.89.vv tdvv Yes   Yes   base 39936 39936 11.1   0   0 366720 293505 81.9   0   0 406656 333441 260737 358400    1.07     1.15
----------------------------------------------------------------------------------------------------------------------------------------------
  1 total                                     39936 39936              366720 293505              406656 333441 260737 358400


Top
 Profile  
Reply with quote  
 Post subject: Re: Vvol use >100% and Space allocation failed write protect
PostPosted: Thu Jan 21, 2021 5:50 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
Hehe. I'm impressed :)

I would probably log a support case and ask how it is possible for Usr Used to be higher that VSize on a non-tdvv2 volume :)

The only thing I could possibly think of would be some garbadge collection or something to that extent where data is recorded to no longer be in use while a new write isn't allowed to overwrite. At that point the data might still be considered a part of the volume, but probably not "assigned" to a "block number" or something. However it seems far-fetched and I'm just making wild guesses here.

If you'ld seen this on a "plain" thin volume, I would have been surprised... Big time.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: Vvol use >100% and Space allocation failed write protect
PostPosted: Thu Jan 21, 2021 8:52 am 

Joined: Sat Nov 24, 2018 4:18 pm
Posts: 35
hi, looks like it is related to fstrim and garbage collection. We recently added fstrim to our backup script, so each VM do it before backup. Then this happens on two bigger vvols from about 400. Yesterday, when it magically goes back I see defragmentation jobs in system activity running.

I solved issue by creating physical copy of affected VV.

I am going to write to the HPE about this, I am on latest 3.3.1 MU5 with all patches


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 5 posts ] 


Who is online

Users browsing this forum: Google [Bot] and 34 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt