HPE Storage Users Group

Hey,

I have a hard time understanding how 3par will release reserved space on tpvv volumes.

From what I've read on the forum the assumptions for tpvv volumes seem to be:

"Used Disk space" (as shown by "showvv -s") is reclaimed on a 16kb chunk level, either by

* writing 0s (and zero_detect enabled on the volume)
* sending UNMAP/TRIM commands

as well as having the Thin Persistence license installed in the system.

This seems to be true - the "Used" column reflects the numbers reported by the specific filesystem very accurately.


What I don't understand is how "Reserved Disk Space" reclaiming works. When a volume fills up, reserved disk space grows accordingly, but it never seems to go noticeably back down for us.

This is a volume I filled up to ~170GB, then blkdiscarded and then overwritten with zeros for good measure. This shrunk Reserved space down to ~140G but stays there now.



A few other offenders from our system:



Judging from the documentation "tunevv" looks like as if it could save space by rearranging the VV-data <-> LD mapping, but we lack the Dynamic Optimization license for that.

Is there anything else we could try to get the reserved VV size down again or at least understand what's going on?

The difference between “Reserved User Size” and “Used User Size” There needs to be a small amount of free buffer space for new writes to land in when they come from a host. This is the “Reserved User Size”. Then there is space that holds the bits of data that the host has actually written this is “Used User Size”.

The “User Used Space” is reclaimed at 16K contiguous blocks real time and freed back to the virtual volume for ongoing writes. Volumes that have crossed the 4GB written per node pair boundary will never be reduced beyond this size, this is in order to preserve performance of the volume and prevent having to delete and recreate LD's unnecessarily.

The “User Reserved Space” is reclaimed at 128MB contiguous regions and given back to the VV's underlying CPG. This 128MB is less granular so you won’t reclaim back as much or as quickly to the CPG. Version 3.1.1 firmware onwards has functionality added to do a scheduled garbage collection in which the array will attempt to move released 16KB blocks to the start of a Logical Disk (LD) in order to create a contiguous 128MB region, which can then be released back to the original CPG by shortening the underlying LD.

Use shovv -s to look in the “Usr Used” column in the CLI which will reclaim quickly. The “Usr Reserved” space will reclaim less quickly as this typically requires the garbage collection process to kick in and run, which from memory is every 90 minutes but can be deferred by other processes and may take longer dependent on the fragmentation. Typically you see this on test systems where VV's are being created, reclaimed, copied and recreated etc with little time for the system to reclaim space within the LD's.

To then release this reclaimed space from the CPG to the system free pool, you’d need to run “compactcpg” from the command line or you can right click on the CPG in the GUI and select compact. Also typically you'd have a compactcpg run regularly via the scheduler preferably at a less busy time to enable the periodic reclaim of released space from the CPG’s to the system free pool.

Tunevv aka DO would allow you to move things about and effectively defrag them on the fly, I really never understand why people try to pass on DO, IMHO it's probably one of the most useful licenses of all. You could speak to the HP account team who may be able to provide temporary licensing. Tunesys might also be an option but I wouldn't use this without the added safety net of DO to allow you to reverse out any changes.

Also I would try a few runs of zero's simply because file systems differ in how they allocate space and won't always overwrite old space, many tend to prefer to write to new space to preserve the ability to undelete etc. In which case the existing blocks (deleted from the filesystems perspective) won't be deallocated.

Cleanur, thank you very much for your answer! You're correct, something has changed in the last few days, these are the respective volume sizes at the moment:



For "alloctest" the size dropped to the 4GB per node limit (2 node cluster), for the other two volumes this hasn't changed noticeable.

We're using ext4 on these volumes. Ext4 partitions a block device into block groups of 128MB per default - see https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout for details. All these block groups are uninitialised by default and will be created on the first write in the block group. When you delete data again I expect that a few bytes will remain set in the respective block groups.

Will a few bytes in each 128MB LD region of a volume prevent 3par from reclaiming them or is there something else going on?

Not entirely sure, have a look at this whitepaper page 23, it could be space isn't actually being deallocated or some vague behavior of the file system or overlying application. Ext4 is thin friendly and supports T10 Unmap (discard) with the right options on mount or fstrim for ongoing thinning, so either of those may be a better method to automate things.

http://www8.hp.com/h20195/v2/GetDocumen ... A3-8987ENW

Yeah, we've got all volumes mounted with discard and the "Used" space shrinks accordingly, so this part works fine.

If I find time I'll do a few test runs with different filesystems and see how it goes.

Regarding the background garbage collection which will cut down the reserved space - there's currently no way to introspect these processes and see what they're doing, or is there?

Sorry no way to monitor, it's a low level system task so won't show in the tasks or scheduler.

The actual reclaim process wakes up every few minutes to scan for volumes with reserved user space available for reclaim.The garbage collection operation is only started if this standard reclaim fails to return space due to fragmented pages (remember LD's are shared). The process will only actually happen at best every ~8 hours, however based on the dispersal level of the pages it's not always possible to return every page immediately. If the level is high then the process could potentially be I/O intensive, so the system will initially look for the low hanging fruit. Over a period of time as the volume and underlying LD's age, then potentially more free blocks accumulate and space becomes easier to return.

Outside of artificial testing of the reclaim process, monitoring this typically isn't a priority as the free user space is always returned immediately to the volume for regrowth as you've already seen. Very few real world volumes shrink and grow at such a regular rate that this is a factor and typically reserved space is measured in a few GB's. Also if all space were immediately returned, then potentially new LD's have to be created to accommodate new incoming writes to the volume. So the system tries to strike a balance between space efficiency and maintaining consistent performance, as such in the real world watching this number for the return of a few GB's isn't that relevant.

It works as above today, but like all 3PAR features the current functionality isn't necessarily set in stone, this is a platform with a lot of head room and who's technology and features are continually evolving.