HPE Storage Users Group

You need to put the -s option on your showvv to better monitor the situation. With out it, you only see the vv reserved space, not how much of that is actually used. The actual used user space should update in near real time as your zero process is working.

A vv will keep a certain amount if space in reserve for immediate growth, and thin reclamation wont reclaim every kb down to the bare "wick" on purpose to avoid a back-and-forth condition of allocate-reclaim-allocate-reclaim... which could excessively fragment a VV if allowed.

I am not sure the exact cut off point... but I ~approximate that when a VV has over 8gb in reserve is when the system will give space back to the CPG.

Remember there are multiple levels of free space in the system. There is unallocated space, plus free space reserved inside CPGs, plus free space reserved inside VVs. Zeroing a VV starts by converting VV "used" space into VV "reserved" space... then some background process will come around eventually and trim any excess VV reserved space back to the CPG... then you can compact a CPG to convert any excess free space back to unallocated space.

_________________
Richard Siemers
The views and opinions expressed are my own and do not necessarily reflect those of my employer.

I've done some testing and it does work on 7400 (3.1.2 MU3), but it take over 12 hours to see results. I've tested by creating 200GB vmdk file full of random data and deleting it after.

The problem I am seeing is with 2008 R2 (or older) Eager Zeroed vmdks.
For example I have 200GB vmdk cleaned with sdelete inside Server 2008 R2. Next we run vmkfstools -y on the ESXi datastore but it doesn't detect these 200GB to be available for unmap.

There is a utility from VMware labs, which does reclaim on OS level (works just like sdelete) but it only works with thin vmdks https://labs.vmware.com/flings/guest-reclaim

Is there a way to reclaim zeroed blocks inside OS on Eager Zeroed vmdks?

Been facing this with my 7200 all week. I'm running ESXi 5.5, so I use the "esxcli storage vmfs unmap -l datastorename"

I've run this on each of my stores, and run the compact CPG as well. I've regained so little space compared to what I've deleted, its blowing my mind.

I see the guest reclaim link you posted... if I delete a large amount of files from a Eager Thick Zeroed VM, does it send those string of zeroes, tripping the inline zero data deduplication, and give me my space back... or is sDelete / guest reclaim tool a necessary?

Also, do I need a maintenance window to run tuneVV, and would it help me at all as far as trying to reclaim? Sorry for all the questions on your topic, just didn't want to start a new one when my problem seems so exactly similar given the arrays.

And just to top mine off... after the compaction I had gained about 100 gigs back... which was good... I was down to 492GB on the array!

Was pretty happy to see 584 usable free afterwards... but about 10 minutes after that... getting ready to start adding up real numbers and see where I SHOULD be... I run the showspace command again.... and I'm down to 464GB! Ugh... perhaps even though the UNMAP commands act like they complete, and the CPG compaction says its complete... stuff is still moving in the background?

I have been thoroughly and utterly un-impressed with the 3pars in the thin reclaim department, this were the big sell from HP but in practice space is very difficult to reclaim (or even accurately calculate)
I was so annoyed that with the third 3par we put in we ignored 3par Thin and provisioned all our Volumes thick and used VMWare thin disks which is a whole lot easier to manage.

Brett

So here's what I've concluded after running some tests for VMware, Eager Thick vmdks and 3par:

esxcli storage vmfs unmap in 5.5 or vmkfstools -y in 5.1 command on the datastore itself will unmap only free space as seen by ESXi. For example you have the following:

2TB thin volume on 3par, reserved user size 100%
This volume is an ESXi 2TB VMFS5 datastore, which has 1.5TB of Eager Thick vmdk files

When you issue unmap command from ESXi it will free up around 0.5TB. ESXi doesn't see free space inside thick vmdk. You will see this reflect on 3par's Volume Reserved user size in approximately 12 hours time (in my case on 7400).
---------------
Now, we look one level below at the OS level inside vmdk files. By default Windows Server prior to 2012 does not zero deleted space so we have have to use sdelete or powershell script (please see below) to write zeros in empty space. For example we have the following:

2TB thin volume on 3par, reserved user size 100%
This volume is an ESXi 2TB VMFS datastore, which has 1.5TB Eager Thick vmdk file
This vmdk file is Windows Server 2008 R2, which shows 1TB free (inside the OS)

Then you zero empty space inside OS (either by sdelete or powershell script) VMware volume will be still showing as 1.5 out of 2TB used. 3par will detect these zeros and it will be reflected in Reserved User Size, it will detect approximately 1TB free (which is returned to the pool).

Conclusion:
In order to "return" unused space on 3par thin provisioned volumes back to the pool you need to run unmap on ESXi datastore as well as zero-delete inside vmdk on OS level as they address different portions of "free" space. Additionally, "free" space will not reflect on 3par instantly; please allow at least 12 hours (maybe someone can comment more on this).

Appendix
Powershell script, alternative to sdelete
http://blog.whatsupduck.net/2012/03/pow ... elete.html

For Linux "zeroing" read this page:
http://intgat.tigress.co.uk/rmy/uml/index.html

All good stuff and very true and I guess different people see the space consumed in different places in their environment. We usually see the majority of consumption happen at the LUN/Datastore layer though, mainly because of the amount of Storage Vmotions and snapshits that are happening in our environment. The constant battle with space reclamation is a major pain point.

Brett

You can Powercli unmap script and schedule to run it overnight. It can also be achieved via vMA, Orchestrator or cron on the ESXi host itself.

The way VMware unmap reclaims space is based effectively on a multivendor compatibility mode, which has to support the lowest common denomimator zero reclaim functionality across mutliple arrays with differing capabilities , hence why it's not automatic. Neither is it very aggressive or particularly efficient and it doesn't actively make use of the ASIC. Although the ASIC will still detect and skip writing the zeros from the balloon file to backend disk, which most other arrays will not. For now this is how VMware have chosen to architect the feature to provide the broadest level of support across vendors.

Just to clarify the hypervisor adds layers of abstraction so space reclaim needs to be tackled on two fronts. Firstly to free up space within a vmdk file you need to run a host based tool such as sdelete, dd or similar as VMware has no knowledge of the freed space within the underlying file system. Then in order to free up space from a vmfs after deleting vmdk files you need to run vmkfstools from ESX which will create the balloon file and force a reclaim back from the datastore.

In terms of space being returned to the pool, 3PAR makes sure the space is zero's out before the space is returned in full for reuse which is a background process intended not to impact FE host IO, which in turn ensures space doesn't have to be zero'd on allocation which would impact new incoming writes to TPVV's. Take a look at the concepts guide for more on how things work under the covers. If you run this is in a real environment you will see significant savings over time, not just in reclaimed space but in slower growth of storage, not all savings are related to zero detect and zero reclaim as 3PAR isn't just about these features it's more the whole (reservationless) package.

Also absolutely make sure you are looking at "Used User Size" and NOT "Reserved User Size", if you are looking at Reserved User Size you have to wait for the array to get around to the reclaim, so don't expect to see the space as soon as the delete button gets pressed.

Fantastic post nsnidanko... thanks for the help.