HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 8 posts ] 
Author Message
 Post subject: Adaptive Optimisation lower tier past 95%
PostPosted: Thu Jun 04, 2020 3:57 am 

Joined: Thu Jun 04, 2020 3:39 am
Posts: 4
Hi, bit of background first, my team has inherited a badly managed 3par. It is a 4 node 7400 split between 2 DC's with 3 tiers, 8TB SSD, 150TB FC and 120TB NL tier. We have no documentation about the current setup and a full mix of AO and pinned volumes in all tiers. We've found that AO on one of the arrays is almost filling the SSD tier and NL tier, the NL tier is currently 97% and increasing day on day. The previous admin kindly suggested that is NL filled to 100% the arrays would cease writing and everything I've read suggest that is a tier runs out of space it will write to the lower tier at slower speed so his comment makes sense at this point. I've tried moving some pinned data up to the FC tier but AO then uses the space on NL again. I've looked as maximum space usage for the AO config and considered switching on of the configs from balanced to performance but unsure what the best option is. If I set to performance and we have some pinned volumes in SSD might this fill the SSD tier? If I set the Maximum Space setting in AO to be less than the current space used will the next run of AO then move data up to FC?

Thanks for taking the time to read.


Top
 Profile  
Reply with quote  
 Post subject: Re: Adaptive Optimisation lower tier past 95%
PostPosted: Thu Jun 04, 2020 4:24 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
dabravokid wrote:
Hi, bit of background first, my team has inherited a badly managed 3par. It is a 4 node 7400 split between 2 DC's with 3 tiers, 8TB SSD, 150TB FC and 120TB NL tier. We have no documentation about the current setup and a full mix of AO and pinned volumes in all tiers. We've found that AO on one of the arrays is almost filling the SSD tier and NL tier, the NL tier is currently 97% and increasing day on day. The previous admin kindly suggested that is NL filled to 100% the arrays would cease writing and everything I've read suggest that is a tier runs out of space it will write to the lower tier at slower speed so his comment makes sense at this point. I've tried moving some pinned data up to the FC tier but AO then uses the space on NL again. I've looked as maximum space usage for the AO config and considered switching on of the configs from balanced to performance but unsure what the best option is. If I set to performance and we have some pinned volumes in SSD might this fill the SSD tier? If I set the Maximum Space setting in AO to be less than the current space used will the next run of AO then move data up to FC?

Thanks for taking the time to read.


Hi.

Nothing you're stating here tells me that this is a badly managed 3PAR. If this 3PAR has all volumes and all snapshots configured at FC tier, I would actually go as far as saying it's a perfectly managed 3PAR.

A few facts.
If a tier runs out of capacity, the 3PAR will allocate capacity from a lower tier.
When you've ran out of capacity on the lowest available tier, the CPG can't grow, so the VVs can't grow and eventually the hosts will get "disk full" messages when writing new data that will most likely cause them to crash. As their drives as full, they could also have issues booting back up again.

The usual configuration of a 3PAR with AO at the time the 7400 was new and shiny was to configure all tiers in one AO config and configure User CPG to a FC CPG in the AO config and Copy CPG to a non-AO FC CPG. In this configuration all new writes will hit FC and you can safely allow AO to fill both SSD and NL to 100%. If you have as little as one volume mis-configured to either write directly to SSD (or even worse, NL) the above statement is void and it is no longer safe to allow AO to fill SSD and NL tier to 100%. In a clean AO system (not running thin or deduped volumes directly on SSD), you are always aiming at maintaining the SSD layer as close to 100% as possible.


You're not saying anything about OS versions or what you're using to manage the system. With 3.2.2 (and later) HPE introduced the t0/t1/t2 min/max settings in the AO config that will allow administrators to configure both a minimum and maximum value per tier/cpg that AO will follow. Prior to 3.2.2 the only option was to use the "growth warning" parameter per CPG to ensure the maximum amount of data AO would put on the CPG/tier (basically the same as t0/t1/t2 max setting). In SSMC, all settings as available. In IMC you can only configure growth warning.



I'm also taking a stab at this one as this is an old system in probably in old configuration with little to no documentation. If you are using Vmware I'm pretty sure that you're runing VMFS5. VMFS5 does not do automatic reclaim and I would very much assume that you have a lot of allocated capacity that AO has moved down to NL which doesn't really contain any data. Vmware is "stupid" in that configuration and when you delete data from a VMFS (either deleting snapshot, deleting VM or using storage vMotion), Vmware will only delete the pointer in their file system and not cleanup the actual data on the datastore for 3PAR to reclaim. See this KB from Vmware on the subject: https://kb.vmware.com/s/article/2057513

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: Adaptive Optimisation lower tier past 95%
PostPosted: Thu Jun 04, 2020 5:06 am 

Joined: Thu Jun 04, 2020 3:39 am
Posts: 4
Hi, thanks for the reply. The OS version we are using is 3.4.0.177 and I'm using the web gui to manage. Yes we do have a sizeable VMware workload in the 3par with some VMFS5 datastores so I will have a look at these and see if I can run a reclaim or move some data into a VMFS6 and then remove the old stores.

The bad news from what you have said is that we do have CPG and copy CPG that are in the SSD and NL as well as FC. Would the next piece of work be to look at tuning all these into FC so AO can take over safely?

As you mentioned we do have the option to set a Maximum Size on a Tier and I'm just wanting to know if I set this now will it then move some data up in the next AO cycle?

Thanks again.


Top
 Profile  
Reply with quote  
 Post subject: Re: Adaptive Optimisation lower tier past 95%
PostPosted: Thu Jun 04, 2020 5:25 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
AO will always look at those values when it runs (or to be more extact, when it starts and plans the entire task).

3.4.0.177 sounds like the version of SSMC and not 3PAR OS.

If you log on to CLI (with SSH) and run "showversion" you will find the 3PAR OS version.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: Adaptive Optimisation lower tier past 95%
PostPosted: Thu Jun 04, 2020 5:35 am 

Joined: Thu Jun 04, 2020 3:39 am
Posts: 4
Sorry, 3.2.2.709 (MU6)+P99,P115,P119,P126 is the OS version we are running.


Top
 Profile  
Reply with quote  
 Post subject: Re: Adaptive Optimisation lower tier past 95%
PostPosted: Thu Jun 04, 2020 6:28 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
dabravokid wrote:
Sorry, 3.2.2.709 (MU6)+P99,P115,P119,P126 is the OS version we are running.

Then you should have the t0/t1/t2 min/max setting available.

If/when you start looking at unmap, be advised that there have been some improvements in the 3PAR reclaim process in later 3.2.2 MU6 patches. I think P144 was the last one with improvements now.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: Adaptive Optimisation lower tier past 95%
PostPosted: Thu Jun 04, 2020 8:37 am 

Joined: Thu Jun 04, 2020 3:39 am
Posts: 4
Hi, yes the setting is there but its currently unset, do you know if for example Tier 2 currently has 42TB of space used and I set the AO Max Space on Tier 2 to 40TB will it then move 2 TB of space upto Tier 1 in a cycle? I'm just being really careful about using these settings as we don't have the option of going up to the next patch and want to ensure we don't have a major failure. Thanks again.


Top
 Profile  
Reply with quote  
 Post subject: Re: Adaptive Optimisation lower tier past 95%
PostPosted: Thu Jun 04, 2020 9:18 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
dabravokid wrote:
Hi, yes the setting is there but its currently unset, do you know if for example Tier 2 currently has 42TB of space used and I set the AO Max Space on Tier 2 to 40TB will it then move 2 TB of space upto Tier 1 in a cycle? I'm just being really careful about using these settings as we don't have the option of going up to the next patch and want to ensure we don't have a major failure. Thanks again.


You almost got it.
If you have one AO config which is currently using 42TB and you set t2 max to 40TB it will not move 2TB to Tier1, but it will (try to) ensure that Tier2 doesn't use more than 40TB. It's been a while since I worked with AO but there is a limitation as to how much it will move per run. Something tells me max 5% of total size of AO config per tier per run with 1TB minimum, but that is just something I think I remember.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 


Who is online

Users browsing this forum: Google [Bot] and 51 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt