3PAR Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 18 posts ]  Go to page Previous  1, 2
Author Message
 Post subject: Re: Adaptive Optimization (AO) - The silver bullet?
PostPosted: Mon Feb 25, 2019 4:33 pm 

Joined: Wed Nov 08, 2017 8:57 am
Posts: 42
MammaGutt wrote:
Sorry for the late response.

Just making this "quick" as I don't have time to calc everything down from frontend IOPS to backend IOPS to realistic backend IOPS on your disks. And comparing the different volumes to eachother.

1.9TB of SSD should take the majority of the hot data (assuming it is a somewhat fit for AO). Simple glance says it will do something like 75% of the IOPS but I didn't take a very deep look now, sorry.

Without doing the exact numbers on that, you should be coming a long way on that with 8x 400/480GB if those are still available for sale. Don't go 4x 920GB... if you get acceptance for 920GB you should do 8 (to keep the same number of SSDs as the 100GB and to minimize the capacity allocated for sparing. You might need to set sparing algoritm to minimal (only lose one SSD to sparing) and use the maximum available set size (which would be R5 7+1). Keep in mind that you should only use maximum set size for AO CPGs, so you would have to do AO for the "NON-AO-SSD-R1" CPG as well. There is a lot of wasted space there today.

Btw, your systems most active volumes/area in terms of write is ????-SQL-PRD-TEMP ... It's small but it's hot.


As for troubleshooting part. I always look at VLUN service time. Most cases when you see performance problems, people notice increase in service time. Your system is really having issues as .srdata is actually mongst the heavy hitters which from my experience tend to happen when it is never able to catch up.

It isn't always the VV/VLUN with the high service time which is at fault but it tells you that something is wrong. QLEN should be low, that this is just as much a host/server configuration as it could be a storage issue.

And yes, my OCD tells me that one physical server should have the same reference on any equipment in your environment... Switchport tagging, SAN zoning, storage array, CMDB ++++


First off. Please, dont be sorry about anything! I appreciate the replies greatly!!

I'm trying to understand these srrgiodensity reports.. I made one and put it in Excel to make it easier to sort..

So these values are number of access (I/O) per minute for the past day, right?

So the SWL-PRD-DATA is only 70GB (if I'm reading this right) and it's averaging 611 I/O R/W per minute? SQL-PRD-TEMP is only 8GB and it's averaging 368 I/O R/W per minute? So these two volumes alone are taking up 78GB in space and almost 1,000 I/O per minute?

I'll admit I'm super green in storage but that seems like a pretty glaring issue.. If I simply moved these two specific volumes to SSD shouldn't that eliminate a lot of the issues I'm seeing?

I know storage is complex AF but if this was your issue, where do you start?

You replied to my other thread regarding the I/O on VV vs PD and I think you can see my FC PD's are being pushed to their max all the time.. I have constant QLEN on the PD Backend command.. I often have it in the front end VV command as well..

Sigh.. Appreciate your help mate..


Attachments:
srrgiodensity-3par.xlsx [15.13 KiB]
Downloaded 44 times
Top
 Profile  
Reply with quote  
 Post subject: Re: Adaptive Optimization (AO) - The silver bullet?
PostPosted: Mon Feb 25, 2019 4:45 pm 

Joined: Wed Nov 08, 2017 8:57 am
Posts: 42
MammaGutt wrote:
Sorry for the late response.

Without doing the exact numbers on that, you should be coming a long way on that with 8x 400/480GB if those are still available for sale. Don't go 4x 920GB... if you get acceptance for 920GB you should do 8 (to keep the same number of SSDs as the 100GB and to minimize the capacity allocated for sparing. You might need to set sparing algoritm to minimal (only lose one SSD to sparing) and use the maximum available set size (which would be R5 7+1).


Also... (4) 920GBs were ordered.. Some political stuff but X went to Y and Y said buy it now.. So, we were going to just do a R5 CPG with them..

MammaGutt wrote:
And yes, my OCD tells me that one physical server should have the same reference on any equipment in your environment... Switchport tagging, SAN zoning, storage array, CMDB ++++


So is it safe to simply rename the hosts in SSMC? Just renaming the display name and adding some details that should have been there (IP, description, etc)?

Thanks again.


Last edited by walter_white on Wed Feb 27, 2019 8:09 am, edited 1 time in total.

Top
 Profile  
Reply with quote  
 Post subject: Re: Adaptive Optimization (AO) - The silver bullet?
PostPosted: Tue Feb 26, 2019 1:08 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 842
Location: Europe
walter_white wrote:
MammaGutt wrote:
Sorry for the late response.

Without doing the exact numbers on that, you should be coming a long way on that with 8x 400/480GB if those are still available for sale. Don't go 4x 920GB... if you get acceptance for 920GB you should do 8 (to keep the same number of SSDs as the 100GB and to minimize the capacity allocated for sparing. You might need to set sparing algoritm to minimal (only lose one SSD to sparing) and use the maximum available set size (which would be R5 7+1).


Also... (4) 920GBs were ordered.. Sigh.. Political bullshit but X went to Y and Y said buy it now.. So, we were going to just do a R5 CPG with them..

MammaGutt wrote:
And yes, my OCD tells me that one physical server should have the same reference on any equipment in your environment... Switchport tagging, SAN zoning, storage array, CMDB ++++


So is it safe to simply rename the hosts in SSMC? Just renaming the display name and adding some details that should have been there (IP, description, etc)?

Thanks again.


So... You buy 4x920GB. With default sparing you lose capacity of 2 drives per 24(it will be a tiny bit less due to 100GB SSDs already there) . If you do R5 3+1 you get 920GB * 1.5 available, with R1 you get 920GB. Probably not enough to solve the issue.

Rename is no problem, just do it in SSMC.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: Adaptive Optimization (AO) - The silver bullet?
PostPosted: Tue Feb 26, 2019 8:09 am 

Joined: Wed Nov 08, 2017 8:57 am
Posts: 42
MammaGutt wrote:

So... You buy 4x920GB. With default sparing you lose capacity of 2 drives per 24(it will be a tiny bit less due to 100GB SSDs already there) . If you do R5 3+1 you get 920GB * 1.5 available, with R1 you get 920GB. Probably not enough to solve the issue.


Did you happen to look at the spreadsheet I attached? If I just moved those two VVs with the majority of the disk I/O to the SSD, and not even worry about AO, would that take a lot of the I/O and have them on SSD?

Wouldn't just doing that make a considerable difference?

Regarding the disk.. Yes, super bummed.. NinjaStar doesn't even allow putting the 4 drives themselves in a R5 so that leaves R1 or mixing them with the existing 92GB drives which I wouldn't think would be a good idea..

EDIT: Looks like NinjaStar doens't allow you to be an idiot and that's why there is no R5 option because it does cage by default.. Whoever installed ours.. Sigh.. Just look..

Attachment:
Cages.jpg
Cages.jpg [ 230.38 KiB | Viewed 952 times ]


Top
 Profile  
Reply with quote  
 Post subject: Re: Adaptive Optimization (AO) - The silver bullet?
PostPosted: Tue Feb 26, 2019 5:21 pm 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 842
Location: Europe
So, based on your attached spreadsheet, the volumes with the hottest blocks aren't big... and again the total IOPS shouldn't be that much.

Of course it will make a difference but if you are hitting the high watermark for FC-10k drives today it will not be enough. Not sure which report it is from but the volumes wit highest avg-tot are small volumes, and I'm guessing the numbers are acc/(GiB*min) so it is active blocks but in total it just doesn't add up. Not the best example but hopefully it shows my point.

VDIDatastore3 is ~550GB and average ~82 acc/(GiB*min). Adding this up becomes 45100 acc/min.
SQL-PRD-TEMP is ~10GB and average ~370 acc/(GiB*min). Adding this up becomes 3700 acc/min.

Even if SQL-PRD-TEMP have hotter blocks (5x hotter), it produces less than 10% of the IOPS than the big VDIDatastore3 volume.

Taking this also a bit further... Minimum disk per tier is 6 for SSD. BP states that the smallest disk shouldn't be less than 50% or the biggest disk (I tend to ignore this as long as I have at least one full stripe for each size and in a best case scenario have equal number of small and large SSDs). So if you follow BP and create a new CPG, you are 2 SSDs short of being supported with 4x 920GB. Also, you will be allowed to configure R5 3+1 but if you intend to use this for anything other than AO, you are at risk of hitting this: https://support.hpe.com/hpsc/doc/public ... cale=en_US

So then you're down to R5 2+1 or R1... Either way you're not going to get a lot of capacity out of those 4x 920GB SSDs... I'm sorry, but if you want a good advise you need to go back to whomever ordered those 4x 920GB and say that you need 4 more for this to actually provide any real change because with only 4 drives you will have a difficult time even creating a supported configuration and you will lose such a big percentage of the actual capacity added it has a terrible "bang for the buck". 8x 400/480GB would have been a HUGE difference at not a big price difference.....


As for those that installed your 3PAR... Did you check "AdmissionTime" on showpd -i ? I'm just taking a stab here but it could be that the system has been expanded over time and only had one cage when the SSDs was installed.....

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: Adaptive Optimization (AO) - The silver bullet?
PostPosted: Wed Feb 27, 2019 8:55 am 

Joined: Wed Nov 08, 2017 8:57 am
Posts: 42
MammaGutt wrote:
So, based on your attached spreadsheet, the volumes with the hottest blocks aren't big... and again the total IOPS shouldn't be that much.

Of course it will make a difference but if you are hitting the high watermark for FC-10k drives today it will not be enough. Not sure which report it is from but the volumes wit highest avg-tot are small volumes, and I'm guessing the numbers are acc/(GiB*min) so it is active blocks but in total it just doesn't add up. Not the best example but hopefully it shows my point.

VDIDatastore3 is ~550GB and average ~82 acc/(GiB*min). Adding this up becomes 45100 acc/min.
SQL-PRD-TEMP is ~10GB and average ~370 acc/(GiB*min). Adding this up becomes 3700 acc/min.

Even if SQL-PRD-TEMP have hotter blocks (5x hotter), it produces less than 10% of the IOPS than the big VDIDatastore3 volume.

Taking this also a bit further... Minimum disk per tier is 6 for SSD. BP states that the smallest disk shouldn't be less than 50% or the biggest disk (I tend to ignore this as long as I have at least one full stripe for each size and in a best case scenario have equal number of small and large SSDs). So if you follow BP and create a new CPG, you are 2 SSDs short of being supported with 4x 920GB. Also, you will be allowed to configure R5 3+1 but if you intend to use this for anything other than AO, you are at risk of hitting this: https://support.hpe.com/hpsc/doc/public ... cale=en_US

So then you're down to R5 2+1 or R1... Either way you're not going to get a lot of capacity out of those 4x 920GB SSDs... I'm sorry, but if you want a good advise you need to go back to whomever ordered those 4x 920GB and say that you need 4 more for this to actually provide any real change because with only 4 drives you will have a difficult time even creating a supported configuration and you will lose such a big percentage of the actual capacity added it has a terrible "bang for the buck". 8x 400/480GB would have been a HUGE difference at not a big price difference.....


As for those that installed your 3PAR... Did you check "AdmissionTime" on showpd -i ? I'm just taking a stab here but it could be that the system has been expanded over time and only had one cage when the SSDs was installed.....


Starting to make more sense to me now.. Thanks so much for breaking this all down..

So what is the best way to try and figure out if it's a problem VM, a problem VV, etc? From what you're saying I should be multiplying the VV space used by the Average Total Acc/Min which will give me the total ACC/Min for each VV..

I'm assuming ACC/Min is equivalent to I/O per seconds * 60?

I've attached another spreadsheet.. It has 3 tabs where I ran srrgiodensity -btsecs -1d -cpg * -rw -withvv -pct each morning.. I added two columns at the end (SpaceGB and Total Acc/Min)..

So am I looking at this right in that the total Acc/Min (space used * average total I/O acc-min) is the number to be concerned about?

One other question.. NinjaStars shows that the 28 SAS drives in R5 can do around 3,000 IOPS.. So is it as simple as saying the 3PAR can do 3,000 * 60 = 180,000 ACC/MIN? And if I take my spreadsheet and add up the entire last column which is the Acc/Min for each VV, I would get the total amount of I/O vs what the SAN is actually capable of?

I guess that can't be right though.. Because that number is 315,000 ACC/MIN..

Appreciate your time mate.. Learning a ton.

EDIT TO ADD: Update.. The order hasn't been placed yet.. I just got a quote for (8) x 480GB SSDs for $13.7K compared to the $11.8K..

Attachment:
srrgiodensity-3par.xlsx [32.46 KiB]
Downloaded 32 times


Top
 Profile  
Reply with quote  
 Post subject: Re: Adaptive Optimization (AO) - The silver bullet?
PostPosted: Wed Feb 27, 2019 4:58 pm 

Joined: Wed Nov 08, 2017 8:57 am
Posts: 42
Side question.. Should I run a tunesys? That would balance the chunklets out across the PD's, right? Not necessarily asking if it would help my issue but should be done, agreed?

Attachment:
2019-02-27_16-56-43.jpg
2019-02-27_16-56-43.jpg [ 137.1 KiB | Viewed 797 times ]


Top
 Profile  
Reply with quote  
 Post subject: Re: Adaptive Optimization (AO) - The silver bullet?
PostPosted: Thu Feb 28, 2019 5:47 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 842
Location: Europe
walter_white wrote:
Starting to make more sense to me now.. Thanks so much for breaking this all down..

So what is the best way to try and figure out if it's a problem VM, a problem VV, etc? From what you're saying I should be multiplying the VV space used by the Average Total Acc/Min which will give me the total ACC/Min for each VV..

I'm assuming ACC/Min is equivalent to I/O per seconds * 60?

I've attached another spreadsheet.. It has 3 tabs where I ran srrgiodensity -btsecs -1d -cpg * -rw -withvv -pct each morning.. I added two columns at the end (SpaceGB and Total Acc/Min)..

So am I looking at this right in that the total Acc/Min (space used * average total I/O acc-min) is the number to be concerned about?

One other question.. NinjaStars shows that the 28 SAS drives in R5 can do around 3,000 IOPS.. So is it as simple as saying the 3PAR can do 3,000 * 60 = 180,000 ACC/MIN? And if I take my spreadsheet and add up the entire last column which is the Acc/Min for each VV, I would get the total amount of I/O vs what the SAN is actually capable of?

I guess that can't be right though.. Because that number is 315,000 ACC/MIN..



Many things to cover here. I'm not 100% sure if acc= iops, but it is some form of measurement of activity. Also remember that there are traffic to volumes (or regions of LDs which is what AO is looking at) which isn't host traffic (disk-scrubber, zero reclamation +++, however not a lot).

What you need to be concerned about is host latency. As long as host latency is good the end user is good.

There is just to many parameters to look at everything (which is why you want the array to do that for you). You see some volumes having a lot of read, but SR will not tell you if those read IOs are hitting cache or hitting disk. If it hits cache then everything is good (today) if it hits disk then you're getting backend IO. Many wise people are saying that the best IO is the one you don't have to do :) The problem with cache hit is that you don't know if the workload is changing and you run out of cache in 30 minutes and starts hitting your disks. Cache is a fantastic super fast thing, but impossible to predict.

Don't look blindly at NinjaSTAR. This is no single IOPS number for 28x 10k SAS drives. It depends on IO size, read/write ratio, RAID level and cache hit. With spinning media, latency will usually increase with load and go thru the roof when maxed.


walter_white wrote:
Side question.. Should I run a tunesys? That would balance the chunklets out across the PD's, right? Not necessarily asking if it would help my issue but should be done, agreed?

Attachment:
2019-02-27_16-56-43.jpg


My guess it is leveling on backend SAS port/bus. If you combine the data on FC disks on cage0 I will assume it matches the data in cage1. If that is the case tunesys will only try to do exactly that.

In a perfect world with a balanced system you would have the same amount of disk and capacity in each cage and every PD would have the exact amount of data stored on them and hitting the same amount of IOPS.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 18 posts ]  Go to page Previous  1, 2


Who is online

Users browsing this forum: No registered users and 19 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt