HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 5 posts ] 
Author Message
 Post subject: Question regarding disk I/O and performance impact..
PostPosted: Mon Sep 10, 2018 7:22 am 

Joined: Wed Nov 08, 2017 8:57 am
Posts: 42
So I have a 7200 with 28 10K FC disks.. The low IOPS threshold is 150 while the high is 250.. NinjaStar has it getting 3,000 I/O. All VV are RAID5..

We've had performance issues with this san in the past so I'm often times SSH'd into it just monitoring it while doing other things..

The other day I happen to clance over while "statvlun -hostsum -ni -sortcol 3,dec" is running and I see one VV over 4,400 read I/O per second..

Attachment:
1.jpg
1.jpg [ 339.31 KiB | Viewed 12775 times ]


I find someone kicked off a backup during the day and the VM was on this VV doing all the I/O.. The question I have is.. How did this pull that much I/O over what the tool says the SAN can even handle? I monitored it for a good 45 minutes and it was consistently that high with peaks up at 7,000..

When we've had issues in the past and the I/O shot up like this, the entire SAN and all VMs came to a crawl until we stopped what was going on.. This time, the performance impact wasn't even that bad..

Can someone explain this to me please? Thanks much!!

Attachment:
2.jpg
2.jpg [ 114.6 KiB | Viewed 12775 times ]


Attachment:
3.jpg
3.jpg [ 138.86 KiB | Viewed 12775 times ]


Along these lines.. I setup a threshold alert to alert me when this SAN goes over 4,000 IO and I never got an alert? Is my threshold not configured right or what?

Attachment:
4.jpg
4.jpg [ 61.85 KiB | Viewed 12775 times ]


Top
 Profile  
Reply with quote  
 Post subject: Re: Question regarding disk I/O and performance impact..
PostPosted: Mon Sep 10, 2018 10:38 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
Sizing is usually done on random IO, backup is usually sequential. That is probably why it was able to do 7k iops.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: Question regarding disk I/O and performance impact..
PostPosted: Tue Sep 11, 2018 9:37 am 

Joined: Wed Nov 08, 2017 8:57 am
Posts: 42
MammaGutt wrote:
Sizing is usually done on random IO, backup is usually sequential. That is probably why it was able to do 7k iops.


Thanks for the reply.. Can you explain what that means?


Top
 Profile  
Reply with quote  
 Post subject: Re: Question regarding disk I/O and performance impact..
PostPosted: Tue Sep 11, 2018 11:27 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
Sequential vs random IO?

If you put all blocks on the disk on a looooooooong row, sequential will read 1-2-3-4-5-6-7-8-9-etc while random is 3-8-15-7-99-54-73-48.

With random the harddisk is "always" waiting for the data you are going to read/write is available under/over the drive head. This is called "seek time".

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: Question regarding disk I/O and performance impact..
PostPosted: Tue Sep 11, 2018 2:12 pm 

Joined: Thu Feb 04, 2016 4:12 pm
Posts: 28
to add..
ninjastars showing a sequential with your array specs (roughly) is about 6,200 IOPS with a Qlen(out standing IO's) of 34 or so.
for an estimate that is not far off on what you saw (your qlen has one entry of 33).

as far as experience...well...this one VV was slammed with a high queue length and the others had no Qlen. so depending on what volumes are doing what workload..and how that workload effects user "experience" it is entirely possible that it went unnoticed or less noticed....

your overall service time and queue length shows a busy, but not drowning array.

you seem to have a good handle on things, if you have done your best effort to optimize the array for your use case (workload). if performance degrades...you can see what it is, and use your best Scottish accent to tell them "im giving it all shes got captain!" :D


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 5 posts ] 


Who is online

Users browsing this forum: No registered users and 81 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt