HPE Storage Users Group
https://3parug.com/

Question regarding disk I/O and performance impact..
https://3parug.com/viewtopic.php?f=18&t=2978		 Page 1 of 1
Author:  walter_white [ Mon Sep 10, 2018 7:22 am ]
Post subject:  Question regarding disk I/O and performance impact..
So I have a 7200 with 28 10K FC disks.. The low IOPS threshold is 150 while the high is 250.. NinjaStar has it getting 3,000 I/O. All VV are RAID5..

We've had performance issues with this san in the past so I'm often times SSH'd into it just monitoring it while doing other things..

The other day I happen to clance over while "statvlun -hostsum -ni -sortcol 3,dec" is running and I see one VV over 4,400 read I/O per second..

Attachment:
1.jpg
1.jpg [ 339.31 KiB | Viewed 12925 times ]


I find someone kicked off a backup during the day and the VM was on this VV doing all the I/O.. The question I have is.. How did this pull that much I/O over what the tool says the SAN can even handle? I monitored it for a good 45 minutes and it was consistently that high with peaks up at 7,000..

When we've had issues in the past and the I/O shot up like this, the entire SAN and all VMs came to a crawl until we stopped what was going on.. This time, the performance impact wasn't even that bad..

Can someone explain this to me please? Thanks much!!

Attachment:
2.jpg
2.jpg [ 114.6 KiB | Viewed 12925 times ]


Attachment:
3.jpg
3.jpg [ 138.86 KiB | Viewed 12925 times ]


Along these lines.. I setup a threshold alert to alert me when this SAN goes over 4,000 IO and I never got an alert? Is my threshold not configured right or what?

Attachment:
4.jpg
4.jpg [ 61.85 KiB | Viewed 12925 times ]
Author:  MammaGutt [ Mon Sep 10, 2018 10:38 am ]
Post subject:  Re: Question regarding disk I/O and performance impact..
Sizing is usually done on random IO, backup is usually sequential. That is probably why it was able to do 7k iops.
Author:  walter_white [ Tue Sep 11, 2018 9:37 am ]
Post subject:  Re: Question regarding disk I/O and performance impact..
MammaGutt wrote:
Sizing is usually done on random IO, backup is usually sequential. That is probably why it was able to do 7k iops.


Thanks for the reply.. Can you explain what that means?
Author:  MammaGutt [ Tue Sep 11, 2018 11:27 am ]
Post subject:  Re: Question regarding disk I/O and performance impact..
Sequential vs random IO?

If you put all blocks on the disk on a looooooooong row, sequential will read 1-2-3-4-5-6-7-8-9-etc while random is 3-8-15-7-99-54-73-48.

With random the harddisk is "always" waiting for the data you are going to read/write is available under/over the drive head. This is called "seek time".
Author:  Proc_rqrd [ Tue Sep 11, 2018 2:12 pm ]
Post subject:  Re: Question regarding disk I/O and performance impact..
to add..
ninjastars showing a sequential with your array specs (roughly) is about 6,200 IOPS with a Qlen(out standing IO's) of 34 or so.
for an estimate that is not far off on what you saw (your qlen has one entry of 33).

as far as experience...well...this one VV was slammed with a high queue length and the others had no Qlen. so depending on what volumes are doing what workload..and how that workload effects user "experience" it is entirely possible that it went unnoticed or less noticed....

your overall service time and queue length shows a busy, but not drowning array.

you seem to have a good handle on things, if you have done your best effort to optimize the array for your use case (workload). if performance degrades...you can see what it is, and use your best Scottish accent to tell them "im giving it all shes got captain!" :D
Page 1 of 1 All times are UTC - 5 hours
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/