HPE Storage Users Group
https://3parug.com/

High IOPS, high bandwith poor performance
https://3parug.com/viewtopic.php?f=18&t=2707
Page 1 of 2

Author:  korny [ Tue Nov 07, 2017 9:36 am ]
Post subject:  High IOPS, high bandwith poor performance

Hello all
I'm currently trying to understand a performance problem I encountered this morning on my 7200C

the 7200C has 42FC disks , and should be able to run around 5500IOPS
This morning, service time was not good at all, and I had 10.000iops , max 12.000

This increase of IOPS was caused by a high total bandwith : around 200MBPS

Image

Then, I'm trying to find which VV was responsible for a high bandwith and I tought the report "Exported Volumes Compare by Performance" could help :

Image[/url]

unfortunately , the sum of the bandwith of each exported volume is around 100MB, when I expect 200MB
Am I doing something wrong ?

My goal is to find which host, or which exported volume can be the responsible of the high total BW
with CLI, statvlun -hostsum could help me to find the BW for each host. But now the problem is over, and I suppose we cannot check in the past with CLI

Author:  RitonLaBevue [ Tue Nov 07, 2017 10:42 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

You could try "help sr" and see the whole bunch of System Reporter command available to you through SSMC and obviously through SSH/CLI.

srstatport -port_type host -btsecs -3h -etsecs -2h
srstatvlun -groupby VV_NAME -btsecs -3h -etsecs -2h

These commands could give you some details on VV performance between 3 and 2 hours ago.

Author:  korny [ Tue Nov 07, 2017 11:26 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

nice !
I'm going to use these srxxx commands to find out the responsible :)

Author:  RitonLaBevue [ Wed Nov 08, 2017 1:36 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

Enjoy the hunt :D

Author:  apol [ Wed Nov 08, 2017 4:42 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

To your first question "unfortunately , the sum of the bandwith of each exported volume is around 100MB, when I expect 200MB. Am I doing something wrong ?"

Do you have more than 10 VVs? I think the "compare by"-report templates per default just comapre the top 10 vvs and ignore the other ones. This could explain the missing bandwith if you did not change the report settings.

Author:  korny [ Wed Nov 08, 2017 5:17 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

Result of the hunt :
with your commands I successed to find out wich host / vv was consuming bandwith, but it resulst that the activity is the same than the days before .. I think my performance problem comes from my 3Par which is overloaded

1 CPG / 42 FC . Theory 5400iops
Nowadays the 3par is running around 5400iops (yesterday, the total was around 10.000iops during some minutes, and all the disks were 0% idle)

I think I just need more performance
Any advice for this point ?

my 3par needs to assume more iops , but does not need more capacity ..
SSD could be a good point to get more IOPS. Do you think I can mix SSD and FC disks in a same CPG ?

I also tried the adaptive flash cache simulator . and hits on the AFC is only around 7% : if I understand well, this solution will not really improve the situation

Author:  korny [ Wed Nov 08, 2017 5:27 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

apol wrote:
To your first question "unfortunately , the sum of the bandwith of each exported volume is around 100MB, when I expect 200MB. Am I doing something wrong ?"

Do you have more than 10 VVs? I think the "compare by"-report templates per default just comapre the top 10 vvs and ignore the other ones. This could explain the missing bandwith if you did not change the report settings.


I just realize that is the point, you're right ! I have around 30VV
in the view "Exported Volumes Compare by Performance.." only 10vv are displayed

But, in the view "exported volume performance" the bandwith diplays the total of all the VV.
and if I compare this total to the "pd performance bandwitch view", the bandwiths are not the same
ie, : at the same time

BW total exported VV : 150MBPS
BW PD : 236 MBPS

So I have still the same question :)
I read that this difference could come from the RAID configuration ..

Author:  apol [ Wed Nov 08, 2017 5:32 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

"I read that this difference could come from the RAID configuration .."

Yes, I think so as well. And it could be that cache prefetching, work done by vaai-commands and so on are involved as well. On the vv-level, that's the reason why there are statvv (contains array-internal stuff) and statvlun (contains stuff from hosts).

Author:  ailean [ Wed Nov 08, 2017 6:48 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

I suspect your extra load is write based, this always hits the disks a lot harder, you may have seen similar levels before but were more read based so easily absorbed by cache and lower disk load.

I think AFC is only read based so why you have a low hit rate on the simulation. Try to nail down the write intensive VVs and if they are re-writing the same blocks a lot or writing lots of new blocks.

If same blocks then adding SSD and Adaptive Optimization might help, if lots of new blocks then you'll probably have to move the VVs to SSD (needing more space).

Hopefully it'll be a small number of aggressive VVs and once they are dealt with things will be happy. Of course you could always poke the server/application owners are ask what they are doing, it might turn out (as often does here) that they didn't know it was doing something and magically the IO falls when they kill some rogue process. :roll:

Author:  cali [ Mon Nov 13, 2017 7:50 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

Look at: statpd

On right you see if Disks are Overloaded.
Idle > 20% and Qlen < 5 is OK

If not add Disks or SSD and enable Tiering.

FC Disks deliver about 150 IOPS.

Page 1 of 2 All times are UTC - 5 hours
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/