HPE Storage Users Group
https://3parug.com/

High system CPU in 3PAR 7400c
https://3parug.com/viewtopic.php?f=18&t=3416
Page 1 of 1

Author:  Roger [ Wed Jun 03, 2020 1:32 pm ]
Post subject:  High system CPU in 3PAR 7400c

Hi,
I have an array 3PAR 7400c MU6

What could be the reasons for high system CPU consumption at certain times of the day ?. I noticed this behaviour but I didn't found documentation about the high system cpu time compared to user cpu time.
For example, at some point in the morning:

Time Secs User% Sys% Idle% Intr/s CtxtSw/s
2020-06-03 10:25:00 PET 1591197900 1.5 59.9 38.6 140033.5 202506.4
2020-06-03 10:30:00 PET 1591198200 1.2 59.1 39.7 144062.5 201053.9
2020-06-03 10:35:00 PET 1591198500 5.7 58.1 36.2 157178.1 199221.6
2020-06-03 10:40:00 PET 1591198800 5.5 55.3 39.2 155715.4 205932.9
2020-06-03 10:45:00 PET 1591199100 7.8 55.5 36.6 160773.3 212354.4
2020-06-03 10:50:00 PET 1591199400 6.5 50.9 42.6 167839.0 212000.9

I have also noticed that when the CPU load reaches approximately 60% it starts to have latency in the hosts.

Why do they occur high system CPU ? Which processes are system CPU intensive ?

Regards

Author:  MammaGutt [ Wed Jun 03, 2020 1:41 pm ]
Post subject:  Re: High system CPU in 3PAR 7400c

A little more info could be nice.

Also output from statcpu when it occurs.

My personal experience is that you see CPU load with compression (that you can't do), dedupe background tasks and generally high load (basically interupts).

If you're not doing dedupe and don't have anything that increases the load at those hours (that could be AO, async replication and host load like database indexing, batch processing, backups to mention a few) I would log a support case to have them review which processes are increasing the load on your system as that could also be anything from hardware issues, to software issues to known bugs.

Author:  Roger [ Thu Jun 04, 2020 12:00 pm ]
Post subject:  Re: High system CPU in 3PAR 7400c

Hi,
Thanks for your reply.
I've been reviewing and I noticed with the command srstatvlun shows the raise of the IO/s and KB/s aprox. in 4:00 and in 7:10 am today (see attached file). A high qlen is observed in the file, too.

On the other hand, today I added space to a VV with configured remote copy (syncronous FC), and during the sincronization of the additional space, the following output was observed in the statcpu:
10:43:39 06/04/2020
node,cpu user sys idle intr/s ctxt/s
0,0 0 55 45
0,1 1 89 10
0,2 0 40 60
0,3 0 59 41
0,4 0 48 52
0,5 0 66 34
0,6 0 42 58
0,7 0 85 15
0,8 0 30 70
0,9 0 31 69
0,10 0 29 71
0,11 0 67 33
0,total 0 53 47 83660 111046

1,0 1 49 49
1,1 0 89 11
1,2 0 40 60
1,3 0 55 45
1,4 0 44 56
1,5 0 64 36
1,6 0 37 63
1,7 0 83 17
1,8 0 31 69
1,9 0 31 69
1,10 100 0 0
1,11 0 70 30
1,total 8 49 42 90614 102367

My asumption is that the interrupts and context switching increases during intensive I/O and this affects to system CPU. But the array is far from its maximum theorical capacity and I don't think the CPU should be affected.

Regards.

Attachments:
File comment: srstatvlun
srstatvlun.txt [19.09 KiB]
Downloaded 1248 times

Author:  MammaGutt [ Fri Jun 05, 2020 6:45 am ]
Post subject:  Re: High system CPU in 3PAR 7400c

What I can see from that output is that when you get an increase in QLen you have a significant increase in write IOs, and write IOs that are significantly larger that what you're used to (as the average increase more than 100%).

None of the CPU cores seems to be alarmingly high, so I don't think CPU usage is related to the increase in latency.

My suggestion as you know when this is happening is to trigger a performance analysis on the Service Processor and review that to understand what hosts and volumes are generating the increased load. You're not telling a lot about the environment but if you have NL drives and using AO you could have a backup task running that is always hitting NL drives because it is running outside of the time period it is analyzing.

If you have sync remote copy, writes are replicated and performance issues on the target array will impact latency on the source array. So if you have that, you should do a performance analysis on that array as well.

Author:  Roger [ Thu Jun 11, 2020 2:56 am ]
Post subject:  Re: High system CPU in 3PAR 7400c

Hi,
I appreciate your collaboration. I thought the high qlen was a consecuence, not the origin of problem.
So, the order of the triggers would be:
High writes => High Qlen => slight increase in service time
is this the correct order of triggers ?

Unfortunately I have not been able to find the process culprit of this sudden increase in writes ratio.

Regards.

Author:  MammaGutt [ Thu Jun 11, 2020 12:24 pm ]
Post subject:  Re: High system CPU in 3PAR 7400c

Roger wrote:
Hi,
I appreciate your collaboration. I thought the high qlen was a consecuence, not the origin of problem.
So, the order of the triggers would be:
High writes => High Qlen => slight increase in service time
is this the correct order of triggers ?

Unfortunately I have not been able to find the process culprit of this sudden increase in writes ratio.

Regards.


That is my guesstimate yes. Increased writes with larger IO size is the reason.

If you know exactly when this occurs, start two SSH sessions.

One does statvlun -vvsum -ni -rw
And one does statvlun -hostsum -ni -rw

That should show host and volume that should help you to narrow it down, even in a virtualized environment.

Author:  Roger [ Tue Jun 23, 2020 4:31 pm ]
Post subject:  Re: High system CPU in 3PAR 7400c

Thank you very much.

Page 1 of 1 All times are UTC - 5 hours
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/