HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 7 posts ] 
Author Message
 Post subject: High system CPU in 3PAR 7400c
PostPosted: Wed Jun 03, 2020 1:32 pm 

Joined: Wed Aug 24, 2016 5:26 pm
Posts: 5
Hi,
I have an array 3PAR 7400c MU6

What could be the reasons for high system CPU consumption at certain times of the day ?. I noticed this behaviour but I didn't found documentation about the high system cpu time compared to user cpu time.
For example, at some point in the morning:

Time Secs User% Sys% Idle% Intr/s CtxtSw/s
2020-06-03 10:25:00 PET 1591197900 1.5 59.9 38.6 140033.5 202506.4
2020-06-03 10:30:00 PET 1591198200 1.2 59.1 39.7 144062.5 201053.9
2020-06-03 10:35:00 PET 1591198500 5.7 58.1 36.2 157178.1 199221.6
2020-06-03 10:40:00 PET 1591198800 5.5 55.3 39.2 155715.4 205932.9
2020-06-03 10:45:00 PET 1591199100 7.8 55.5 36.6 160773.3 212354.4
2020-06-03 10:50:00 PET 1591199400 6.5 50.9 42.6 167839.0 212000.9

I have also noticed that when the CPU load reaches approximately 60% it starts to have latency in the hosts.

Why do they occur high system CPU ? Which processes are system CPU intensive ?

Regards


Top
 Profile  
Reply with quote  
 Post subject: Re: High system CPU in 3PAR 7400c
PostPosted: Wed Jun 03, 2020 1:41 pm 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1104
Location: Europe
A little more info could be nice.

Also output from statcpu when it occurs.

My personal experience is that you see CPU load with compression (that you can't do), dedupe background tasks and generally high load (basically interupts).

If you're not doing dedupe and don't have anything that increases the load at those hours (that could be AO, async replication and host load like database indexing, batch processing, backups to mention a few) I would log a support case to have them review which processes are increasing the load on your system as that could also be anything from hardware issues, to software issues to known bugs.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: High system CPU in 3PAR 7400c
PostPosted: Thu Jun 04, 2020 12:00 pm 

Joined: Wed Aug 24, 2016 5:26 pm
Posts: 5
Hi,
Thanks for your reply.
I've been reviewing and I noticed with the command srstatvlun shows the raise of the IO/s and KB/s aprox. in 4:00 and in 7:10 am today (see attached file). A high qlen is observed in the file, too.

On the other hand, today I added space to a VV with configured remote copy (syncronous FC), and during the sincronization of the additional space, the following output was observed in the statcpu:
10:43:39 06/04/2020
node,cpu user sys idle intr/s ctxt/s
0,0 0 55 45
0,1 1 89 10
0,2 0 40 60
0,3 0 59 41
0,4 0 48 52
0,5 0 66 34
0,6 0 42 58
0,7 0 85 15
0,8 0 30 70
0,9 0 31 69
0,10 0 29 71
0,11 0 67 33
0,total 0 53 47 83660 111046

1,0 1 49 49
1,1 0 89 11
1,2 0 40 60
1,3 0 55 45
1,4 0 44 56
1,5 0 64 36
1,6 0 37 63
1,7 0 83 17
1,8 0 31 69
1,9 0 31 69
1,10 100 0 0
1,11 0 70 30
1,total 8 49 42 90614 102367

My asumption is that the interrupts and context switching increases during intensive I/O and this affects to system CPU. But the array is far from its maximum theorical capacity and I don't think the CPU should be affected.

Regards.


Attachments:
File comment: srstatvlun
srstatvlun.txt [19.09 KiB]
Downloaded 39 times
Top
 Profile  
Reply with quote  
 Post subject: Re: High system CPU in 3PAR 7400c
PostPosted: Fri Jun 05, 2020 6:45 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1104
Location: Europe
What I can see from that output is that when you get an increase in QLen you have a significant increase in write IOs, and write IOs that are significantly larger that what you're used to (as the average increase more than 100%).

None of the CPU cores seems to be alarmingly high, so I don't think CPU usage is related to the increase in latency.

My suggestion as you know when this is happening is to trigger a performance analysis on the Service Processor and review that to understand what hosts and volumes are generating the increased load. You're not telling a lot about the environment but if you have NL drives and using AO you could have a backup task running that is always hitting NL drives because it is running outside of the time period it is analyzing.

If you have sync remote copy, writes are replicated and performance issues on the target array will impact latency on the source array. So if you have that, you should do a performance analysis on that array as well.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: High system CPU in 3PAR 7400c
PostPosted: Thu Jun 11, 2020 2:56 am 

Joined: Wed Aug 24, 2016 5:26 pm
Posts: 5
Hi,
I appreciate your collaboration. I thought the high qlen was a consecuence, not the origin of problem.
So, the order of the triggers would be:
High writes => High Qlen => slight increase in service time
is this the correct order of triggers ?

Unfortunately I have not been able to find the process culprit of this sudden increase in writes ratio.

Regards.


Top
 Profile  
Reply with quote  
 Post subject: Re: High system CPU in 3PAR 7400c
PostPosted: Thu Jun 11, 2020 12:24 pm 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1104
Location: Europe
Roger wrote:
Hi,
I appreciate your collaboration. I thought the high qlen was a consecuence, not the origin of problem.
So, the order of the triggers would be:
High writes => High Qlen => slight increase in service time
is this the correct order of triggers ?

Unfortunately I have not been able to find the process culprit of this sudden increase in writes ratio.

Regards.


That is my guesstimate yes. Increased writes with larger IO size is the reason.

If you know exactly when this occurs, start two SSH sessions.

One does statvlun -vvsum -ni -rw
And one does statvlun -hostsum -ni -rw

That should show host and volume that should help you to narrow it down, even in a virtualized environment.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: High system CPU in 3PAR 7400c
PostPosted: Tue Jun 23, 2020 4:31 pm 

Joined: Wed Aug 24, 2016 5:26 pm
Posts: 5
Thank you very much.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 


Who is online

Users browsing this forum: No registered users and 40 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt