HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 8 posts ] 
Author Message
 Post subject: CPU Utilitization messages on only 1 node and cpu
PostPosted: Tue Sep 05, 2017 4:26 am 
User avatar

Joined: Thu Sep 01, 2011 8:27 am
Posts: 20
Hi,

We've setup threshold alerts via SSMC, one of them is for the CPU utilization.
We've set the following parameters (output of 'showsralertcrit'):
Code:
System_CPU_Utilization_Major    enabled hires  major    cpu  -recur=10/10,idle_pct<10


Now we get major alerts like the one below, but only on 1 of our 9 array's:
Code:
System Reporter threshold alert System_CPU_Utilization_Major with condition idle_pct<10,recur>=10/10 has been satisfied by cpu on node 0:1 with value(s) 0.0,10 recurrences


Two things are strange:
  • The error always shows on node 0 and cpu 1
  • Look like there's no averaging happening of CPU usage

Anyone know what process(es) are running on node 0, cpu 1?
Anyone know a better way to alert on CPU usage?

Regards,
Martien


Top
 Profile  
Reply with quote  
 Post subject: Re: CPU Utilitization messages on only 1 node and cpu
PostPosted: Wed Sep 06, 2017 6:28 am 
User avatar

Joined: Thu Sep 01, 2011 8:27 am
Posts: 20
Logged a case with HPE for this issue. Now wait to find out what the problem is :)


Top
 Profile  
Reply with quote  
 Post subject: Re: CPU Utilitization messages on only 1 node and cpu
PostPosted: Wed Sep 06, 2017 7:40 am 

Joined: Wed Nov 09, 2011 12:01 pm
Posts: 392
Maybe check shownode to see which is master and shownet to see which is the Active management port?

Any management tools/plugins talking to it other then SSMC?

Is it a 2 node doing lots of stuff?

Maybe graph CPU/Disk/Host IO for correlating spikes during the CPU busy?


Top
 Profile  
Reply with quote  
 Post subject: Re: CPU Utilitization messages on only 1 node and cpu
PostPosted: Wed Sep 06, 2017 7:55 pm 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1571
Location: Europe
2-node system running dedupe and seeing high cpu util on the non-master node?

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: CPU Utilitization messages on only 1 node and cpu
PostPosted: Thu Sep 07, 2017 10:33 am 
User avatar

Joined: Thu Sep 01, 2011 8:27 am
Posts: 20
Update...

Dedupe defragmentation process always runs on this node and core, which is a fairly CPU intensive task, but only for about 3 to 4 minutes.

My threshold is set for 10 occurrences in 10 hi-res samples with a cpu idle percentage of 10. So every 5-minutes the CPU idle is less than 10% for 10 times in a row.

That was unexpected for the HPE engineer, he's now checking performance logs etc.

Still waiting :D


Top
 Profile  
Reply with quote  
 Post subject: Re: CPU Utilitization messages on only 1 node and cpu
PostPosted: Fri Sep 08, 2017 3:35 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1571
Location: Europe
Biite wrote:
Update...

Dedupe defragmentation process always runs on this node and core, which is a fairly CPU intensive task, but only for about 3 to 4 minutes.


To my knowledge it runs on all non-master nodes and is a low priority task. So if you had 4 nodes it would run on 3 nodes.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: CPU Utilitization messages on only 1 node and cpu
PostPosted: Fri Sep 08, 2017 4:57 am 

Joined: Wed Nov 19, 2014 5:14 am
Posts: 505
I'm informed this is normal behavior if running dedupe.

Garbage collection runs on any none master node, load distribution will depend on CPG / DDS numbers. If the CPU is busy it will reclaim at a low priority, however if there's plenty of free CPU cycles it will take as much as is needed until it either completes or CPU front end demand increases, at which point it will throttle back again.

So in a two node system running dedupe, one of the nodes will periodically have higher CPU utilization but this is by design.


Last edited by JohnMH on Mon Sep 11, 2017 4:42 am, edited 1 time in total.

Top
 Profile  
Reply with quote  
 Post subject: Re: CPU Utilitization messages on only 1 node and cpu
PostPosted: Fri Sep 08, 2017 8:52 am 
User avatar

Joined: Thu Sep 01, 2011 8:27 am
Posts: 20
It's indeed normal behaviour. In our case the defrag of one of our DDS stores runs for about 50 to 70 minutes, which is just inside the window of the threshold alert defined.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 


Who is online

Users browsing this forum: No registered users and 41 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt