HPE Storage Users Group
https://3parug.com/

7400c Very slow CLI on COM and SSH and dropping from MC/VSP
https://3parug.com/viewtopic.php?f=18&t=3478		 Page 1 of 1
Author:  senate014 [ Thu Oct 08, 2020 8:24 am ]
Post subject:  7400c Very slow CLI on COM and SSH and dropping from MC/VSP
Hi there

I have a weird problem with a 7400c where after a couple of hours the systems CLI via SSH and Serial COM grinds to a very slow, sluggish performance when issuing any CLI commands as well as the system dropping it's connection intermittently from the Management Console and Virtual Service Processor.

Both Node's Mgmt RJ45 ports are plugged into a single Cisco Catalyst 4948 as well as the Hyper-V host server that holds the Management console and Virtual Service Processor VM's.

The Cisco Switch has a separate VLAN for the 3PAR Management interfaces that the Hyper-V host is also connected to.. So it's essentially isolated from the rest of the network via this VLAN.

The 7400c is running 3.3.1MU3 and is patched up to P116
It has 68 Virtual Volumes and a Snapshot policy that snaps all VV's every hour and retains the VV's for 4 days. Essentially each VV has 96 snapshots...

We have swapped out Node-0 already and plan to swap out Node-1 if we cannot figure out what the problem is here...

We have also tried running the "setsysmgr restart" command on several occasions which sometimes speeds up the CLI performance and stops it dropping from the MC/VSP...

One thing that did sort out the issue was stopping the Snapshot policies running.... but when we stared them backup again the CLI/SSH sessions started grinding to a holt along with the system dropping in and out of connection from the MC/VSP....

Rebooting one Node at a time also brings the performance back to normal, but again, it starts to slow and drop in and out from the MC/VSP after a couple of hours.

At no point do any host servers experience any issue with VV performance... It seems to be some sort of weird performance/connection issue with actually managing the system....

I've attached the current CPU usage....

Any suggestions or help would be much appreciated :)

Thanks

Attachments:
image_2020_10_08T12_57_04_724Z.png
image_2020_10_08T12_57_04_724Z.png [ 54.59 KiB | Viewed 9498 times ]
Author:  MammaGutt [ Thu Oct 08, 2020 9:47 am ]
Post subject:  Re: 7400c Very slow CLI on COM and SSH and dropping from MC/
It might be relevant, but weak CPU/low memory is why 7ks are recommended to stay at 3.2.2.

Did these suddenly appear or could they have been present for most of the time the system has been running 3.3.1?

Does all commands in CLI appear equally sluggish or are some worse than other?
Author:  senate014 [ Thu Oct 08, 2020 10:13 am ]
Post subject:  Re: 7400c Very slow CLI on COM and SSH and dropping from MC/
MammaGutt wrote:
It might be relevant, but weak CPU/low memory is why 7ks are recommended to stay at 3.2.2.

Did these suddenly appear or could they have been present for most of the time the system has been running 3.3.1?

Does all commands in CLI appear equally sluggish or are some worse than other?


Hi there and thanks for getting back to me.....

Most quick commands that display minimal information are okish... commands like "showtask" or "checkhealth" take forever...

I have just noticed that the 7400c disconnects from the MC rouhgly ever hour on the hour which is around the same time the hourly snapshots are taking place....

At the moment, we have also found that the system is unbalanced... the system had odd sets of disk in 4 out of 5 cages... 15x SAS Spinning Disk and 7x SSD's in 4 out of 5 cages.... The customer is going to install some more SSD and SAS Spinning Disk to rebalance the system by running "admithw" and "tunesys" once the drives have been installed.

I have also suspended all snapshot policies for the customer and the customer will be deleting all snapshots tonight so the system is back to the state where it just has the 68x base VV's.

The system has always been on 3.3.1MU3 P116.

Cheers!
Author:  MammaGutt [ Thu Oct 08, 2020 10:22 am ]
Post subject:  Re: 7400c Very slow CLI on COM and SSH and dropping from MC/
Just make sure when you rebalance that you get the drives in the correct order (they should be equal in pairs). So if there is SAS in the 15 first and SSD in the next 7 slots, you need to move the first SSD to get an even number of SAS drives in a row.

I find it hard to believe that the system has always been 3.3.1 MU3 P117 considering that was release maybe a year ago, while the final 7k left the HPE factory in 2016 :).

But you are saying that this problem hasn't been there forever on 3.3.1 MU3 and that it happened at some point...

Does the SP disconnect happen at the same time as you snap? In that case I would be even more certain about resource issues.

Checkhealth and showtask are always slow as they do a lot of things. What about showpd, showport, showdate, shownet and statport?
Page 1 of 1 All times are UTC - 5 hours
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/