HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 8 posts ] 
Author Message
 Post subject: Anyone Experiance Latency During Host Reboots?
PostPosted: Tue Jun 05, 2018 5:11 pm 
User avatar

Joined: Sat May 03, 2014 2:01 pm
Posts: 71
Location: Dallas, TX
We are on the 3rd week of troubleshooting an issue that has recently started on one of our 7400 4-nodes where if we reboot a connected host we see latency on all of the other hosts zoned to the system.

Environment is a mixture of 21 Windows bare-metal and 34 ESXi 6.5 hosts. SAN switching is an HA pair of HP branded DCX8510 (SN8000B-8) Directors

The latency doesn't show up on the array via performance logging or systemreporter graphs, but we do see it on windows host perfmons as 1000-6000ms IO stalls on the windows hosts (they are high transaction/sec SQL clusters) and in less aggressively but still present in vCenter/vROps on VM data stores. Nasty business for sure.

If we reboot either an ESXi server or Windows server we see the issue on both ESXi and Windows servers that are zoned to the 7400. The same ESXi servers are also zoned to a 7440c and an 8400 and none of those arrays see latency when either Windows or ESXi servers reboot.

HPE Engineering is stating the issue is RSCN related and having us reduce the number of paths and a bunch of other best practice type stuff but haven't given us root cause as to why this particular array is started having issues with RSCN (assuming that is the issue). We deployed all post MU4 patches to the array, changed our fabric principal to our directors, changed out uplink cabling, reset ports, engaged VMWare,etc

No port error counters for CRCs, no errors in showportlesb on 3PAR ports, nothing shows in MAPS for the SAN switches except that HPE shows RSCN events logged during the latency events (which makes sense if a host reboots)

Has anyone seen anything like this and if so what was the fix?

_________________
Bryan W
Senior Architect/Manager of System Infrastructure, Dallas TX
https://www.linkedin.com/in/bryanlwhite


Top
 Profile  
Reply with quote  
 Post subject: Re: Anyone Experiance Latency During Host Reboots?
PostPosted: Tue Jun 05, 2018 9:49 pm 

Joined: Sun Dec 21, 2014 3:05 pm
Posts: 72
I am facing a very similiar issue in an AIX environment with high virtualization. We have had HPE send out SAN analyzer, etc..

We also ran a script on the 3par to stop fabric IOCTLs to reduce chatter between 3pars and fabric name server which did help.


Top
 Profile  
Reply with quote  
 Post subject: Re: Anyone Experiance Latency During Host Reboots?
PostPosted: Wed Jun 06, 2018 8:22 am 

Joined: Wed Nov 09, 2011 12:01 pm
Posts: 392
Are all the hosts zoned to the same ports or do you have any separation say between windows and esx hosts?

Any differences in the zoning method or the host port configs between the issue array and the other ones?

Seems an odd issue, at the rate our admins reboot servers I can certainly say we've not seen it here. :roll:


Top
 Profile  
Reply with quote  
 Post subject: Re: Anyone Experiance Latency During Host Reboots?
PostPosted: Wed Jun 06, 2018 12:06 pm 

Joined: Sun Dec 21, 2014 3:05 pm
Posts: 72
We have an 8440. We zoned so that each host WWN is zoned to a port pair e.g. 2:2:2/3:2:2.

Issue was happening across multiple ports.

We changed our zoning to peer zones, reduced our fan-out ratio to 1:102 (initiators per storage port), and implemented the skip ioctl script on all our arrays.


Top
 Profile  
Reply with quote  
 Post subject: Re: Anyone Experiance Latency During Host Reboots?
PostPosted: Wed Jun 06, 2018 9:39 pm 
User avatar

Joined: Sat May 03, 2014 2:01 pm
Posts: 71
Location: Dallas, TX
Thanks, for the reply’s. We have the skipioctl on the suspect array and are setting the last of the hosts to 1:2 on mirrored port pairs isolated between esx and Windows. Just weird that we never saw the issue until recently with our old zoning scheme.

_________________
Bryan W
Senior Architect/Manager of System Infrastructure, Dallas TX
https://www.linkedin.com/in/bryanlwhite


Top
 Profile  
Reply with quote  
 Post subject: Re: Anyone Experiance Latency During Host Reboots?
PostPosted: Thu Jun 07, 2018 8:42 am 

Joined: Wed Nov 09, 2011 12:01 pm
Posts: 392
Could be any combo of array, fabric, host patches or config tweaks causing a change in behaviour I guess.


For 3PAR I use 1:1 zones, 4 paths per Host, separate ESX from other hosts and spread the hosts between node pairs to try to keep it even and below 64 hosts per port.


I know the smartsan stuff is meant to cause extra load on switches and I'm sure there are other recent features that might change behaviours. There's also been some 3PAR HBA firmware patches this year that might fix issues on that end.


Hopefully the config tweaks you're doing at the moment will help.


Top
 Profile  
Reply with quote  
 Post subject: Re: Anyone Experiance Latency During Host Reboots?
PostPosted: Fri Jun 08, 2018 4:24 pm 
User avatar

Joined: Sat May 03, 2014 2:01 pm
Posts: 71
Location: Dallas, TX
Initial testing points to the zoning resolving the symptoms.

_________________
Bryan W
Senior Architect/Manager of System Infrastructure, Dallas TX
https://www.linkedin.com/in/bryanlwhite


Top
 Profile  
Reply with quote  
 Post subject: Re: Anyone Experiance Latency During Host Reboots?
PostPosted: Mon Jun 11, 2018 5:00 am 

Joined: Wed Nov 09, 2011 12:01 pm
Posts: 392
BryanW wrote:
Initial testing points to the zoning resolving the symptoms.


Excellent! Glad you're finding some progress in this.

What were the changes from/to in the end?


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 


Who is online

Users browsing this forum: No registered users and 64 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt