HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 8 posts ] 
Author Message
 Post subject: Resent event when doing FW upgrade to emu4 from MU2 on 3.2.2
PostPosted: Wed Sep 13, 2017 9:31 pm 

Joined: Mon Jul 29, 2013 9:01 pm
Posts: 62
Hi all,

Doing an upgrade on 8440 from 3.2.2 mu2 to emu +patches.(HPE doing the work)

Had just don't Node 0 restart and was checking the lun Paths in VMware.

2 of the 4 paths were down....Basically all paths via Node 1.....very weird to say the least.

HP update team insisted there were 4 paths showing to Hosts and typing the command show host -pathsummary...it reported 4 paths.

In VMware the Paths show 2 dead. could not convince HP team that this was true (even showing them this on the shared desktop view).

I logged into the fibre switches and the paths in the fibre switches for the node 1 connections show NO light.

At this point hp went "oh" I said we could no proceed with a node 1 restart (which in hindsight I probably) could off.

I asked for support to jump in and the update team passed over to support.

The sfp's had to be hard reset on Node 1. Once this was done the paths came back up and we rolled back node 0

(I have truncated this story as I had to drive to data center and I replaced sfp's in the switches and ran some test fibre just incase)

After roll back HP investigated and found its a known issue.....(not known by many outside of hP apparently).

The Emulex cards in ALL 3pars have a FW issue ...which is not due to be addressed until 3.2.2 mu6.

We were the first known incident of the fault actually happening.......

They found via the logs etc the exact systems the fw issue is .....

The global update team were not aware of it....and have now been advised and we have scheduled another attempt to bring the 3par to emu4 (our other 8000 series we did a while ago with no issue)

Basically my understanding is every Emulex adapter in a 3par has this fault until 3.2.2 mu6 is applied.

Nothing like being first


Top
 Profile  
Reply with quote  
 Post subject: Re: Resent event when doing FW upgrade to emu4 from MU2 on 3
PostPosted: Mon Sep 18, 2017 11:36 am 

Joined: Thu Jul 06, 2017 3:28 pm
Posts: 41
Can you elaborate on the method needed to bring ports back online after the bug was encountered? I am wondering if I will have the same issue upgrading from 3.2.2 MU2 to 3.3.1 with HPE.

Thank you


Top
 Profile  
Reply with quote  
 Post subject: Re: Resent event when doing FW upgrade to emu4 from MU2 on 3
PostPosted: Wed Sep 20, 2017 7:29 pm 

Joined: Mon Jul 29, 2013 9:01 pm
Posts: 62
Ok what we found on checking that at our end we saw no light in the fibre switches on the 3par Node affected.
Also saw paths down in VMware.

Hp said all paths to hosts were up showhosts -path summary (command syntax so forgive me)

The check that's needs to be done is if data is flowing command

statport -d 30 -host -iter 1

This shows no data going through the fibre ports on the node.

At that stage a reset of the sfp's is done (I don't know the command as HP did this)

Then re check above and all is fine


Top
 Profile  
Reply with quote  
 Post subject: Re: Resent event when doing FW upgrade to emu4 from MU2 on 3
PostPosted: Thu Sep 21, 2017 4:07 am 

Joined: Wed Nov 09, 2011 12:01 pm
Posts: 392
Glad it was you who was first, our server admins don't show much interest during upgrades as the persistent ports stuff just makes it transparent and they are used to it just working. ;)

Out of interest did you run a showhost -pathsum while the Node was down?

Typically during upgrades I watch the port states on Brocade/HPE SNA (go offline then come back) and once down do a showhost -pathsum to confirm which hosts have reduced number of paths, also a shownode.

Once the ports come back I do shownode till it's back in the cluster and then another showhost -pathsum to check all paths are back.

I'm assuming with this bug everything would have looked normal with the 3PAR checks but at least my fabric check would have been big red light. :o


Top
 Profile  
Reply with quote  
 Post subject: Re: Resent event when doing FW upgrade to emu4 from MU2 on 3
PostPosted: Thu Sep 21, 2017 9:07 am 

Joined: Mon Apr 28, 2014 4:51 pm
Posts: 38
Location: Rockford, MI
The only way I know to reset the SFP is to reset the port, this can be done using the process below or by physically resetting the SFP.

To view all of your SFP's enter: showport -sfp
To get a detailed list you can enter: showport -sfp -d
If you know the N:S:P you can enter: showport -sfp -d N:S:P

To reset the port you can issue the following (this should also reset the SFP):
controlport offline N:S:P:
controlport rst N:S:P:

If that doesn't work you can force a re-registration of the port with the fabrics name server by issuing the command: controlport nssync N:S:P:

Another thing I do anytime I am working with support is that I join the HPE Room from a VM where I have Putty installed and pre-configured to capture all output to a test file. This way I can always go back and reverse engineer what they have done which I typically organize and enter it into my knowledge base.

_________________
Adam Berus - I.S. Tech Lead - Wolverine Worldwide Inc.
x1-7200, x2-7400, x1-7450, x1-8400, x2-8440
3PAR Technologies: AFC, AO, FilePersona, RemoteCopy (RCIP)


Top
 Profile  
Reply with quote  
 Post subject: Re: Resent event when doing FW upgrade to emu4 from MU2 on 3
PostPosted: Mon Sep 25, 2017 8:50 pm 

Joined: Mon Jul 29, 2013 9:01 pm
Posts: 62
ailean wrote:

Out of interest did you run a showhost -pathsum while the Node was down?




This is what the upgrade team you using to say everything was ok rather than checking data was actually going through the fibre links in node 1.


Top
 Profile  
Reply with quote  
 Post subject: Re: Resent event when doing FW upgrade to emu4 from MU2 on 3
PostPosted: Mon Sep 25, 2017 8:53 pm 

Joined: Mon Jul 29, 2013 9:01 pm
Posts: 62
aberus wrote:
The only way I know to reset the SFP is to reset the port, this can be done using the process below or by physically resetting the SFP.

To view all of your SFP's enter: showport -sfp
To get a detailed list you can enter: showport -sfp -d
If you know the N:S:P you can enter: showport -sfp -d N:S:P

To reset the port you can issue the following (this should also reset the SFP):
controlport offline N:S:P:
controlport rst N:S:P:

If that doesn't work you can force a re-registration of the port with the fabrics name server by issuing the command: controlport nssync N:S:P:

Another thing I do anytime I am working with support is that I join the HPE Room from a VM where I have Putty installed and pre-configured to capture all output to a test file. This way I can always go back and reverse engineer what they have done which I typically organize and enter it into my knowledge base.



You could be right.....by this stage I was planning for a NOde 1 replacement...the SFP reset was a shot in the dark really.

whats concerning is every Emulex pbased 3par has this possible issue until 3.2.2 MU6 when a FW update is scheduled to fix it


Top
 Profile  
Reply with quote  
 Post subject: Re: Resent event when doing FW upgrade to emu4 from MU2 on 3
PostPosted: Tue Sep 26, 2017 3:12 pm 

Joined: Thu Jul 06, 2017 3:28 pm
Posts: 41
sjm wrote:
aberus wrote:
The only way I know to reset the SFP is to reset the port, this can be done using the process below or by physically resetting the SFP.

To view all of your SFP's enter: showport -sfp
To get a detailed list you can enter: showport -sfp -d
If you know the N:S:P you can enter: showport -sfp -d N:S:P

To reset the port you can issue the following (this should also reset the SFP):
controlport offline N:S:P:
controlport rst N:S:P:

If that doesn't work you can force a re-registration of the port with the fabrics name server by issuing the command: controlport nssync N:S:P:

Another thing I do anytime I am working with support is that I join the HPE Room from a VM where I have Putty installed and pre-configured to capture all output to a test file. This way I can always go back and reverse engineer what they have done which I typically organize and enter it into my knowledge base.



You could be right.....by this stage I was planning for a NOde 1 replacement...the SFP reset was a shot in the dark really.

whats concerning is every Emulex pbased 3par has this possible issue until 3.2.2 MU6 when a FW update is scheduled to fix it



Thank you for the update! Another thing to add to my checklist during upgrades.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 


Who is online

Users browsing this forum: No registered users and 47 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt