HPE Storage Users Group
https://3parug.com/

Resent event when doing FW upgrade to emu4 from MU2 on 3.2.2
https://3parug.com/viewtopic.php?f=18&t=2649		 Page 1 of 1
Author:  sjm [ Wed Sep 13, 2017 9:31 pm ]
Post subject:  Resent event when doing FW upgrade to emu4 from MU2 on 3.2.2
Hi all,

Doing an upgrade on 8440 from 3.2.2 mu2 to emu +patches.(HPE doing the work)

Had just don't Node 0 restart and was checking the lun Paths in VMware.

2 of the 4 paths were down....Basically all paths via Node 1.....very weird to say the least.

HP update team insisted there were 4 paths showing to Hosts and typing the command show host -pathsummary...it reported 4 paths.

In VMware the Paths show 2 dead. could not convince HP team that this was true (even showing them this on the shared desktop view).

I logged into the fibre switches and the paths in the fibre switches for the node 1 connections show NO light.

At this point hp went "oh" I said we could no proceed with a node 1 restart (which in hindsight I probably) could off.

I asked for support to jump in and the update team passed over to support.

The sfp's had to be hard reset on Node 1. Once this was done the paths came back up and we rolled back node 0

(I have truncated this story as I had to drive to data center and I replaced sfp's in the switches and ran some test fibre just incase)

After roll back HP investigated and found its a known issue.....(not known by many outside of hP apparently).

The Emulex cards in ALL 3pars have a FW issue ...which is not due to be addressed until 3.2.2 mu6.

We were the first known incident of the fault actually happening.......

They found via the logs etc the exact systems the fw issue is .....

The global update team were not aware of it....and have now been advised and we have scheduled another attempt to bring the 3par to emu4 (our other 8000 series we did a while ago with no issue)

Basically my understanding is every Emulex adapter in a 3par has this fault until 3.2.2 mu6 is applied.

Nothing like being first
Author:  storsnapper [ Mon Sep 18, 2017 11:36 am ]
Post subject:  Re: Resent event when doing FW upgrade to emu4 from MU2 on 3
Can you elaborate on the method needed to bring ports back online after the bug was encountered? I am wondering if I will have the same issue upgrading from 3.2.2 MU2 to 3.3.1 with HPE.

Thank you
Author:  sjm [ Wed Sep 20, 2017 7:29 pm ]
Post subject:  Re: Resent event when doing FW upgrade to emu4 from MU2 on 3
Ok what we found on checking that at our end we saw no light in the fibre switches on the 3par Node affected.
Also saw paths down in VMware.

Hp said all paths to hosts were up showhosts -path summary (command syntax so forgive me)

The check that's needs to be done is if data is flowing command

statport -d 30 -host -iter 1

This shows no data going through the fibre ports on the node.

At that stage a reset of the sfp's is done (I don't know the command as HP did this)

Then re check above and all is fine
Author:  ailean [ Thu Sep 21, 2017 4:07 am ]
Post subject:  Re: Resent event when doing FW upgrade to emu4 from MU2 on 3
Glad it was you who was first, our server admins don't show much interest during upgrades as the persistent ports stuff just makes it transparent and they are used to it just working. ;)

Out of interest did you run a showhost -pathsum while the Node was down?

Typically during upgrades I watch the port states on Brocade/HPE SNA (go offline then come back) and once down do a showhost -pathsum to confirm which hosts have reduced number of paths, also a shownode.

Once the ports come back I do shownode till it's back in the cluster and then another showhost -pathsum to check all paths are back.

I'm assuming with this bug everything would have looked normal with the 3PAR checks but at least my fabric check would have been big red light. :o
Author:  aberus [ Thu Sep 21, 2017 9:07 am ]
Post subject:  Re: Resent event when doing FW upgrade to emu4 from MU2 on 3
The only way I know to reset the SFP is to reset the port, this can be done using the process below or by physically resetting the SFP.

To view all of your SFP's enter: showport -sfp
To get a detailed list you can enter: showport -sfp -d
If you know the N:S:P you can enter: showport -sfp -d N:S:P

To reset the port you can issue the following (this should also reset the SFP):
controlport offline N:S:P:
controlport rst N:S:P:

If that doesn't work you can force a re-registration of the port with the fabrics name server by issuing the command: controlport nssync N:S:P:

Another thing I do anytime I am working with support is that I join the HPE Room from a VM where I have Putty installed and pre-configured to capture all output to a test file. This way I can always go back and reverse engineer what they have done which I typically organize and enter it into my knowledge base.
Author:  sjm [ Mon Sep 25, 2017 8:50 pm ]
Post subject:  Re: Resent event when doing FW upgrade to emu4 from MU2 on 3
ailean wrote:

Out of interest did you run a showhost -pathsum while the Node was down?




This is what the upgrade team you using to say everything was ok rather than checking data was actually going through the fibre links in node 1.
Author:  sjm [ Mon Sep 25, 2017 8:53 pm ]
Post subject:  Re: Resent event when doing FW upgrade to emu4 from MU2 on 3
aberus wrote:
The only way I know to reset the SFP is to reset the port, this can be done using the process below or by physically resetting the SFP.

To view all of your SFP's enter: showport -sfp
To get a detailed list you can enter: showport -sfp -d
If you know the N:S:P you can enter: showport -sfp -d N:S:P

To reset the port you can issue the following (this should also reset the SFP):
controlport offline N:S:P:
controlport rst N:S:P:

If that doesn't work you can force a re-registration of the port with the fabrics name server by issuing the command: controlport nssync N:S:P:

Another thing I do anytime I am working with support is that I join the HPE Room from a VM where I have Putty installed and pre-configured to capture all output to a test file. This way I can always go back and reverse engineer what they have done which I typically organize and enter it into my knowledge base.



You could be right.....by this stage I was planning for a NOde 1 replacement...the SFP reset was a shot in the dark really.

whats concerning is every Emulex pbased 3par has this possible issue until 3.2.2 MU6 when a FW update is scheduled to fix it
Author:  storsnapper [ Tue Sep 26, 2017 3:12 pm ]
Post subject:  Re: Resent event when doing FW upgrade to emu4 from MU2 on 3
sjm wrote:
aberus wrote:
The only way I know to reset the SFP is to reset the port, this can be done using the process below or by physically resetting the SFP.

To view all of your SFP's enter: showport -sfp
To get a detailed list you can enter: showport -sfp -d
If you know the N:S:P you can enter: showport -sfp -d N:S:P

To reset the port you can issue the following (this should also reset the SFP):
controlport offline N:S:P:
controlport rst N:S:P:

If that doesn't work you can force a re-registration of the port with the fabrics name server by issuing the command: controlport nssync N:S:P:

Another thing I do anytime I am working with support is that I join the HPE Room from a VM where I have Putty installed and pre-configured to capture all output to a test file. This way I can always go back and reverse engineer what they have done which I typically organize and enter it into my knowledge base.



You could be right.....by this stage I was planning for a NOde 1 replacement...the SFP reset was a shot in the dark really.

whats concerning is every Emulex pbased 3par has this possible issue until 3.2.2 MU6 when a FW update is scheduled to fix it



Thank you for the update! Another thing to add to my checklist during upgrades.
Page 1 of 1 All times are UTC - 5 hours
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/