HPE Storage Users Group

we crossed messages. Please see the update i've made on the ISLs. sorry for the confusion.

_________________
The goal is to achieve the best results by following the clients wishes. If they want to have a house build upside down standing on its chimney, it's up to you to figure out how do it, while still making it usable.

Allright Thanks

just had some time to look at your pictures. if all the hostports and 3par ports are clean of errors, and correctly set for speed and fillword, I would have a good look at those ISLS between the C7000 and the main brocade switches.

as they are 4gb, both sides should be fixed on 4gbit.
please check each port on every side, are they clean of any errors? (reset the counters, do 2 or 3 test runs and watch the counters, any increase?)
another thing to watch is if you run out of buffer-to-buffer credits or not.

(post a portstatsshow of the isl ports, after resetting counters and doing the tests, if you are in doubt what to look for)

_________________
The goal is to achieve the best results by following the clients wishes. If they want to have a house build upside down standing on its chimney, it's up to you to figure out how do it, while still making it usable.

I've changed the ISLs between C7000 and the main switches to 4gb fixed, cleared the portstats. No increase in errors but the performance was still poor.

I've then decided to change the 3par ports (on the brocade) to 4gb fixed just for the fun of it and to my surprise this changed the game

IO meter for 32kb write 0% random is now

IOP: 10600
MBs pr sec: 332
Avarage time ms: 0,936
Max time ms: 39,976

If i increase the block size to 5mb transfer rate is about 1GB pr sec!

Maybe my understanding of how mix of FC port speed works is wrong? Our entire FC infrastructure was running 4GB until we got our 3par so I did not think twice about it.
We are about to install another c7000 with 8gbit FC and it would be a pity to force it down to 4GB

Interesting! I’ve started a HP case and apparently the storage guy from HP has heard about this before.

Quote:
“As discussed, can you please generate a new Insplore so that I can see if the problem you are getting is related to the issue with connecting to the 3Par at 8GB. If this is the case, a workaround is to connect to the 3Par at 4GB.”

New reply from HP

"I’ve received the Insplore and it does report a number of “Invalid CRC” and “Invalid Word” errors against different ports on the 3Par. Attached is the output from the Insplore that shows the number of “InvCRC” and “InvWord” errors being reported against each port in the 3Par.

Can you try the following:

Set the FILLWORD setting to 3 on all the switch ports being used by the 3par. Do you get the performance issue when running at 8GB? Use the “portcfgfillword” command on the switch to change this.

If you are still having a problem, please do the following as a workaround:

• Set the 3Par ports to 4GB using the command: controlport rate 4 <node:slot:port>. You need to run this for all the 3par fibre ports connected to the SAN switches.
• Set the SAN switches to 4GB

Has this workaround corrected the issue?

A proper fix will be made available at a later date, but I do not have an ETA.

If it is not acceptable for you that the 3par runs at 4GB until the fix is made available, we can look to replace the fibre SFPs within the 3Par"

Obviously the fillword is not the issue as 3 is the default value on the 8Gb switch and I’ve already established that the workaround “correct” the issue. It’s beginning to point I direction of a known issue with the SFPs!

HP confirmed a known issue and offered to replace the SFPs which I accepted. I will post my findings when they arrive.

Probably handy to note the part numbers of the old and new SFPs in case someone else comes across this. Glad to see a potential end to your issues.

Indeed… I’m new to 3PAR so I can’t contribute much in this forum…yet… but here you go

It turns out that HP get there SFP produced in at least two different companies (Avago and Finisar) and label them with the same product number, they don’t have the same finish on the casing though
The only way to spot the difference is to get the information from CLI (supportshow).

(HP-A) SFPs require replacement (Avago)
(HP-F) is fine (Finisar)

The performance is now as it should be

---------Port 0:1:1----------
N:S:P : 0:1:1
State : OK
Manufacturer : HP-A
Part Number : AJ718A
Serial Number : 3CB324AM1A
Revision : A
MaxSpeed(Gbps) : 8.5
Qualified : Yes
TX Disable : No
TX Fault : No
RX Loss : No
RX Power Low : No
DDM Support : Yes

---------Port 0:1:2----------
N:S:P : 0:1:2
State : OK
Manufacturer : HP-A
Part Number : AJ718A
Serial Number : 3CB324AN9G
Revision : A
MaxSpeed(Gbps) : 8.5
Qualified : Yes
TX Disable : No
TX Fault : No
RX Loss : No
RX Power Low : No
DDM Support : Yes

---------Port 1:1:1----------
N:S:P : 1:1:1
State : OK
Manufacturer : HP-F
Part Number : AJ718A
Serial Number : MY83230D6R
Revision : A
MaxSpeed(Gbps) : 8.5
Qualified : Yes
TX Disable : No
TX Fault : No
RX Loss : No
RX Power Low : No
DDM Support : Yes

---------Port 1:1:2----------
N:S:P : 1:1:2
State : OK
Manufacturer : HP-F
Part Number : AJ718A
Serial Number : MY83230D70
Revision : A
MaxSpeed(Gbps) : 8.5
Qualified : Yes
TX Disable : No
TX Fault : No
RX Loss : No
RX Power Low : No
DDM Support : Yes

---------Port 2:1:1----------
N:S:P : 2:1:1
State : OK
Manufacturer : HP-A
Part Number : AJ718A
Serial Number : 3CB324AM09
Revision : A
MaxSpeed(Gbps) : 8.5
Qualified : Yes
TX Disable : No
TX Fault : No
RX Loss : No
RX Power Low : No
DDM Support : Yes

---------Port 2:1:2----------
N:S:P : 2:1:2
State : OK
Manufacturer : HP-F
Part Number : AJ718A
Serial Number : MY8323095W
Revision : A
MaxSpeed(Gbps) : 8.5
Qualified : Yes
TX Disable : No
TX Fault : No
RX Loss : No
RX Power Low : No
DDM Support : Yes

---------Port 3:1:1----------
N:S:P : 3:1:1
State : OK
Manufacturer : HP-A
Part Number : AJ718A
Serial Number : 3CB324ANER
Revision : A
MaxSpeed(Gbps) : 8.5
Qualified : Yes
TX Disable : No
TX Fault : No
RX Loss : No
RX Power Low : No
DDM Support : Yes

---------Port 3:1:2----------
N:S:P : 3:1:2
State : OK
Manufacturer : HP-F
Part Number : AJ718A
Serial Number : MY83230966
Revision : A
MaxSpeed(Gbps) : 8.5
Qualified : Yes
TX Disable : No
TX Fault : No
RX Loss : No
RX Power Low : No
DDM Support : Yes

We had the same/similar problem with the following hardware:
HP BL460c Gen8 (554FLB and 554M Emulex Cards)
FlexFabric
3PAR T400
HP 8/24 8GB fiber switches

We also contacted HP, and even told them about we have corrected the fill word, and yet they told us to change it.... HP support isn't exactly known for their excellence... (VMware support blamed HP, HP Proliant team blamed Storage team, Storage team blamed fillword, which was correct to begin with)

After testing around a little, we found that we only had the problem with ESXi servers connected to the SAN, and after multiple test with different linux kernels we found that the problem lied in the lpfc driver.

Updating the lpfc driver from vmware's own driver repository solved the problem for us:
https://my.vmware.com/web/vmware/detail ... ductId=285

Also make sure you have the newest Emulex firmware if your are running the Oneconnect cards.

Newest firmware for the 554 series:
http://h20565.www2.hp.com/portal/site/h ... cachetoken