HPE Storage Users Group

So I will open up the can of worms by stating that I do not believe the issue is the 3PAR itself. When I first started noticing that my brand new 7200 had read latency spikes I was shocked. It's been in service since December 2014 and has seen these read spikes ever since. I opened up a ticket with HP just to clear the 3PAR itself from being the issue.

System Specs:
7200 - 8Gb FC
24 900GB 10K drives split across cage0 and cage1
RAID 5 CPG in 3+1 128Kib step size with 32GB growth increment
All vv on thin provisioned
All VMDK are thick eager-zeroed
Emulex LPe16000 16Gb FC cards with 16Gb SFP+ hard set to 8Gb on cards and on the HP8/8 FC switch
ESXi 5.5 U2
HP DL360e G8 with latest firmware installed on all hardware.

without graphs on 3par frontend vlun IO, cache busyness, histograms of IOs serviced in the 3par and backend drive busyness, it is impossible to say what exactly is going on.

The short spikes might be 3par related, due to overloading of the system/drives, they might also be vmware related due to cpu saturation, it might even ben SAN related, due to port sharing on a cisco switch, etc.

What is the point you are exactly trying to make?

_________________
The goal is to achieve the best results by following the clients wishes. If they want to have a house build upside down standing on its chimney, it's up to you to figure out how do it, while still making it usable.

I'm trying to see if anyone has any ideas. I have involved HP 3PAR support and they looked into the performance on the 3PAR side from SPOCC. I agree with their assessment that the 3PAR is performing as expected given the limitations of the 10K disks. From previous experience with the 3PAR products I definitely wouldn't expect to be seeing spikes like this when the old infrastructure was a HP VSA iSCSI environment.

I made changes to the Brocades for the fill-word issue which at least cleared those errors but has made no improvement to the latency spikes. I made sure to hard set everything at 8Gb. The FC cards are 16Gb so I wanted to make sure everything was configured to talk at the same speed and avoid autonegotiation issues like those I have seen in the past with P2000, VNX/VNXe and DataCore storage.

I apologize if my question wasn't clear earlier. I have been known to have an occasional "squirrel" moment.

I'll work on pulling the other information from the 3PAR and VMware hosts.

No apologies needed, i was a maybe a bit coarse in my wording. i'll try to help where ever i can.

As these are such very short spikes, they will be harder to catch.
Let us know if you need assistence with cli commands to capture the 3par perfomance.

_________________
The goal is to achieve the best results by following the clients wishes. If they want to have a house build upside down standing on its chimney, it's up to you to figure out how do it, while still making it usable.

Only using vSphere performance graphing can be deceiving.
I'd measure from inside the VM as well just to be sure you observe same behavior there.

ESX server measure latency at the top SCSI layer, which means it will catch VAAI offloading commands like XCOPY.
This is used for storage vMotion, cloning etc and will have higher measured latency as it works at 8 MB/command by default.
(then 70 ms don't seems to bad...)

Another option is to disable VAAI temporary to make sure you have data you can trust.

http://kb.vmware.com/kb/2012288