HPE Storage Users Group

Hello,

we have a 3PAR installation with 2 7200 with some SSDs and we're benchmarking it.

The setup :

- 2x 7200 up to date 3.2.1 MU2 + P07
- a VV with 8 SSD disks on each, SAS and NL
- FC 8Gb for production
- Remote Copy over FC 8Gb for synchronous replication
- 2 ESXi 5.1 with VMWare IO Analyzer for a 50/50 workload (BL490/8G FC)
- HP B-series 8G fabrics

In production, the 2 storeserv will be distant about 1km over dedicated link. As we discover performances problems, the 2 storages are now in the same room for investigation, we have still performances issues.

When we launch a benchmark on a SSD VV without RemoteCopy, we get expected IOPS and latency performances, on each storage, getting approx 30k IOPS under 5ms latency, which is fine.

When we enable RemoteCopy, IO fall down, from 30k IOPS to 10k IOPS in less than 10 minutes of workload and latency grow up from 0.5ms to 7ms in the same time.
Running the bench during several hours, IOPS continue to decrease to 0, and latency to go on and on.

As we have 2 other tiers, SAS and NL, benchmarks are good as expected on these tier, with or without replication.

It seems that we hurt a limit using RemoteCopy on SSD.

All around SAN, HBA, FC ports, SFP, has been checked and seem ok.
Interrupt coalescence on RCFC ports has been enabled/disabled with no improvement.

A case is open but I'm not very confident.

What could be the problem ?

How about :

- queue depth on RCFC ports ?
- sort of delayed acknowledge ?

How could we check these metrics ?

Delayed acknowleges can be monitored via the on-node-system reporter for cache stats.

I suspect something really strange and that support will find something. 3PAR is very chatty about rcfc-ports, if there were any problem on these you'd probably get dozens of mails/events regarding that problem.

Any AO-stuff going on? Auto-AO-CPGs (created automatically) involved?

_________________
When all else fails, read the instructions.

AO is disabled.

We got no error or warning on RCFC ports. But, during bench process, we got mail alert about quorum server unreachable. We don't really understand this error, the quorum process is running over IP, so...


CPU on the controller is not high during benchmark.

We will redo bench with performance charts on the 6 layers to look at.

I spent some time with performance reporter.

Here is a chart during 2 benchmarks, 2min and 4min long :


Light blue : CPU system % on the primary site
Purple : CPU system % on the secondary site (replication target)


Why is there a such difference between them ?
What kind of workload could use more cpu on the target side ?
Why does the target cpu stuck at 10-15% during some time after the end of the benchmark, with explosive latency on host ports ?

We got a answer from HP.

We benchmarked the arrays with VMware IOAnalyzer with 10 VMs on 2 ESX, seated on a unique Datastore/VV.

Seems that multiplicating the VV and balancing the 10 VMs on 4 VVs permits to have more acceptable performances and charts.

Is there any best practices about placement, VV number in a high IOPS scenario with RemoteCopy ?

Yeah, that's a bad design for a test, your limitation wasn't the array.

There's only so much Io that can go around in a datastore, the recommendation is to have around 8-12 vms per datastore, but not high io machines, spread your high io machines between datastores, or you'll see performance issues from the datastore not being able to provide the io, no matter what array you have behind it.

I'm not a fan of people benchmarking, as the majority of the time, the tests are pointless, there's nothing better than loading the array with real vms and data and not worrying.

We don't have performance issues on one datastore/VV without RemoteCopy.

The real problem is not the number of datatore in a vmware/array PoV but a problem with how RemoteCopy handles replicating VVs, something between cache or mono-threaded process ??

From ours tests, do not expect to get more than 10k stable IOPS in a 4k 50% workload from a replicated VV.