HPE Storage Users Group

We currently run a 2-node V400 with 96 FC disks. We have more than enough capacity but we are running into queue depth/latency issues. We have ~160 blade servers that connect via 8Gb FC through the SAN to our V400 array. We run ~150 server vms that always stay on and run a variety of applications such as sharepoint, exchange, IIS, etc (this is a testing environment).

We also have ~300 client vms in the same vSphere environment. We are constantly allocating clients and servers for a specific test case and kicking off these scenarios. While our CPU/RAM/network appears more than adequate, our storage is (no surprise) the bottleneck. As mentioned, we have more than enough capacity but when we do have issues it appears to be due to the mixed load of IO requests and this hits the queue depth/latency pretty hard.

So.... what would YOU recommend as your next step? Is it worth adding another array, in this case a 7400? We do plan to add more and more servers and clients as time goes on. Or should we just add SSDs? Are SSDs in the V400/7400 worth the cost for an environment like ours? What about a drastically different approach of VSAN or VSA or server-side SSDs? Do we just need more spindle count in our existing CPGs to handle the requests?

96 spindles for 460 VMs sounds a little low, are those 15k drives?
If you have plenty of space it sounds like they are fairly large drives, we found 15k 450GB/600GB to have given us a reasonable balance between space/performance when expanding our arrays with our usage pattern.
We don't have very many high IO VMs thou so not much of a driver for us to invest in SSD yet (it's not exactly cheap on 3PAR).

Well it depends on where your problem lays, but it sounds like spindle contention in which case you can either limit the I/O for specific data via QOS (storage or VMware) or add more disk and you have plenty of scope within the current 10400 without needing to break out to another array. If your workload is write intensive, given you have plenty of free space, you could also consider converting some of the busier stuff to Raid 10 via DO, meaning less backend I/O overhead.

Whether SSD will be of benefit will be dependent on the I/O density of your data, you can get a feel for this via the System Reporter AO reports which will show the amount of data that would be able to take advantage of SSD. If you need help interpreting this then speak to HP Presales who can provide this FOC. With the new 480GB and 920GB SSD drives cost per GB has reduced significantly.

VSA & VSAN solutions are good on paper, but the need to mirror everything between nodes over an IP stack will add even more latency, especially in a write intensive environment, I would steer away from this unless it's for a very specific use case.

Thanks for the feedback guys!

Yeah, we have 600GB 15k disks. We were using RAID10 but switched out after not seeing much improvement. A lot of our requests are reads though, so that makes sense.

I will have to look at the System Reporter AO and see what results I get.

On the VSAN, etc solutions those were also my findings and conclusion... I was just curious to hear others' thoughts on the matter.

Really I am thinking with AO and some SSDs we could handle the boot storms and a lot of the crazy servers without having to throw more spindles at it when we dont need capacity.

That sounds like the perfect target for AO and SSD.

_________________
Richard Siemers
The views and opinions expressed are my own and do not necessarily reflect those of my employer.

Set up system reporter and run reports on the physical drives to get service times, that will tell you how busy they are. SSD only helps if the blocks are on the SSD and the lag for AO to get data on the right tier may negate the benefit. We find AO working only on usage patterns that stay regularly predictable. I am a big fan of a rising tide raises all boats and maybe adding more 15K drives would help. We run 384 15K drives and 128 10K drives and then 128 NL drives and we find NL drives are always over subscribed and will be adding more 10K drives and relegating the NLdrices to snaps and low IO file servers. We find AO's lag times to not help when workload changes.

AO today is not going to help with transitory spikes, that's what cache is for. But large sustained spikes such as boot storms will overwhelm cache, so AO is probably not going to help in that case. How you alleviate that may very well be outside of the storage. Just adding spinning disk is basically risk free as you'll benefit from the aggregate performance of the pool, getting the most out of SSD is a little more nuanced and requires some analysis and subsequent tweaking.

What would be considered bad service times on 600GB 15k FC disks?

I understand AO cannot adapt pro-actively unless the IO patterns stay predictable, but wouldn't SSD's help with service times, queue depths, etc by processing the IO requests quicker than 15k disks can?

15K drives I like to see <10ms on average and prefer around 5ms and 7K drives I like <20ms, I will tell you I am crushing my array currently and see 20-40ms on 7K and 10-15ms on 15K and no complaints. An important number to look at is the node cache performance, outside our backup windows we average over 80% cache hit on reads which means 80% of the time the IO is coming from cache so the disk speed is fairly irrelevant in all but 20% of the IOPS. The problem with 7xxx line is is has less cache than the V400 and my understanding is they just doubled the cache on the V400 so the V400 has a considerable cache advantage. Unfortunately HP is not offering an upgrade path for V400 customers to get the larger cache.

My 2 cents: ssd's can be very helpful in desktop environments. What you want to do is use space efficient desktops where possible (linked clones), and provision your desktops out of the ssd tier. That way AO can more down the less important blocks, but you get the performance you need.

With regard to cache (and I know you'll love this hdtvguy), hp's answer is stay tuned. Supposedly adaptive flash cache is coming in 3.1.3 MU1