HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 16 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: SSD Performance 3PAR 7200
PostPosted: Thu Jun 09, 2016 1:06 pm 

Joined: Wed Jun 08, 2016 5:00 am
Posts: 8
Hi Guys,

Greetings to everyone :D , i hope someone could help me with an issue.
We have some new 1,92TB SSDs in use in our environment.
2 x 3PAR 7200 with OS Version 3.2.2 (MU2) , the SSDs are mounted in an additional shelf.
8 x 1,92 TB SSD per site , 1 primary site and the other as backup.
RC Syncronous is configured.
CPG is an R1 with 1 VV inside in an dedicated RC Group only for that VV.
We have very poor performance when it comes to common file copy operations in VMs located at that LUN , sometimes performance goes to 30MB/s to 50MB/s (with large files (6GB))

We have done some troubleshooting and performance testing with an HPE Technican. (Onsite)
But we have found nothing what could cause this behavior.
according to i/o meter , everything is fine , troughput , io/s ..
But again, when it comes to common file operations :) ... performance is very poor ..
Does anyone have an idea? This would be very great.

Thank you , Daniel


Top
 Profile  
Reply with quote  
 Post subject: Re: SSD Performance 3PAR 7200
PostPosted: Thu Jun 09, 2016 1:33 pm 
User avatar

Joined: Fri Jun 21, 2013 1:03 pm
Posts: 109
Location: USA
How is your RC configuration setup?
What kind of links do you have between the sites?
What kind of latency are you seeing between the sites?
What kind of utilization are you seeing on the RC links?


Top
 Profile  
Reply with quote  
 Post subject: Re: SSD Performance 3PAR 7200
PostPosted: Thu Jun 09, 2016 3:19 pm 

Joined: Wed Jun 08, 2016 5:00 am
Posts: 8
Hi,

we use the flags auto_failover and path_management.
in the rc group is only the ssd vv.
between the 2 sites we have 2 x 8gb FC Links at 2 Fabrics.
on the rc links is an average throughput of 15m

but look . ...

latency is about 1300ms and qlen about 640 at the rc links
wtf...


Top
 Profile  
Reply with quote  
 Post subject: Re: SSD Performance 3PAR 7200
PostPosted: Thu Jun 09, 2016 3:22 pm 
User avatar

Joined: Fri Jun 21, 2013 1:03 pm
Posts: 109
Location: USA
What's the cable length between the two sites?


Top
 Profile  
Reply with quote  
 Post subject: Re: SSD Performance 3PAR 7200
PostPosted: Thu Jun 09, 2016 3:28 pm 

Joined: Wed Jun 08, 2016 5:00 am
Posts: 8
not much more than about 100m , single mode


Top
 Profile  
Reply with quote  
 Post subject: Re: SSD Performance 3PAR 7200
PostPosted: Thu Jun 09, 2016 4:31 pm 
User avatar

Joined: Fri Jun 21, 2013 1:03 pm
Posts: 109
Location: USA
FC configuration somewhere, latency should be under 1ms at that distance unless you have a faulty nic.


Top
 Profile  
Reply with quote  
 Post subject: Re: SSD Performance 3PAR 7200
PostPosted: Fri Jun 10, 2016 3:26 am 

Joined: Wed Jun 08, 2016 5:00 am
Posts: 8
hi

:roll: i posted you the svt ms of the RC Ports.
i will check the latency whith the checkrclink command .

greetz


Top
 Profile  
Reply with quote  
 Post subject: Re: SSD Performance 3PAR 7200
PostPosted: Fri Jun 10, 2016 3:47 am 

Joined: Wed Jun 08, 2016 5:00 am
Posts: 8
Latency: 0.226 ms
Lost pings: 0 %
Through-put: > 6038088 Kbits/second
FC Invalid Words (ne): 0
FC CRC Errors (ne): 0
FC Invalid Words (fe): 0
FC CRC Errors (fe): 0


Link 0:1:2 is SUITABLE for Remote Copy Use


Latency: 0.232 ms
Lost pings: 0 %
Through-put: > 5840546 Kbits/second
FC Invalid Words (ne): 0
FC CRC Errors (ne): 0
FC Invalid Words (fe): 0
FC CRC Errors (fe): 0


Link 1:1:1 is SUITABLE for Remote Copy Use

but when i do a statport i see this.

10:46:00 06/10/2016 r/w I/O per second KBytes per sec Svt ms IOSz KB
Port D/C Cur Avg Max Cur Avg Max Cur Avg Cur Avg Qlen
0:1:2 Data t 1990 2020 3229 28941 36770 53835 331.99 316.39 14.5 18.2 640
1:1:1 Data t 2387 1596 4811 45625 30253 86901 306.14 400.34 19.1 19.0 639


Top
 Profile  
Reply with quote  
 Post subject: Re: SSD Performance 3PAR 7200
PostPosted: Mon Jun 13, 2016 1:27 am 

Joined: Fri Jun 27, 2014 2:01 am
Posts: 386
Latency of RC port is somewhat special...
When you got nothing to replicate, latency is very high :)
And most of the time this latency is quite high ^^

Here is what i got from an array (7450, 4 nodes, 32 SSD 1.9 TiB):
cli% statport -rcfc
08:25:35 06/13/2016 r/w I/O per second KBytes per sec Svt ms IOSz KB
Port D/C Cur Avg Max Cur Avg Max Cur Avg Cur Avg Qlen
0:2:4 Data t 654 654 654 9969 9969 9969 991.40 991.40 15.2 15.2 640
1:2:3 Data t 860 860 860 13021 13021 13021 662.32 662.32 15.1 15.1 639
2:2:4 Data t 859 859 859 13343 13343 13343 778.43 778.43 15.5 15.5 640
3:2:3 Data t 832 832 832 15741 15741 15741 742.05 742.05 18.9 18.9 640
-------------------------------------------------------------------------------------
4 Data t 3205 3205 52074 52074 781.32 781.32 16.2 16.2 2559
Press the enter key to stop...


Everything is ok there :)


Top
 Profile  
Reply with quote  
 Post subject: Re: SSD Performance 3PAR 7200
PostPosted: Mon Jun 13, 2016 6:59 pm 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1002
Location: Europe
as RitonLaBevue says, RC latency looks normal.

IIRC each RCFC ports has 384 outstanding/waiting read I/Os with a timeout of 6000msec or 10000msec or something like that. For RCFC ports with low or no activity, latency is "incorrectly" reported as sky-high. Just try and create a new RC Target and add some RCFC links without adding any volumes (or adding a volume which doesn't do any writes) and you'll see the timeout and outstanding/waiting I/Os.


However, 50-80MB/s write for a single volume on 8 dedicated SSDs is low. It's hard to give a lot of tips without any more information, but on a general level I would start by doing two things.
1. Understand the IO. (read/write ratio, IOsize, look at statvlun and statvv to understand where the problem is likely to be and verify that MPIO is load-balancing across nodes and ports, is other traffic impacting?)
2. Elimination ..... Try stopping the RC group to see whether RC is related or not... Is this dedupe or not? Is this relevant for just one or multiple servers? What OS and could it be OS/configuration related... etc

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 16 posts ]  Go to page 1, 2  Next


Who is online

Users browsing this forum: Google [Bot] and 29 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt