HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 6 posts ] 
Author Message
 Post subject: Using the adaptive queue depth algorithm in EXS
PostPosted: Thu Feb 11, 2010 10:14 am 

Joined: Thu Feb 11, 2010 9:53 am
Posts: 5
Is any one using VMware's adaptive queue depth algorithm (AQDA) with 3PAR storage? I am setting up a new environment and looking for guidance.

Today we have eight vSphere 4 (ESXi) servers connected to a two node F400. At this time we do not plan to connect any other servers to the F400 nor do we plan to connect any other storage to the ESX servers. All the ESX servers have Emulex LPe11002 HBAs.

Given that design would you use AQDA? If so, what would you set the Disk.QFullSampleSize and Disk.QFullThreshold value to? If you would not use AQDA what would set the HBA's target throttle value to?

In either case what would you set the Disk.SchedNumReqOutstanding value to?

Also, is there any change I need to make on the HBA (i.e. firmware settings) when changing the ESX queue depth.

Thanks

_________________
Steve Remsing
Sr. Research Engineer
The Dow Chemical Company


Top
 Profile  
Reply with quote  
 Post subject: Re: Using the adaptive queue depth algorithm in EXS
PostPosted: Thu Feb 11, 2010 3:06 pm 
Site Admin
User avatar

Joined: Tue Aug 18, 2009 10:35 pm
Posts: 1328
Location: Dallas, Texas
Hello SRemsing,

The safe answer is "It Depends".

Adaptive queue throttling can occur from either PORT congestion OR LUN congestion. So depending on how your environment uses the storage, and how busy it is will determine if you need it or not.

When the number of QUEUE FULL or BUSY conditions reaches the QFullSampleSize value, the LUN queue depth reduces to half of the original value. When the number of good status conditions received reaches the QFullThreshold value, the LUN queue depth increases one at a time.

For 3Par arrays its recommended that QFullSampleSize is set to 32, I would suggest QFull
Threshold to be set at 8. No HBA settings required that I know of.

Disk.SchedNumReqOutstanding is a per virtual machine setting (CORRECTION: its a system wide setting that effects Queue Depth per virtual machine --RS ) .

If you havn't already, take a look at: http://kb.vmware.com/kb/1008113

_________________
Richard Siemers
The views and opinions expressed are my own and do not necessarily reflect those of my employer.


Top
 Profile  
Reply with quote  
 Post subject: Re: Using the adaptive queue depth algorithm in EXS
PostPosted: Fri Feb 12, 2010 7:40 am 

Joined: Thu Feb 11, 2010 9:53 am
Posts: 5
Thank you for the quick reply.

Your explanation of the QFullSampleSize conflicts with others that I have seen or been given. Some have suggested that the QFullSampleSize is the per LUN queue depth. I have not been able to find anything from VMWare that defines these parameters. Do you have a reference? Assuming the QFullSampleSize is as you described (and I believe it is), what is the per LUN queue depth that the server will use and how do I set it?

Also, Disk.SchedNumReqOutstanding is not a per VM setting, it is set on the ESX host. Documentation from either 3PAR or VMware (I made a note of it but can't find the original source) suggested setting it to the same as the queue depth. If I understand this parameter correctly that could be a bad decision if you have a single VM spike in I/O traffic. Couldn't the other VMs on the same ESX host face I/O resource starvation?

_________________
Steve Remsing
Sr. Research Engineer
The Dow Chemical Company


Top
 Profile  
Reply with quote  
 Post subject: Re: Using the adaptive queue depth algorithm in EXS
PostPosted: Fri Feb 12, 2010 8:05 am 

Joined: Thu Feb 11, 2010 9:53 am
Posts: 5
I just reread http://kb.vmware.com/kb/1008113 and found it contains the following description of Disk.QFullSampleSize and Disk.QFullThreshold

Quote:
When enabled, it works by examining the completed I/O operations that fit within a window of Disk.QFullSampleSize operations. If the count of queue full errors is equal to or more than the value of Disk.QFullThreshold within that window, then the queue depth is throttled.


Also, I found the reference to the Disk.SchedNumReqOutstanding pararmeter. It is on page 3 of VMware's Scalable Storage Performance document. This was written for ESX 3.5 and not has not been updated for 4.0 (yet).

_________________
Steve Remsing
Sr. Research Engineer
The Dow Chemical Company


Top
 Profile  
Reply with quote  
 Post subject: Re: Using the adaptive queue depth algorithm in EXS
PostPosted: Fri Feb 12, 2010 11:03 am 
Site Admin
User avatar

Joined: Tue Aug 18, 2009 10:35 pm
Posts: 1328
Location: Dallas, Texas
sremsing wrote:
Some have suggested that the QFullSampleSize is the per LUN queue depth. I have not been able to find anything from VMWare that defines these parameters. Do you have a reference?


http://kb.vmware.com/kb/1010922 and http://kb.vmware.com/kb/1267 might be the references you are looking for. QFullSampleSize is not the physical HBA queue depth its a sampling window, queue depth is set from the ESX console (see http://www-dl.emulex.com/support/vmware/734/vmware.pdf ) and is the target parameter being throttled by the algorithm.

I did misunderstand Disk.SchedNumReqOutstanding, you are correct it is a system wide setting. However it sets the per VM lun queue depth globally (not individually like I assumed). I agree with you, I would prefer to have this set to a fraction of the physical HBAs queue depth based on the ratio of HBA ports to VMs. The documented defaults may assume you have multiple active fibre ports. If you have 4 active HBAs then thats a total of 4 times the physical queue depth. Having the Disk.SchedNumReqOutstanding setting equal to the Emulex disk queue setting would result in 1 VM being limited to 25% of the available hardware.

_________________
Richard Siemers
The views and opinions expressed are my own and do not necessarily reflect those of my employer.


Top
 Profile  
Reply with quote  
 Post subject: Re: Using the adaptive queue depth algorithm in EXS
PostPosted: Mon Feb 15, 2010 2:40 pm 

Joined: Thu Feb 11, 2010 9:53 am
Posts: 5
Richard,

Thank you for your replies. We are going to test the adaptive queue depth algorithm with the following values:

Emulex HBA driver lpfc_lun_queue_depth = 32, 64, 128 (we are going to test each)
Disk.QFullSampleSize = 32
Disk.QFullThreshold = 8
Disk.SchedNumReqOutstanding = half the per LUN queue depth on the Emulex HBA

_________________
Steve Remsing
Sr. Research Engineer
The Dow Chemical Company


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 6 posts ] 


Who is online

Users browsing this forum: No registered users and 45 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt