HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 12 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Will ebay disks work?
PostPosted: Mon Mar 18, 2019 8:18 am 

Joined: Mon Mar 18, 2019 7:36 am
Posts: 4
Hey guys and girls,

I just started out a new job as a sysadmin for a company that uses 3pars. I never worked with them, but I'm starting to learn really fast, and I am already growing quite fond of 3par.

I currently have an issue with one of our 8200's. Apparently there was no monitoring in place, so nobody saw the issue before it was too late.

Short story: cpg full, volume read-only, and no way to fix.

Long storage: We have a pool with 46x6TB NL disks, with a raid6 CPG on top of that (178TB max usable). On that CPG are 15 x 16TB volumes (thin provisioned) that are used by a single Windows VM. Using windows storage spaces those 15 volumes are aggregated into a single 240TB volume used by a file server. Nobody thought about the thin provisioning, until the file server filled up, and then the 178TB limit was hit..

I see two possible paths to get out of this situation: 1. remove data and shrink, 2. add disks.

Option 1: We want to remove data again, but we can't. Since one of the VV's is in the failed state (it cannot allocate from the CPG), the volume is reported to Windows in the failed state, and is read-only. So we cannot remove any data from the volume :(

Option 2, adding disks, is the best idea of course. But according to our supplier, HPE will take 3-4 weeks to deliver some disks (!).

One of the idea's I'm playing with, is buying two disks from ebay (pn 823123-001), to have the problem fixed within a week and then replacing them when the original new hp disks arrive.

Is this a possibility? Can I just plug in any compatible disk and will it work? Or will I run into licensing issues and such?


Top
 Profile  
Reply with quote  
 Post subject: Re: Will ebay disks work?
PostPosted: Mon Mar 18, 2019 9:13 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
AxisNL wrote:
One of the idea's I'm playing with, is buying two disks from ebay (pn 823123-001), to have the problem fixed within a week and then replacing them when the original new hp disks arrive.

Is this a possibility? Can I just plug in any compatible disk and will it work? Or will I run into licensing issues and such?


The perfect answer it depends.

If you can find 3PAR-branded drives (which 823123-001 seems to be) it "should work". However you probably will have difficulties getting support on those drives.

It also depends on what type of licenses you got. If you have the old licenses, 46 drives and 8200 you are 2 drive licenses away from hitting the drive cap (so you need licenses). If you have the new All-Inclusive it doesn't count drives anymore. "showlicense" in CLI will say if it counts disks or not.

One thing that you've not looked into is (temporarily) reduce the amount of spare in the system.

I'm guessing you're doing default sparing which will have 2 drives worth of spares. It is really not ideal to reduce it, but from the looks of it you're way beyond the point of what is recommended. Download this guide ( https://support.hpe.com/hpsc/doc/public ... =c04204279 )and look at removespare command. This will rather instantly remove capacity from spare pool and allocate it to your capacity pool.

removespare 0:a will remove all spares on PD 0. Do that for all drives and you should get ~10TB of additional space to help you survive until you can source more drives and licenses. Once you add more drives it will auto-trigger "admithw" which will re-create spares .... Once new drives have balanced you should run "admithw" again to ensure you have the correct spare capacity again.

Keep in mind, doing this with the spare will put you in a position where you ONLY have RAID redundancy, so you need to replace failed drives quickly until you've gotten more drives!

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: Will ebay disks work?
PostPosted: Mon Mar 18, 2019 9:58 am 

Joined: Mon Mar 18, 2019 7:36 am
Posts: 4
MammaGutt wrote:
AxisNL wrote:
One thing that you've not looked into is (temporarily) reduce the amount of spare in the system.

I'm guessing you're doing default sparing which will have 2 drives worth of spares. It is really not ideal to reduce it, but from the looks of it you're way beyond the point of what is recommended. Download this guide ( https://support.hpe.com/hpsc/doc/public ... =c04204279 )and look at removespare command. This will rather instantly remove capacity from spare pool and allocate it to your capacity pool.

removespare 0:a will remove all spares on PD 0. Do that for all drives and you should get ~10TB of additional space to help you survive until you can source more drives and licenses. Once you add more drives it will auto-trigger "admithw" which will re-create spares .... Once new drives have balanced you should run "admithw" again to ensure you have the correct spare capacity again.

Keep in mind, doing this with the spare will put you in a position where you ONLY have RAID redundancy, so you need to replace failed drives quickly until you've gotten more drives!

You are my hero! I disabled some spare blocks on a few drives, and this allowed the CPG and the volume to grow again. Now the server admin can log in and delete some data again, so we don't run into this issue again the next weeks. New drivers are ordered, I'll be sure to run the admithw command again to check the spares!


Top
 Profile  
Reply with quote  
 Post subject: Re: Will ebay disks work?
PostPosted: Mon Mar 18, 2019 10:24 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
Not sure how storage spaces work, but with windows mount points unmap is not forwarded to the 3PAR. Only way to reclaim deleted capacity is running a script which writes zero blocks to empty parts of the disk. Also not sure how/if that would impact storage spaces.

But yeah, tl:dr, it might not help simply deleting files in VM (assuming hyper-v)

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: Will ebay disks work?
PostPosted: Tue Mar 19, 2019 1:44 am 

Joined: Tue Aug 25, 2015 3:01 am
Posts: 49
As recommended do a "showlicense" and check your licensing model.
I got caught out once by this from moving disks between two of our 3par 8200's.
plugged the 8 new disks in and disk license limit was hit doh!

I purchased an 8 disk license to add the disks but it took approx 2 weeks for the reseller to explain to HPE what i was trying to do.

but now we have transition to all inclusive licensing.


Top
 Profile  
Reply with quote  
 Post subject: Re: Will ebay disks work?
PostPosted: Tue Mar 19, 2019 6:24 am 

Joined: Mon Mar 18, 2019 7:36 am
Posts: 4
MammaGutt wrote:
Not sure how storage spaces work, but with windows mount points unmap is not forwarded to the 3PAR. Only way to reclaim deleted capacity is running a script which writes zero blocks to empty parts of the disk. Also not sure how/if that would impact storage spaces.

But yeah, tl:dr, it might not help simply deleting files in VM (assuming hyper-v)

Thanks, I'm looking into trimming and unmap commands in windows!


Top
 Profile  
Reply with quote  
 Post subject: Re: Will ebay disks work?
PostPosted: Tue Mar 19, 2019 7:30 am 

Joined: Tue Sep 11, 2012 1:58 am
Posts: 17
Location: Mechelen, Belgium
another consideration:
we create a 1TB thick provisioned virtual volume on each of our 3PARs, as a sort of reserve.
If we ever do run into space allocation issues, we can delete such a volume quickly and create a little bit of buffer.


Top
 Profile  
Reply with quote  
 Post subject: Re: Will ebay disks work?
PostPosted: Tue Mar 19, 2019 7:32 am 

Joined: Mon Mar 18, 2019 7:36 am
Posts: 4
nicDM77 wrote:
another consideration:
we create a 1TB thick provisioned virtual volume on each of our 3PARs, as a sort of reserve.
If we ever do run into space allocation issues, we can delete such a volume quickly and create a little bit of buffer.

Smart!


Top
 Profile  
Reply with quote  
 Post subject: Re: Will ebay disks work?
PostPosted: Tue Mar 19, 2019 7:34 am 

Joined: Tue Sep 11, 2012 1:58 am
Posts: 17
Location: Mechelen, Belgium
AxisNL wrote:
Thanks, I'm looking into trimming and unmap commands in windows!

Not sure if this is still applicable with "modern age" windows servers, but sdelete might be worth looking at (again, if still applicable :?: )

https://docs.microsoft.com/en-us/sysinternals/downloads/sdelete


Top
 Profile  
Reply with quote  
 Post subject: Re: Will ebay disks work?
PostPosted: Thu Mar 21, 2019 11:33 am 

Joined: Tue Jan 22, 2019 2:06 pm
Posts: 26
Location: Calgary, AB
AxisNL wrote:
nicDM77 wrote:
another consideration:
we create a 1TB thick provisioned virtual volume on each of our 3PARs, as a sort of reserve.
If we ever do run into space allocation issues, we can delete such a volume quickly and create a little bit of buffer.

Smart!


But also should not be needed if you are properly monitoring and maintaining your storage.

By chance have you force a comptactcpg -f -pat * run on the 3pars?

Even though that runs on its own, I have gained 20TB yesterday from running it on production...


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 12 posts ]  Go to page 1, 2  Next


Who is online

Users browsing this forum: Google [Bot] and 41 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt