HP3par 7200 18 disks degraded how to replace

coolirc · **Joined:** Sat Jul 28, 2018 12:20 pm **Posts:** 24

Hello all

i'm new to this community , we have an HP3par 7200 which we bought in 2014 now we have alarms for 18 degraded disks and i am searching for an advice i've found on the net how to replace procedure but they don't talk about data on the disks or something. so i'm asking how can i replace the 18 degraded disks , one by one or all a time , what commands should i perform before proceeding should i consider a down time or a better time for replacing the disks during weekend etc ? what will happen to the data on the disks , should i consider a backup or something ? will the actual data on the disks be lost?
should i consider putting the esxi into maintenance mode and shutting down the vm's ?
thanks for help.

here's the issued command

Code:

HP3PAR_7200 cli% showpd -failed -degraded
                             --Size(MB)--- ----Ports----
Id CagePos Type RPM State      Total  Free A      B      Capacity(GB)
1:0:0   FC    15 degraded  278528  4096 0:0:1  1:0:1*          300
1:1:0   FC    15 degraded  278528  2048 0:0:1* 1:0:1           300
1:2:0   FC    15 degraded  278528  3072 0:0:1  1:0:1*          300
1:3:0   FC    15 degraded  278528  3072 0:0:1* 1:0:1           300
1:4:0   FC    15 degraded  278528  4096 0:0:1  1:0:1*          300
1:5:0   FC    15 degraded  278528  3072 0:0:1* 1:0:1           300
1:6:0   FC    15 degraded  278528  4096 0:0:1  1:0:1*          300
1:7:0   FC    15 degraded  278528  4096 0:0:1* 1:0:1           300
1:8:0   FC    15 degraded  278528  5120 0:0:1  1:0:1*          300
1:9:0   FC    15 degraded  278528  4096 0:0:1* 1:0:1           300
1:10:0  FC    15 degraded  278528  4096 0:0:1  1:0:1*          300
1:11:0  FC    15 degraded  278528  4096 0:0:1* 1:0:1           300
1:12:0  NL     7 degraded  923648     0 0:0:1  1:0:1*         1000
1:13:0  NL     7 degraded  923648     0 0:0:1* 1:0:1          1000
1:14:0  NL     7 degraded  923648     0 0:0:1  1:0:1*         1000
1:15:0  NL     7 degraded  923648     0 0:0:1* 1:0:1          1000
1:16:0  NL     7 degraded  923648     0 0:0:1  1:0:1*         1000
1:17:0  NL     7 degraded  923648     0 0:0:1* 1:0:1          1000
---------------------------------------------------------------------
total                     8884224 45056
HP3PAR_7200 cli%

MammaGutt · **Posted:** Sun Jul 29, 2018 12:50 am

What does checkhealth -svc -detail say?

Looks to me like all drives in cage1 has a problem so I'm guessing the problem is with the cage and not the 18 drives.

coolirc · **Joined:** Sat Jul 28, 2018 12:20 pm **Posts:** 24

MammaGutt wrote:

What does checkhealth -svc -detail say?

Looks to me like all drives in cage1 has a problem so I'm guessing the problem is with the cage and not the 18 drives.

hmm i think you are right

Code:

HP3PAR_7200 cli% checkhealth -svc -detail
Checking alert
Checking ao
Checking cabling
Checking cage
Checking dar
Checking date
Checking file
Checking ld
Checking host
Checking license
Checking network
Checking node
Checking pd
Checking pdch
Checking port
Checking qos
Checking rc
Checking snmp
Checking task
Checking vlun
Checking vv
Checking sp
Component -------------------Description-------------------- Qty
Alert     New alerts                                           4
Cabling   Wrong I/O module or port                             2
host      Hosts not seen by multiple nodes                     9
Host      Host ports not configured for virtual port support   4
Network   Too few working admin network connections            1
PD        PDs that are degraded                               18
QoS       Unable to check QoS                                  1

Component --Identifier-- ----------------------------------------------------------------------------------------Description----------------------------------------------------------------------------------------
Alert     sw_cp:1:FC_r5  CPG 1 (FC_r5) could not grow with its normal grow parameters.-The following parameters were used:  createald -wait 0 -cpsd FC_r5 -ssz 6 -ha mag -t r5 -p -devtype NL -n tp-1-sd-2 -sz 8192 
Alert     sw_sysmgr      Total FC raw space usage at 6449G (above 95% of total 6528G)                                                                                                                               
Alert     sw_sysmgr      Total NL raw space usage at 21228G (above 95% of total 21648G)  

Alert     sw_os          An Update is Available                                                                                                                                                                     
Cabling   cage1          Cable in (cage1, I/O 0, DP-1) should be in (cage1, I/O 1, DP-1)                                                                                                                            
Cabling   cage1          Cable in (cage1, I/O 1, DP-1) should be in (cage1, I/O 0, DP-1)                                                                                                                            
host      SRV_ESXi01_112 Host is not seen by multiple nodes, only seen from node 1                                                                                                                                  
host      SRV_ESX06_92   Host is not seen by multiple nodes, only seen from node 1                                                                                                                                  
host      SRV_ESX3_68    Host is not seen by multiple nodes, only seen from node 1                                                                                                                                  
host      SRV_ESX2_107   Host is not seen by multiple nodes, only seen from node 1                                                                                                                                  
host      SRV_ESX04_106  Host is not seen by multiple nodes, only seen from node 1                                                                                                                                  
host      SRV_ESXi11_67  Host is not seen by multiple nodes, only seen from node 1                                                                                                                                  
host      SRV_ESXi12_66  Host is not seen by multiple nodes, only seen from node 1                                                                                                                                  
host      SRV_ESXi09_70  Host is not seen by multiple nodes, only seen from node 1                                                                                                                                  
host      SRV_ESXi09_74  Host is not seen by multiple nodes, only seen from node 1                                                                                                                                  
Host      Port:1:1:1     Port WWN not found on FC Fabric attached to Port:0:1:1                                                                                                                                     
Host      Port:1:1:2     Port WWN not found on FC Fabric attached to Port:0:1:2                                                                                                                                     
Host      Port:0:1:1     Port WWN not found on FC Fabric attached to Port:1:1:1                                                                                                                                     
Host      Port:0:1:2     Port WWN not found on FC Fabric attached to Port:1:1:2                                                                                                                                     
Network   --             Node 1 has no admin network link detected                                                                                                                                                  
PD        disk:18        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:19        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:20        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:21        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:22        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:23        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:24        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:25        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:26        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:27        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:28        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:29        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:30        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:31        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:32        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:33        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:34        Degraded States: Invalid_connection                                                                                                                                                        
PD        disk:35        Degraded States: Invalid_connection                                                                                                                                                        
QoS       --             Unable to check QoS - This system is not licensed for System Reporter features                                                                                                             
HP3PAR_7200 cli% 

i think the cables connections are incorrect ,
Cabling cage1 Cable in (cage1, I/O 0, DP-1) should be in (cage1, I/O 1, DP-1)
Cabling cage1 Cable in (cage1, I/O 1, DP-1) should be in (cage1, I/O 0, DP-1)

MammaGutt · **Posted:** Sun Jul 29, 2018 10:39 am

With no guarantee I think you can do the following:

Move cable from cage1, I/O 1, DP-1 to cage1, I/O 1, DP-2
Move cable from cage1, I/O 0, DP-1 to cage1, I/O 1, DP-1
Move cable from cage1, I/O 1, DP-2 to cage1, I/O 1, DP-1

Between each step do showpd -p (I think this is showpd -path) to verify that all PDs have two paths before moving the next cable.

Finish with admithw and a new checkhealth.

coolirc · **Joined:** Sat Jul 28, 2018 12:20 pm **Posts:** 24

MammaGutt wrote:

With no guarantee I think you can do the following:

Move cable from cage1, I/O 1, DP-1 to cage1, I/O 1, DP-2
Move cable from cage1, I/O 0, DP-1 to cage1, I/O 1, DP-1
Move cable from cage1, I/O 1, DP-2 to cage1, I/O 1, DP-1

Between each step do showpd -p (I think this is showpd -path) to verify that all PDs have two paths before moving the next cable.

Finish with admithw and a new checkhealth.

Hello thanks for your answer but i still did not figure out what is what .

where is cage 1 and where is I/O 0 and where is I/O 1 .

DP1 and DP2 is already mentionned on the cage but still .

thanks

MammaGutt · **Posted:** Sun Jul 29, 2018 1:44 pm

coolirc wrote:

MammaGutt wrote:

With no guarantee I think you can do the following:

Move cable from cage1, I/O 1, DP-1 to cage1, I/O 1, DP-2
Move cable from cage1, I/O 0, DP-1 to cage1, I/O 1, DP-1
Move cable from cage1, I/O 1, DP-2 to cage1, I/O 1, DP-1

Between each step do showpd -p (I think this is showpd -path) to verify that all PDs have two paths before moving the next cable.

Finish with admithw and a new checkhealth.

Hello thanks for your answer but i still did not figure out what is what .

where is cage 1 and where is I/O 0 and where is I/O 1 .

DP1 and DP2 is already mentionned on the cage but still .

thanks

Cage number should be visable thru a LED in the front. Node cage is always cage0.

To find I/O 0 and I/O 1, look all the way to the left or right on the back on the disk cages (between PSUs and I/O modules). You will see a red tab with 0 and green tab with 1. That is I/O 0 and I/O 1. Same with the nodes. When I look at your pictures now, you have labeled your cables correctly (red on both ends for one cable and green on both cables for the other) but you have connected the green cable to the red node and the red cable to the green node..... So the best would be to change the ports on the node end, but I've never done that so I can't say anything as to what types or errors you might run into....

If you have a spare 3PAR backend cable you could use that so you get the "right color coded cable" in the right node and cage .... But I would just suggest to do what I've suggested above and re-label the cables with spare labels.

coolirc · **Joined:** Sat Jul 28, 2018 12:20 pm **Posts:** 24

hello

i re-pluged my cables according to this figure now the number of degraded disks decreased to 6

would you like me to proceed with your figure or stay as i am ?

Code:

HP3PAR_7200 cli% showpd -path
                         ---------Paths---------      
Id CagePos Type -State-- A           B           Order
 0 0:0:0   FC   normal   1:0:1       0:0:1       1/0  
 1 0:1:0   FC   normal   1:0:1       0:0:1       0/1  
 2 0:2:0   FC   normal   1:0:1       0:0:1       1/0  
 3 0:3:0   FC   normal   1:0:1       0:0:1       0/1  
 4 0:4:0   FC   normal   1:0:1       0:0:1       1/0  
 5 0:5:0   FC   normal   1:0:1       0:0:1       0/1  
 6 0:6:0   FC   normal   1:0:1       0:0:1       1/0  
 7 0:7:0   FC   normal   1:0:1       0:0:1       0/1  
 8 0:8:0   FC   normal   1:0:1       0:0:1       1/0  
 9 0:9:0   FC   normal   1:0:1       0:0:1       0/1  
0:10:0  FC   normal   1:0:1       0:0:1       1/0  
0:11:0  FC   normal   1:0:1       0:0:1       0/1  
0:12:0  NL   normal   1:0:1       0:0:1       1/0  
0:13:0  NL   normal   1:0:1       0:0:1       0/1  
0:14:0  NL   normal   1:0:1       0:0:1       1/0  
0:15:0  NL   normal   1:0:1       0:0:1       0/1  
0:16:0  NL   normal   1:0:1       0:0:1       1/0  
0:17:0  NL   normal   1:0:1       0:0:1       0/1  
1:0:0   FC   normal   1:0:2       0:0:2       1/0  
1:1:0   FC   normal   1:0:2       0:0:2       0/1  
1:2:0   FC   normal   1:0:2       0:0:2       1/0  
1:3:0   FC   normal   1:0:2       0:0:2       0/1  
1:4:0   FC   normal   1:0:2       0:0:2       1/0  
1:5:0   FC   normal   1:0:2       0:0:2       0/1  
1:6:0   FC   normal   1:0:2       0:0:2       1/0  
1:7:0   FC   normal   1:0:2       0:0:2       0/1  
1:8:0   FC   normal   1:0:2       0:0:2       1/0  
1:9:0   FC   normal   1:0:2       0:0:2       0/1  
1:10:0  FC   normal   1:0:2       0:0:2       1/0  
1:11:0  FC   normal   1:0:2       0:0:2       0/1  
1:12:0  NL   normal   1:0:2       0:0:2       1/0  
1:13:0  NL   normal   1:0:2       0:0:2       0/1  
1:14:0  NL   normal   1:0:2       0:0:2       1/0  
1:15:0  NL   normal   1:0:2       0:0:2       0/1  
1:16:0  NL   normal   1:0:2       0:0:2       1/0  
1:17:0  NL   normal   1:0:2       0:0:2       0/1  
0:18:0  NL   normal   1:0:1       0:0:1       1/0  
0:19:0  NL   normal   1:0:1       0:0:1       0/1  
0:20:0  NL   normal   1:0:1       0:0:1       1/0  
0:21:0  NL   normal   1:0:1       0:0:1       0/1  
0:22:0  NL   normal   1:0:1       0:0:1       1/0  
0:23:0  NL   normal   1:0:1       0:0:1       0/1  
1:18:0  NL   degraded 1:0:2\0:0:1 0:0:2\1:0:1 0/1  
1:19:0  NL   degraded 1:0:2\0:0:1 0:0:2\1:0:1 1/0  
1:20:0  NL   degraded 1:0:2\0:0:1 0:0:2\1:0:1 0/1  
1:21:0  NL   degraded 1:0:2\0:0:1 0:0:2\1:0:1 1/0  
1:22:0  NL   degraded 1:0:2\0:0:1 0:0:2\1:0:1 0/1  
1:23:0  NL   degraded 1:0:2\0:0:1 0:0:2\1:0:1 1/0  
------------------------------------------------------
total                                              

Code:

HP3PAR_7200 cli% admithw
Checking for drive table upgrade packages
Checking nodes...

Checking volumes...

Checking system LDs...

Checking ports...

Checking state of disks...
The following disks are NOT in an acceptable state:
Id CagePos Type -State-- --Detailed_State--
42 1:18:0  NL   degraded Invalid_connection
43 1:19:0  NL   degraded Invalid_connection
44 1:20:0  NL   degraded Invalid_connection
45 1:21:0  NL   degraded Invalid_connection
46 1:22:0  NL   degraded Invalid_connection
47 1:23:0  NL   degraded Invalid_connection
-------------------------------------------
 6 total                                   

Enter c to continue despite this issue or q to quit and fix the issue manually:
c


Checking cabling...

Checking cage firmware...

Checking if this is an upgrade that added new types of drives...

Checking for disks to admit...
0 disks admitted


Checking admin volume...
Admin volume exists.

Checking if logging LDs need to be created...
No new logging LDs need to be created

Checking if preserved data LDs need to be created...
No new preserved data LDs need to be created

Checking if system scheduled tasks need to be created...

Checking if the rights assigned to extended roles need to be updated...
No need to update extended roles rights.

Rebalancing and adding FC spares...
FC spare chunklets rebalanced; number of FC spare chunklets increased by 0 for a total of 544.
Rebalancing and adding NL spares...
NL spare chunklets rebalanced; number of NL spare chunklets increased by 0 for a total of 1260.
Rebalancing and adding SSD spares...
No SSD PDs present


System Reporter data volume exists.


Checking system health...
Checking alert
Checking cabling
Checking cage
Checking dar
Checking date
Checking host
Checking ld
Checking license
Checking network
Checking node
Checking pd
Checking port
Checking rc
Checking snmp
Checking task
Checking vlun
Checking vv
Component -------------------Description-------------------- Qty
Alert     New alerts                                           4
host      Hosts not seen by multiple nodes                     9
Host      Host ports not configured for virtual port support   4
LD        LDs with reduced availability                        2
Network   Too few working admin network connections            1
PD        PDs that are degraded                                6


admithw has completed.

Code:

HP3PAR_7200 cli% checkhealth
Checking alert
Checking cabling
Checking cage
Checking dar
Checking date
Checking host
Checking ld
Checking license
Checking network
Checking node
Checking pd
Checking port
Checking rc
Checking snmp
Checking task
Checking vlun
Checking vv
Component -------------------Description-------------------- Qty
Alert     New alerts                                           4
host      Hosts not seen by multiple nodes                     9
Host      Host ports not configured for virtual port support   4
LD        LDs with reduced availability                        2
Network   Too few working admin network connections            1
PD        PDs that are degraded                                6

HP3PAR_7200 cli% 

MammaGutt · **Posted:** Mon Jul 30, 2018 2:14 am

Oh lord.... Those 8 PDs wasn't in your first print screens. I assumed all PDs in cage1 was degraded.....

I'm going to take a wild guess on this one.

At one point in time everything was good and great, and cage1 had 16 PDs. At some later point in time the 3PAR was probably moved and recabled, resulting in the incorrect cabling and the 16 PDs complaining about the cabling being incorrect/changed.
Nothing was done and at a later time, 8 additional PDs was added to cage1 when the cabling was incorrect, and those 8 PDs assumed that everything was good (considering every upgrade procedure should include a step where you check that you have a health system prior to doing any changes). So now have 16 PDs expecting the cabling to be the correct one and complaining about it being wrong, and 8 PDs assuming everything is good. So then you recable the 3PAR to make it correct so the 16 PDs go "all OK" while the last 8 is now complaining that the correct cabling is not the cabling it is expecting.

So .... the only way I know would fix this, is to empty one PD at a time, completely remove it with the correct set of commands and re-admit it. The bad news is that you have 0 chunklets free on your NL drives so you can't empty one out.....

It could be that this could also be fixed by using servicemag and trick the system into thinking one drive has failed and replacing (and re-admitting) it with the same drive, but I wouldn't even know where to start on that ... so others need to shed some light on that if it is possible.....

Might be a good time to get in touch with HPE Support.

coolirc · **Joined:** Sat Jul 28, 2018 12:20 pm **Posts:** 24

Hello Thanks for your reply , i think the cause of the 0 free chunklets would be that i created a raw volumes and exported to vmware vcenter so i think i should i can move the recently created NL volume and then move the vms and data from that volume and replace the disks one by one and see.

MammaGutt · **Posted:** Mon Jul 30, 2018 6:07 am

If you can free up space, than you should try that.

You don't have to replace them as they are not broken, but you need to remove them and re-add them...

setpd ldalloc off <DiskID>
movepdtospare -f -vacate -nowait <DiskID>
showpdch -mov (too see status/what is left, if it doesn't complete, you can just just tunesys cpg <NL CPG>)
removespare PDID:a
dismisspd <DiskID>

Remove PD
Reinstall PD
admithw

But as mentioned there might be a smoother way to do this with servicemag so others might give some advise here as well.

HPE Storage Users Group

HP3par 7200 18 disks degraded how to replace

Who is online