HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 6 posts ] 
Author Message
 Post subject: 4-node 7450c - POR STATE Leds and internal error register
PostPosted: Fri Apr 02, 2021 8:49 pm 

Joined: Mon Mar 29, 2021 10:17 am
Posts: 4
Hi 3parugers,

Please, if I could borrow your 3par combined wisdom to help me bring life back to a 3par 4-Node 7450c system I purchased 2nd hand. One node (node-3) boots ok. I can connect to the console at least and I can login as root and console.

The other 3 nodes (nodes 0,1,2) do nothing, except blink the fault and hotplug leds. No console. Never does any external led flash green.

Internally, on the node mainboard, beneath where an expansion card sits, there are 5 additional diagnostic leds, labeled as POR STATE. I can see these by removing the expansion card and connecting the node without its cover. Only leds 0 and 1 light green on the dead nodes (0,1,2). On node 3, leds 0,1,3,4 light green.

My thinking is, that for 3/4 nodes to apparently all fail simultaneously, all having the same symptom, but still functioning diagnostics, all saying the same. Either something catastrophic has occurred, damaging 3/4 nodes at the same time in the same way, which seems pretty unlikely. Or, some event has caused the entire cluster to panic, placing 3/4 nodes into some sort of defensive state in order to avoid potential, actual failure.

Could you offer any insight into what this tells me, if this may be the case, and how I might recover these 3 nodes? I am hoping there be a procedure to clear or reset the nodes internal error register somehow.

Many thanks in advance.


Top
 Profile  
Reply with quote  
 Post subject: Re: 4-node 7450c - POR STATE Leds and internal error registe
PostPosted: Sat Apr 03, 2021 2:36 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
My first thought is when did these fail? Have they ever worked for you?

Just to verify you are getting no console output at all on the three nodes not working?

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: 4-node 7450c - POR STATE Leds and internal error registe
PostPosted: Sat Apr 03, 2021 3:36 am 

Joined: Mon Mar 29, 2021 10:17 am
Posts: 4
I bought "As-Is" a 3par rack containing a 4-Node 7450c ( E7X62A ), having 4x E7X87-63001 Controller modules, 8 x HP M6710 ( QR490A ), and a DL320e SP. No disks were supplied. I am new to 3par. My understanding has been gathered entirely over the past week, trying to get this kit operational.

I've never seen this kit operating, so I may have purchased 3 dead nodes, and 1 maybe ok / maybe not node. I have cabled up just the 4 Nodes. Power, network, io, and powered the lot on. And what I describe here, is how this I received this kit.

Node 3, power on, almost immediately flashes some green. Leds for network, fibre (onboard + 4x 8gb expansion), all flash some green and look healthy. IO connector leds appear healthy, although saying no link. With a console cable connected, I get output and can logon.

Nodes 0,1,2, Power on, never do leds flash, except for Hotplug and Fault, which flash together at the same time. The controller will power on, flash those two leds a few times, then go dark for a few seconds, then come back on blinking forever. With a console cable connected, I get zip. Messing about with the console, unplugging, plugging back in, etc. sometimes I get a scrambled character. One time, I got a quarter page of scrambled characters. Progress I thought, more is better than less right?

Internally on these nodes, there is 5x led for POR STATE (Power-On-Reset), which show leds 0,1 green. Node 3, these same leds show 0,1,3,4 green.

I have noticed that on the enclosure for Nodes 0 and 1, both battery leds show failed. Enclosure for Nodes 2 and 3, appear ok, however, initially, the battery led indicated warning / sw fixable fault, but that seems to have cleared.

I have swapped the ssd-boot disk between nodes. Node 3 boots ok using ssd-boot from other nodes, but not vice-versa. I have connected Node 3's ssd-boot disk to my linux workstation via SATA cable, and noticed the log partition was 100%. Errors were throughout recent history regarding unavailable space to write logs, so I deleted a 36Gb system log archive.

I have tried swapping all parts from Node 3 with another Node. Node 3 always boots ok, other nodes do not. Always same issue. blinking hotplug and fault.

I have tried removing / swapping the button battery. Currently I have one node with button battery removed, I am hoping that eventually whatever it was keeping charged will reset, and maybe clear whatever is stopping it from posting.

My next step I suppose is to examine the logs on the other nodes.


Top
 Profile  
Reply with quote  
 Post subject: Re: 4-node 7450c - POR STATE Leds and internal error registe
PostPosted: Sat Apr 03, 2021 6:16 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
Sounds like dead nodes to me.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: 4-node 7450c - POR STATE Leds and internal error registe
PostPosted: Sat Apr 03, 2021 7:24 am 

Joined: Mon Mar 29, 2021 10:17 am
Posts: 4
I am thinking much the same.

I am now reflecting upon how I came to acquire this machine. I bought it as a complete system from auction. No doubt, the previous owner went out of business when their super redundant, mission critical, best in class, 240 All-Flash disk array went belly up.

Can't blame the IT decision makers really. Who would ever expect 3/4 nodes of an enterprise storage system failing simultaneously, for no apparent reason, when they are specifically designed entirely around that NOT happening.

Perhaps, if I can get another 12 nodes, and 24 disks enclosures, at a 3 in 4 failure rate, I might put a running system together.

And at that fail rate, we should expect to see more of these come available, as the liquidators move in.


Top
 Profile  
Reply with quote  
 Post subject: Re: 4-node 7450c - POR STATE Leds and internal error registe
PostPosted: Sat Apr 03, 2021 9:38 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
My experience is that old storage systems tend to have latent hardware issues displaying themselves when power is removed and/or systems are transported.

It wouldn't be surpriced if the system was working when it was shutdown. For you to buy a system without drives gives you and/or the broker a chance to verify if the system is really operational until you get a hold of drives.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 6 posts ] 


Who is online

Users browsing this forum: No registered users and 16 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt