3PAR Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 13 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: 3PAR/SP/CLI/SSMC connectivity
PostPosted: Tue Aug 28, 2018 10:48 am 

Joined: Mon Jun 02, 2014 1:47 pm
Posts: 33
Location: United States
So my 3PAR SP has been messed up since my upgrade to 3.2.1 MU5. I've got a case open with HPE, but things aren't going well there. Initially, I was able to at least connect to the 3PAR with the management console. I ended up re-imaging the SP, but during the MOB, at the very end of the process, it cannot connect to the 3PAR with some sort of certificate error.

After going around and around with support, last week, one of the last thing HPE support had me do was a 'setsysmgr clearcache' on the array. I still could not get the SP to connect, so they want to dispatch a field engineer. I did nothing more and was out of the office between Thursday and yesterday.

This morning, I tried to connect with the management console, and it won't connect, just like the SP. I tried to connect with CLI, and it never connects. The only thing is, I can still connect to the array with SSH. I've been able to connect to it with SSH all along. I don't understand why the SP, CLI, and management console cannot connect. It wasn't clear what the setsysmgr clearcache command did, but it almost seems to have made things even worse.

Does anyone know why I'd be able to connect via SSH and suddenly, not through the SP, CLI, or management console?


Top
 Profile  
Reply with quote  
 Post subject: Re: 3PAR/SP/CLI/SSMC connectivity
PostPosted: Tue Aug 28, 2018 2:58 pm 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 961
Location: Europe
I'm pretty sure you got this covered, but there could have been an update to SSL/TLS standards at some point, so if your CLI/SP/SSMC wasn't upgraded they might have issues with that ... It sounds extremely far-fetched and I would assume SP was upgraded as a part of the 3PAR OS upgrade and that the SP you re-imaged was also the correct one ...

So my last guess would be some sort of stuck process. SP <-> 3PAR exchange some sort of certificates after initial logon with 3paradm to ensure logins even after password changes. SSMC <-> 3PAR also exchange some certificates but that would be visible as an error when logging on to the SSMC admin interface.

CLI is stupid and the only thing I can think of why this doesn't work while SSH is would be if the CLI version is old or you're using something other than standard credentials when logging on.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: 3PAR/SP/CLI/SSMC connectivity
PostPosted: Wed Aug 29, 2018 4:22 am 

Joined: Wed Nov 09, 2011 12:01 pm
Posts: 344
Things to try;
SSMC - Check the admin console to see if a useful error, might be cert or just login needed. I've had to reboot/restart the SSMC server/service in the past after 3PAR upgrade before it would connect again (might have been a but in earlier SSMC version). Might have to remove the array and reboot SSMC and add again.

CLI - Check it's a matching version to the 3PAR OS, again maybe cert related so look for a folder (I can't remember where) which the CLI keeps certs in and delete the cert for that array.

SSH is what I use rather then CLI, the comma separated table output option is about the only thing I use CLI for as getting stuff out of SSMC is crap compared to the old GUI.

Not sure about the SP issues, not had problems from that end in the past, my guess is that the command you used deleted the SSL cert cache on the array so that it would request/accept new ones from previously known devices. Maybe there was a change in the format of the cert store that invalidated old cached copies.


Top
 Profile  
Reply with quote  
 Post subject: Re: 3PAR/SP/CLI/SSMC connectivity
PostPosted: Wed Aug 29, 2018 10:40 am 

Joined: Mon Jun 02, 2014 1:47 pm
Posts: 33
Location: United States
The problem originated with the SP after the OS install finished. I upgraded the SP first to the compatible version, then the OS. The SP lost connectivity after the OS install. I figured I'd remove and re-add the array on the SP to re-establish communication. It was giving the key exchange error at that time. HPE Support has had me re-image the SP a bunch of times, to no avail. Now they're dispatching someone to come on-site to re-image the SP, which I've already done.

I really feel like the problem is on the array because it's not allowing the SP to communicate with it.


Top
 Profile  
Reply with quote  
 Post subject: Re: 3PAR/SP/CLI/SSMC connectivity
PostPosted: Wed Aug 29, 2018 11:39 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 961
Location: Europe
geddy01 wrote:
I really feel like the problem is on the array because it's not allowing the SP to communicate with it.

I agree.

Also surpriced they don't try to redeploy SP remote. I see this resulting in the problem not fixed and the guy coming onsite is only there for the SP, not looking at the 3PAR.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: 3PAR/SP/CLI/SSMC connectivity
PostPosted: Wed Aug 29, 2018 12:24 pm 

Joined: Mon Jun 02, 2014 1:47 pm
Posts: 33
Location: United States
MammaGutt wrote:
geddy01 wrote:
I really feel like the problem is on the array because it's not allowing the SP to communicate with it.

I agree.

Also surpriced they don't try to redeploy SP remote. I see this resulting in the problem not fixed and the guy coming onsite is only there for the SP, not looking at the 3PAR.


Exactly, which is why I'd love to fix this myself before he gets here. I've already re-deployed the SP multiple times, both physical AND virtual. It can't pass the validation at the end of MOB when it tries to connect with the array. I just can't figure it out. The last thing one of the engineers on the phone had me run was a 'setsysmgr clearcache' command. I think things got even worse after that.

I've tried to recreate the self-signed cert, but that didn't help any. There's something on the array that's "stuck" or something. I've rebooted both nodes separately.

We're a dark site to HPE, so they can't remotely re-deploy the SP. I suspect they'd run into the same issue though, with the number of tries I've made at it.


Top
 Profile  
Reply with quote  
 Post subject: Re: 3PAR/SP/CLI/SSMC connectivity
PostPosted: Wed Aug 29, 2018 3:13 pm 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 961
Location: Europe
geddy01 wrote:
We're a dark site to HPE, so they can't remotely re-deploy the SP. I suspect they'd run into the same issue though, with the number of tries I've made at it.


The engineer which is coming to you will most likely have a plan of action which is to redeploy SP. When that doesn't work he needs to hope that the remote case owner is available for a new action plan. If not then the engineer goes home and probably gets three followup questions when the redeploy didn't work so he needs to go back to try a few things... Again that doesn't work and case owner is still not available..... Two weeks down the line the case is elevated and they need logs which I'm guessing you can't provide since you are a dark site and here the story goes :)

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: 3PAR/SP/CLI/SSMC connectivity
PostPosted: Thu Aug 30, 2018 4:08 am 

Joined: Wed Nov 09, 2011 12:01 pm
Posts: 344
Yeah it might take a while with no direct support link, I think we didn't go beyond MU3 on 3.2.1 before going to 3.2.2 but I had a look at the MU5 release notes and it does seem to replace a lot of the SSL/SSH packages so wouldn't be surprised if something linked to that caused an issue. Maybe double check all your package versions against the expected list in the release notes. Any post MU5 updates related to network side?

Also out of interest as it's 3.2.1 does the old IMC GUI fail too?


Top
 Profile  
Reply with quote  
 Post subject: Re: 3PAR/SP/CLI/SSMC connectivity
PostPosted: Thu Aug 30, 2018 6:49 am 

Joined: Mon Jun 02, 2014 1:47 pm
Posts: 33
Location: United States
ailean wrote:
Yeah it might take a while with no direct support link, I think we didn't go beyond MU3 on 3.2.1 before going to 3.2.2 but I had a look at the MU5 release notes and it does seem to replace a lot of the SSL/SSH packages so wouldn't be surprised if something linked to that caused an issue. Maybe double check all your package versions against the expected list in the release notes. Any post MU5 updates related to network side?

Also out of interest as it's 3.2.1 does the old IMC GUI fail too?


We're at 3.2.1 MU5. I've tried the old IMC as well. That also will not connect.

Out of curiosity, does anyone know - when you run through the MOB when you re-deploy the SP, it gives you the option to load a backed up key/cert from a previous version. We have four arrays and never once has anyone asked us to backup a key/cert for anything. Is this cert the same self-signed cert from the array when you run 'showcert'?


Top
 Profile  
Reply with quote  
 Post subject: Re: 3PAR/SP/CLI/SSMC connectivity
PostPosted: Thu Aug 30, 2018 7:51 am 

Joined: Wed Nov 09, 2011 12:01 pm
Posts: 344
I've not built an SP directly but have watched 3PAR/HP/HPE engineers do it when we've had SPs swapped out or the boot disk replaced. I don't remember them bothering to copy anything (disk being dead obviously not possible) to the new disk, normally a fresh install and type in all the details again with same IP address. It's been a while but I think they managed to do it all from the SP console, with no need to direct connect to the array.
But obviously as you have more stuff not connecting then the SP it looks more of an array issue (assuming nothing blocking ports at the network side as was working before upgrade).


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 13 posts ]  Go to page 1, 2  Next


Who is online

Users browsing this forum: MSN [Bot] and 25 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt