3PAR Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 14 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Australian Tax Office, 2 day downtime - 3PAR switchover
PostPosted: Tue Dec 13, 2016 1:42 pm 

Joined: Wed Mar 05, 2014 10:55 am
Posts: 73
Seems like The Australian Tax Office was ready to switch over from their old EMC arrays, and something went horribly wrong.

http://www.theregister.co.uk/2016/12/13 ... o_offline/


Top
 Profile  
Reply with quote  
 Post subject: Re: Australian Tax Office, 2 day downtime - 3PAR switchover
PostPosted: Tue Dec 13, 2016 6:03 pm 

Joined: Tue Nov 25, 2014 4:48 pm
Posts: 20
Been following this story......I find it hard to believe. Two 3par arrays both corrupted and its the 3par's fault?

My .02 is its a virus or cryptoware and they were using remote copy to a hot site that then replicated the "corrupt" data to the recovery site. (replication is NOT backup just FYI)

Again just my .02 but I have had a 8k crash (only 3par in 8 year to go hard down) but never in a way that lost all the data and then replicated the issue to another array.......

But for ATO sake they better be right tossing HPE under the bus or there could be hell to pay.


Top
 Profile  
Reply with quote  
 Post subject: Re: Australian Tax Office, 2 day downtime - 3PAR switchover
PostPosted: Fri Dec 16, 2016 7:59 am 
Site Admin
User avatar

Joined: Tue Aug 18, 2009 10:35 pm
Posts: 1141
Location: Ft Worth, Texas
This will be an interesting story to watch. 3PAR has migrated a lot of EMC arrays successfully, this migration tool is not new or unvetted... I think odds are that this is human error, the question will probably boil down to if the human at fault was a ATO employee or a HPe employee.

_________________
Richard Siemers
Storage Admin, Pier 1 Imports
The opinions and positions expressed are my own and do not necessarily reflect those of Pier 1 Imports.


Top
 Profile  
Reply with quote  
 Post subject: Re: Australian Tax Office, 2 day downtime - 3PAR switchover
PostPosted: Thu Jan 05, 2017 7:43 pm 

Joined: Thu Jan 05, 2017 7:34 pm
Posts: 3
Anyone have an update outside of ATO's press releases ?

I work for a Government department down the road from ATO. The internal (non HPe) staff that I've talked to only seem to only know rumors with zero technical insight.
We're a HPe shop and our HPe account contacts are very tight lipped, playing the "I don't know as I don't work on that account" card.


Top
 Profile  
Reply with quote  
 Post subject: Re: Australian Tax Office, 2 day downtime - 3PAR switchover
PostPosted: Fri Jan 06, 2017 1:09 am 
Site Admin
User avatar

Joined: Tue Aug 18, 2009 10:35 pm
Posts: 1141
Location: Ft Worth, Texas
I think both sides are tight lipped right now, sounds like lawyers are involved.

Posts online say there was data corruption, and the bad data was replicated to the secondary system. I suspect the ATO told the world that it was a SAN hardware failure BEFORE a qualified root cause analysis was performed... if it turned out this was caused by customer error, not SAN failure, then that could explain why no one wants to talk about it anymore =)

I am trying to think of the most likely scenario how luns can get corrupted... perhaps a non-clustered file system (NTFS) on a LUN that was accidently shared to a 2nd non-clustered host that tried to mount the volume while they were in use.

_________________
Richard Siemers
Storage Admin, Pier 1 Imports
The opinions and positions expressed are my own and do not necessarily reflect those of Pier 1 Imports.


Top
 Profile  
Reply with quote  
 Post subject: Re: Australian Tax Office, 2 day downtime - 3PAR switchover
PostPosted: Fri Feb 03, 2017 12:49 pm 

Joined: Tue Nov 08, 2016 6:17 am
Posts: 3
Anymore news on this a month later? Would be interested to know the root cause of this one.


Top
 Profile  
Reply with quote  
 Post subject: Re: Australian Tax Office, 2 day downtime - 3PAR switchover
PostPosted: Fri Feb 17, 2017 1:45 pm 

Joined: Mon Feb 03, 2014 9:40 am
Posts: 115
Another update from the register:
https://www.theregister.co.uk/2017/02/1 ... ce_outage/

Looks like HPE is blaming faulty PD firmware. Who has more details on the subject?


Top
 Profile  
Reply with quote  
 Post subject: Re: Australian Tax Office, 2 day downtime - 3PAR switchover
PostPosted: Wed Apr 26, 2017 12:39 am 

Joined: Sun Apr 23, 2017 5:45 pm
Posts: 12
I heard that the root cause was a faulty cable that had been crimped incorrectly or similar would you believe!

As a result data was slowly being corrupted on one of the nodes or something. An issue then arose which might have been unrelated but the "failover" didn't occur as it should and the techs didn't know why so they rebooted the whole array and this then killed everything as the corrupt data came into effect and was replicated throughout the whole array or something.

The ATO didn't have any backups either :shock:

I say "or something" above as I only heard a very brief rendition of what occurred. It could be a fair way off the mark, except the faulty cable was definitely the root cause.

Edit: Just found this - https://www.theregister.co.uk/2017/02/2 ... ser_error/

Looks like the faulty cable was due to someone moving a shelf while in production and wrecking the cabling!!


Top
 Profile  
Reply with quote  
 Post subject: Re: Australian Tax Office, 2 day downtime - 3PAR switchover
PostPosted: Wed Apr 26, 2017 5:46 pm 
Site Admin
User avatar

Joined: Tue Aug 18, 2009 10:35 pm
Posts: 1141
Location: Ft Worth, Texas
What? User error? Government agencies never make mistakes!

_________________
Richard Siemers
Storage Admin, Pier 1 Imports
The opinions and positions expressed are my own and do not necessarily reflect those of Pier 1 Imports.


Top
 Profile  
Reply with quote  
 Post subject: Re: Australian Tax Office, 2 day downtime - 3PAR switchover
PostPosted: Wed May 31, 2017 1:49 pm 

Joined: Mon Feb 03, 2014 9:40 am
Posts: 115
Further development on this story:
https://www.theregister.co.uk/2017/05/3 ... reporting/

Does anyone know HPE advisory for disks affected by firmware bug?


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 14 posts ]  Go to page 1, 2  Next


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt