3PAR Users Group
https://3parug.com/

Australian Tax Office, 2 day downtime - 3PAR switchover
https://3parug.com/viewtopic.php?f=25&t=2315
Page 1 of 2

Author:  Darking [ Tue Dec 13, 2016 1:42 pm ]
Post subject:  Australian Tax Office, 2 day downtime - 3PAR switchover

Seems like The Australian Tax Office was ready to switch over from their old EMC arrays, and something went horribly wrong.

http://www.theregister.co.uk/2016/12/13 ... o_offline/

Author:  turbogizzmo [ Tue Dec 13, 2016 6:03 pm ]
Post subject:  Re: Australian Tax Office, 2 day downtime - 3PAR switchover

Been following this story......I find it hard to believe. Two 3par arrays both corrupted and its the 3par's fault?

My .02 is its a virus or cryptoware and they were using remote copy to a hot site that then replicated the "corrupt" data to the recovery site. (replication is NOT backup just FYI)

Again just my .02 but I have had a 8k crash (only 3par in 8 year to go hard down) but never in a way that lost all the data and then replicated the issue to another array.......

But for ATO sake they better be right tossing HPE under the bus or there could be hell to pay.

Author:  Richard Siemers [ Fri Dec 16, 2016 7:59 am ]
Post subject:  Re: Australian Tax Office, 2 day downtime - 3PAR switchover

This will be an interesting story to watch. 3PAR has migrated a lot of EMC arrays successfully, this migration tool is not new or unvetted... I think odds are that this is human error, the question will probably boil down to if the human at fault was a ATO employee or a HPe employee.

Author:  MuZ [ Thu Jan 05, 2017 7:43 pm ]
Post subject:  Re: Australian Tax Office, 2 day downtime - 3PAR switchover

Anyone have an update outside of ATO's press releases ?

I work for a Government department down the road from ATO. The internal (non HPe) staff that I've talked to only seem to only know rumors with zero technical insight.
We're a HPe shop and our HPe account contacts are very tight lipped, playing the "I don't know as I don't work on that account" card.

Author:  Richard Siemers [ Fri Jan 06, 2017 1:09 am ]
Post subject:  Re: Australian Tax Office, 2 day downtime - 3PAR switchover

I think both sides are tight lipped right now, sounds like lawyers are involved.

Posts online say there was data corruption, and the bad data was replicated to the secondary system. I suspect the ATO told the world that it was a SAN hardware failure BEFORE a qualified root cause analysis was performed... if it turned out this was caused by customer error, not SAN failure, then that could explain why no one wants to talk about it anymore =)

I am trying to think of the most likely scenario how luns can get corrupted... perhaps a non-clustered file system (NTFS) on a LUN that was accidently shared to a 2nd non-clustered host that tried to mount the volume while they were in use.

Author:  thatirishguy [ Fri Feb 03, 2017 12:49 pm ]
Post subject:  Re: Australian Tax Office, 2 day downtime - 3PAR switchover

Anymore news on this a month later? Would be interested to know the root cause of this one.

Author:  nsnidanko [ Fri Feb 17, 2017 1:45 pm ]
Post subject:  Re: Australian Tax Office, 2 day downtime - 3PAR switchover

Another update from the register:
https://www.theregister.co.uk/2017/02/1 ... ce_outage/

Looks like HPE is blaming faulty PD firmware. Who has more details on the subject?

Author:  ben.g [ Wed Apr 26, 2017 12:39 am ]
Post subject:  Re: Australian Tax Office, 2 day downtime - 3PAR switchover

I heard that the root cause was a faulty cable that had been crimped incorrectly or similar would you believe!

As a result data was slowly being corrupted on one of the nodes or something. An issue then arose which might have been unrelated but the "failover" didn't occur as it should and the techs didn't know why so they rebooted the whole array and this then killed everything as the corrupt data came into effect and was replicated throughout the whole array or something.

The ATO didn't have any backups either :shock:

I say "or something" above as I only heard a very brief rendition of what occurred. It could be a fair way off the mark, except the faulty cable was definitely the root cause.

Edit: Just found this - https://www.theregister.co.uk/2017/02/2 ... ser_error/

Looks like the faulty cable was due to someone moving a shelf while in production and wrecking the cabling!!

Author:  Richard Siemers [ Wed Apr 26, 2017 5:46 pm ]
Post subject:  Re: Australian Tax Office, 2 day downtime - 3PAR switchover

What? User error? Government agencies never make mistakes!

Author:  nsnidanko [ Wed May 31, 2017 1:49 pm ]
Post subject:  Re: Australian Tax Office, 2 day downtime - 3PAR switchover

Further development on this story:
https://www.theregister.co.uk/2017/05/3 ... reporting/

Does anyone know HPE advisory for disks affected by firmware bug?

Page 1 of 2 All times are UTC - 5 hours
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/