3PAR Users Group
https://3parug.com/

Australian Tax Office, 2 day downtime - 3PAR switchover
https://3parug.com/viewtopic.php?f=25&t=2315
Page 2 of 2

Author:  kurionvale [ Thu Jun 01, 2017 11:04 pm ]
Post subject:  Re: Australian Tax Office, 2 day downtime - 3PAR switchover

Here what the Tax Commissioner said about the 3PAR incident:

https://www.ato.gov.au/Media-centre/Spe ... y-30-2017/

Author:  Richard Siemers [ Sat Jun 03, 2017 2:39 pm ]
Post subject:  Re: Australian Tax Office, 2 day downtime - 3PAR switchover

Quote:
Preliminary analysis shows the following problems:
the fibre optic cables feeding the SAN were not optimally fitted
disk drives on the SAN had software bugs that made the stored data on the drives inaccessible or unable to be read
some monitoring features were not activated, including a “back-to-base” tool to report operating errors.

The SAN design and configuration meant we had an over emphasis on performance features rather than stability or resilience - a relatively small disk drive failure had a large impact - only 12 of some 800 disk drives failed, but they impacted most ATO systems.
The recovery was slower because some of the recovery tools required were stored on the same SAN that failed.


It's peculiar the order in which they reveal the facts. The details that stick out to me are:

They did not have phone home monitoring configured.
They had 12 drives fail. Unclear if it was all at once, or randomly over a long period of time while not being monitored.
They had parts of the DR on the array.

The drive firmware bug... not enough information to make a call here. Would the bug have been mitigated if phone home was working? Did it cause many drives to fail at once, or was a high rate of failures over a period of time the system was not being monitored? The Field notice on this would be a good read if someone has it.

Author:  kurionvale [ Thu Jun 08, 2017 7:36 pm ]
Post subject:  Re: Australian Tax Office, 2 day downtime - 3PAR switchover

ATO has released the full report with much detail of what happened:

https://www.ato.gov.au/uploadedFiles/Co ... port_w.pdf

Author:  nsnidanko [ Fri Jun 09, 2017 12:21 pm ]
Post subject:  Re: Australian Tax Office, 2 day downtime - 3PAR switchover

Very interesting - 3PAR went to test/dev environment and production data moved to Hitachi VSP :lol:

Page 2 of 2 All times are UTC - 5 hours
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/