3PAR Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 14 posts ]  Go to page Previous  1, 2
Author Message
 Post subject: Re: Australian Tax Office, 2 day downtime - 3PAR switchover
PostPosted: Thu Jun 01, 2017 11:04 pm 

Joined: Sun Feb 05, 2017 9:07 pm
Posts: 2
Here what the Tax Commissioner said about the 3PAR incident:

https://www.ato.gov.au/Media-centre/Spe ... y-30-2017/


Top
 Profile  
Reply with quote  
 Post subject: Re: Australian Tax Office, 2 day downtime - 3PAR switchover
PostPosted: Sat Jun 03, 2017 2:39 pm 
Site Admin
User avatar

Joined: Tue Aug 18, 2009 10:35 pm
Posts: 1142
Location: Ft Worth, Texas
Quote:
Preliminary analysis shows the following problems:
the fibre optic cables feeding the SAN were not optimally fitted
disk drives on the SAN had software bugs that made the stored data on the drives inaccessible or unable to be read
some monitoring features were not activated, including a “back-to-base” tool to report operating errors.

The SAN design and configuration meant we had an over emphasis on performance features rather than stability or resilience - a relatively small disk drive failure had a large impact - only 12 of some 800 disk drives failed, but they impacted most ATO systems.
The recovery was slower because some of the recovery tools required were stored on the same SAN that failed.


It's peculiar the order in which they reveal the facts. The details that stick out to me are:

They did not have phone home monitoring configured.
They had 12 drives fail. Unclear if it was all at once, or randomly over a long period of time while not being monitored.
They had parts of the DR on the array.

The drive firmware bug... not enough information to make a call here. Would the bug have been mitigated if phone home was working? Did it cause many drives to fail at once, or was a high rate of failures over a period of time the system was not being monitored? The Field notice on this would be a good read if someone has it.

_________________
Richard Siemers
Storage Admin, Pier 1 Imports
The opinions and positions expressed are my own and do not necessarily reflect those of Pier 1 Imports.


Top
 Profile  
Reply with quote  
 Post subject: Re: Australian Tax Office, 2 day downtime - 3PAR switchover
PostPosted: Thu Jun 08, 2017 7:36 pm 

Joined: Sun Feb 05, 2017 9:07 pm
Posts: 2
ATO has released the full report with much detail of what happened:

https://www.ato.gov.au/uploadedFiles/Co ... port_w.pdf


Top
 Profile  
Reply with quote  
 Post subject: Re: Australian Tax Office, 2 day downtime - 3PAR switchover
PostPosted: Fri Jun 09, 2017 12:21 pm 

Joined: Mon Feb 03, 2014 9:40 am
Posts: 115
Very interesting - 3PAR went to test/dev environment and production data moved to Hitachi VSP :lol:


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 14 posts ]  Go to page Previous  1, 2


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt