Show Posts
1
Server Move and Outage Reports / Re: 11/23/2015 16:45 Down again?
« on: November 24, 2015, 01:42:46 PM »Thanks for the update, Kes!I have some new information, and it's pretty much a case of a series of unfortunate events.
A little background; data is stored on mirrored disks which are then mirrored to another pair which in turn are mirrored to another pair and so on. In addition there are several backup disks which come in to play should one of the main disks fail.
And boy did it fail.
Apparently disc 2 of the primary pair failed so the server called up backup drive A and started rebuilding the mirror with drive 1 and drive A.
At the completion of the rebuild drive 1 failed. Drive A, now the primary drive of the primary pair, called up backup drive B and started rebuilding the mirror.
Midway through the second rebuild drive A failed taking out the primary mirrored pair.
Somewhere in this process one of the controller boards died as well.
Some of you may be wondering why there was no alert of a predictive failure, good question.
Apparently the data center failed to do a firmware update some time back that would have sent the predictive failure alert for all three drive so there was no notice given until things started dying.
FedEx should be delivering the new hardware as I am typing this if they haven't already and it will all be installed as quickly as possible.
The real delay comes from restoring the terabytes of data from tape which takes hours and hours.
As I said, a series of unfortunate events but they are working on it as quickly as they can.
Firmware being upgrade for alerts to be sent sounds odd. Why didn't they implement monitoring scripts that sends alerts when a drive drops from the raid? Seems they are making bs excuses to what happened. I bet a drive dropped from the raid it never got replace, and now that two drives from the array died the whole system went poof!