11/23/2015 16:45 Down again? UP 1322 28 Nov 2105!

[cockedandglocked]

A few questions that I know nobody here can answer:

-Why did they not have redundant hardware in place? No self-respecting computer company doesn't have a plan (and spare hardware) in place to prevent a hardware failure from taking out company operations. "they are trying to expedite replacement parts"... in other words, they only had 1 of whatever broke.

-If the storage disk array is what blew up, and they are "working to restore from backups", does that mean we lose hours, days, weeks, or months of data?

-Which IT guy there is getting canned?


I fully understand that hardware failures happen, I've experienced lots of them myself... but anyone who halfway knows what they're doing, would never experience a catastrophic hardware failure like this. Redundant systems should be in place, where one a hardware bank goes offline, catches fire, or whatever, the system kicks itself over to the redundant hardware and operations continue. Google, Amazon, Facebook, and many others never experience downtime, and I'm pretty sure it isn't because their hardware never fails.

Maybe time for CG to switch hosts?

[Son]

A few questions that I know nobody here can answer:

-Why did they not have redundant hardware in place? No self-respecting computer company doesn't have a plan (and spare hardware) in place to prevent a hardware failure from taking out company operations. "they are trying to expedite replacement parts"... in other words, they only had 1 of whatever broke.

-If the storage disk array is what blew up, and they are "working to restore from backups", does that mean we lose hours, days, weeks, or months of data?

-Which IT guy there is getting canned?


I fully understand that hardware failures happen, I've experienced lots of them myself... but anyone who halfway knows what they're doing, would never experience a catastrophic hardware failure like this. Redundant systems should be in place, where one a hardware bank goes offline, catches fire, or whatever, the system kicks itself over to the redundant hardware and operations continue. Google, Amazon, Facebook, and many others never experience downtime, and I'm pretty sure it isn't because their hardware never fails.

Maybe time for CG to switch hosts?

"I find your logic disturbing."  -Said the owners of the provider...


and (unrelated to to this particular situation) every leftist ever.

[MZ]

wow, I thought it was only my computer misbehaving again.........

[Piper Cub]

I'm glad it's really down and not just me! I thought it might've gotten blocked or my IP been banned. That happened to me on another forum, someone was spamming and when the admin went to block the spammer they had a typo in the IP address and I got blocked instead.

[grampz]

Oh my. WTH... I'm IN!

[swalt]

A few questions that I know nobody here can answer:

-Why did they not have redundant hardware in place? No self-respecting computer company doesn't have a plan (and spare hardware) in place to prevent a hardware failure from taking out company operations. "they are trying to expedite replacement parts"... in other words, they only had 1 of whatever broke.

-If the storage disk array is what blew up, and they are "working to restore from backups", does that mean we lose hours, days, weeks, or months of data?

-Which IT guy there is getting canned?


I fully understand that hardware failures happen, I've experienced lots of them myself... but anyone who halfway knows what they're doing, would never experience a catastrophic hardware failure like this. Redundant systems should be in place, where one a hardware bank goes offline, catches fire, or whatever, the system kicks itself over to the redundant hardware and operations continue. Google, Amazon, Facebook, and many others never experience downtime, and I'm pretty sure it isn't because their hardware never fails.

Maybe time for CG to switch hosts?

If its down for a few days I can imagine their customers losing a good chunk of change.  How many lawsuits will follow?   

[BAJ475]

First, does anyone have an estimated time when .net will be back up?
Although I do not know just how it works, when I tried to do a DNS lookup on Calguns.net it could not return a numeric IP address.  So it appears to me the the problem is more than just the servers that Calguns.net runs on and extends to DNS servers.  Anyone know more?

[Red-Osier77]

three days

[tuolumnejim]

I bet they are waiting for Black Friday sales to pick up the new hardware at steep discount.

Newegg FTW! 

[Librarian]

First, does anyone have an estimated time when .net will be back up?

Post 134 in this thread:

... we are expecting services to restore by roughly 6am PST, November 26th.

[1bulletBarney]

I just checked ODOT (Oregon Dept of Transportation) and the site is down. My mom lives in Bend and I was thinking about going up there, I can check road conditions in real time. This outage may being affecting .gov sites and that would not be good...

[Wnick308]

So with the main site down does that mean we should get out and go shooting? Or maybe just go outside?

[jctheguy]

Frys?!?! 

[ninety]

So its true?

I was smacking my router around, thinking it was a local issue. Then I thought to try some other sites.

Yes, I do this all day at work so my troubleshooting skills should be better, but I am getting older and forgetful. 

LoL Me too

rebooted modem ,router released IP , checked settings , ran virus scan..

[Red-Osier77]

did obamacare sign ups start yesterday

[Son]

So with the main site down does that mean we should get out and go shooting? Or maybe just go outside?

Hitting the range tonight for the regular CG Santa Clara monthly meet up.  6 PM.  If you're local c'mon down.  If you're not local, c'mon down!

[Killer Bee]

First, does anyone have an estimated time when .net will be back up?

TWO WEEKS!!!

[Killer Bee]

<snip
-If the storage disk array is what blew up, and they are "working to restore from backups", does that mean we lose hours, days, weeks, or months of data?
<snip>

no worries, NSA has a copy of everything.. maybe they'll share?

[edwardm]

I'm with you.  Problems and their consequences are often not entirely avoidable.  But you can mitigate outages by preparing (and spending money) for outages. 

Assuming they're using a SAN from a vendor like NetApp, IBM, EMC, or Hitachi, they should have a support contract.  FFS, when just a disk dies on one of my SANs, the vendor is willing to send a warm body to swap the SOB.  And a good support contract also means parts are on the way before you even have to ask.  If it really hit the fan, there would be a team of people driving gear to one of my facilities, pulling it out of labs at their corporate HQ, or otherwise doing whatever they had to for a return to service. 

With the nature of the outage (customer data, but also their DNS servers appear to be offline, and thus no mailhost/MX records), it looks like they stored everything on the SAN (bad move #1), had no secondary site (bad move #2), have no disaster recovery (DR) plan (stupid stupid move #1), and basically put zero thought into recovering from a worst-case scenario. 

I'd still put $5 on "everything lives in virtual machines and every virtual machine lives on one SAN".  Derp. 

A few questions that I know nobody here can answer:

-Why did they not have redundant hardware in place? No self-respecting computer company doesn't have a plan (and spare hardware) in place to prevent a hardware failure from taking out company operations. "they are trying to expedite replacement parts"... in other words, they only had 1 of whatever broke.

-If the storage disk array is what blew up, and they are "working to restore from backups", does that mean we lose hours, days, weeks, or months of data?

-Which IT guy there is getting canned?


I fully understand that hardware failures happen, I've experienced lots of them myself... but anyone who halfway knows what they're doing, would never experience a catastrophic hardware failure like this. Redundant systems should be in place, where one a hardware bank goes offline, catches fire, or whatever, the system kicks itself over to the redundant hardware and operations continue. Google, Amazon, Facebook, and many others never experience downtime, and I'm pretty sure it isn't because their hardware never fails.

Maybe time for CG to switch hosts?

[FP562]

Wow, Nov 26, that is pretty bad hardware failure .

[AR15barrels]

It was all the micro aggression in Randall banned thread...I know it.

And I didn't even get to see it...
I got a couple text messages about it while sitting on the beach in mexico (thank you sprint for free coverage in mexico now) and I responded back to my guys to go ahead and take the site down since I didn't need it for a few days.
I should have asked them to screen grab that thread and text it to me before they did...

[AR15barrels]

and already five posts! Post whore 

You newbies don't know what post whoring is...

[AR15barrels]

is randall banned from here

Not yet, but I can try!

[LowThudd]

So Mexico is your alibi on the seerver going down? Good one!

[Red-Osier77]

is randall banned from here

Not yet, but I can try! whats up man

[Pinto]

So Mexico is your alibi on the seerver going down? Good one!

No, I think he said being banned was the reason he ordered the site "taken down". The Mexico part is just puffing.

[readysetgo]

It was all the micro aggression in Randall banned thread...I know it.

And I didn't even get to see it...
I got a couple text messages about it while sitting on the beach in mexico (thank you sprint for free coverage in mexico now) and I responded back to my guys to go ahead and take the site down since I didn't need it for a few days.
I should have asked them to screen grab that thread and text it to me before they did...

I knew it!  Well...might as well come clean before you see it...I was advocating in favor of the ban (more or less)...catching a lot of flack...you're a popular guy, why (?), who knows...  

[josey wales]

Wonder if Melvino is behind this.  His computer was always malfunctioning, my friend.

[skyhawk]

I'm with you.  Problems and their consequences are often not entirely avoidable.  But you can mitigate outages by preparing (and spending money) for outages. 

Assuming they're using a SAN from a vendor like NetApp, IBM, EMC, or Hitachi, they should have a support contract.  FFS, when just a disk dies on one of my SANs, the vendor is willing to send a warm body to swap the SOB.  And a good support contract also means parts are on the way before you even have to ask.  If it really hit the fan, there would be a team of people driving gear to one of my facilities, pulling it out of labs at their corporate HQ, or otherwise doing whatever they had to for a return to service. 

With the nature of the outage (customer data, but also their DNS servers appear to be offline, and thus no mailhost/MX records), it looks like they stored everything on the SAN (bad move #1), had no secondary site (bad move #2), have no disaster recovery (DR) plan (stupid stupid move #1), and basically put zero thought into recovering from a worst-case scenario. 

I'd still put $5 on "everything lives in virtual machines and every virtual machine lives on one SAN".  Derp. 

I'd agree that nearly everything is probably running from VM and shared storage. But they alluded to a cascade hardware failure and also having to do restores.  It could be their chassis cooling failed, causing several disks to die off - more than the RAID level they use could tolerate, necessitating a complete restore after they get equipment spares. It could have been the data center cooling too.  I'm just spitballing based on the limited info we have.  If it was just a SAN switch, a restore would not be required.

I am a believer in your own DNS being hosted by a 3rd party.  At least that way when things go tango uniform in your data center, you can quickly point A records at a backup site to at least give clients status updates. This is a lot faster than updating your DNS servers at the registrar and waiting for propagation, just to give status updates or receive mail.  It also allows you to have a lower priority MX record always published and pointing at a backup store & forward mail service, to be used when you have problems like this.

They should also revive their twitter account, it is a great way to give status updates.

[Ubermcoupe]

First, does anyone have an estimated time when .net will be back up?

Post 134 in this thread:

... we are expecting services to restore by roughly 6am PST, November 26th.

Dang - seems so far off.

The Great-Great CGN outage of 2015.