Show Posts
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
Any ETA as to when .net may be back up?
Hopefully soon but I'm stepping back from this one and letting the guys at GeoVario/Professional Edge handle it.
I think I've done enough damage for today.
C'mon peeps! If you can't read Calguns, I know you have time to read through the thread.
The link to the status page at GeoVario has been posted plenty of times.
http://www.professionaledgellc.com/Quote11/28/2015 - 1:07am PST - Update 15: Another 400+ websites are now back online, unfortunately for us (well me that is) - we chose to remove the software load-balancers and go single server for tonight - note the data in file format (never again will we trust snapshot or backup formats) is however in 3 places now, that we can login and see the files - anyway - this means that we had to click at least 1 time on a configuration button on every single website, so while that was happening I also looked at all your websites, some look great - no problems at all... MOST have a problem you can find after clicking on a few pages, and SOME just don't work at all because they use Wordpress, Joomla, Drupal, etc.. which needs all the base software files working (i.e. when you see a Server 500 Error page, go click on the Web-Options button and then click on the Error_Log for the domain - and voilla you'll see something like the ".htacess" file is corrupted or the index.php is missing, or the backup folder is missing, etc..) - for this last class of website error that is giving an Error 500 - just reupload the same software files you did before and your website will return. Or open a ticket and tell us what software you need installed and we'll do it for you.No, really, this twitch in my eyebrow is normal ...
11/28/2015 - 6:11am PST - Update 16: Another WebCluster back online, who needs sleep right? well... me.
Librarian,
The Error 500 translation, along with the narrative explanation of the sleepless web hoster, seem to indicate that the website backbone is now operational and that the individual websites must now reload their software to restore function to the sites.
That makes it seem that the proverbial ball is now in Calguns court to make the site work.
Am I missing something here?
No, I'm not an addict or anything like that...............
Does Calguns have a copy of the software?
If Calguns doesn't have a copy of the required software, and the sleepless website hoster is waiting for Calguns to re-install the software, then we're going to be down for a very, very, long time.
C'mon peeps! If you can't read Calguns, I know you have time to read through the thread.
The link to the status page at GeoVario has been posted plenty of times.
http://www.professionaledgellc.com/Quote11/28/2015 - 1:07am PST - Update 15: Another 400+ websites are now back online, unfortunately for us (well me that is) - we chose to remove the software load-balancers and go single server for tonight - note the data in file format (never again will we trust snapshot or backup formats) is however in 3 places now, that we can login and see the files - anyway - this means that we had to click at least 1 time on a configuration button on every single website, so while that was happening I also looked at all your websites, some look great - no problems at all... MOST have a problem you can find after clicking on a few pages, and SOME just don't work at all because they use Wordpress, Joomla, Drupal, etc.. which needs all the base software files working (i.e. when you see a Server 500 Error page, go click on the Web-Options button and then click on the Error_Log for the domain - and voilla you'll see something like the ".htacess" file is corrupted or the index.php is missing, or the backup folder is missing, etc..) - for this last class of website error that is giving an Error 500 - just reupload the same software files you did before and your website will return. Or open a ticket and tell us what software you need installed and we'll do it for you.No, really, this twitch in my eyebrow is normal ...
11/28/2015 - 6:11am PST - Update 16: Another WebCluster back online, who needs sleep right? well... me.
Librarian,
The Error 500 translation, along with the narrative explanation of the sleepless web hoster, seem to indicate that the website backbone is now operational and that the individual websites must now reload their software to restore function to the sites.
That makes it seem that the proverbial ball is now in Calguns court to make the site work.
Am I missing something here?
No, I'm not an addict or anything like that...............
So much for a midnight update.
Anyone ready to start taking bets on the over/under of this being resolved before Monday? I'm starting to have my doubts.
It seems inevitable. Gonna be soon.
But I like the lack of a language filter and the boobies (some of which seem to have disappeared though).
I guess there is a line, thin as it may be.
This is like when your parents go away for a week and tell you to behave, but you have raging blowout beer-pong, stripper parties until your parents come home and the hammer comes down.
Kinda like that.
QuoteProfessional Edge / GeoVario Hosting, OFFLINE
Unfortunately as you are seeing this page, you have come during the time of the worst hardware failure in our company's history.
The website you are expecting, or the email account you were trying to reach went offline at roughly 4:45pm PST 11/23/2015; after we experienced a cascade hardware problem that affected our Storage Area Network disk arrays.
At this time, all static files, email accounts and most of our web and application servers are offline, database servers were not affected, but without application and web servers the content is not reachable - we are actively working to restore service, however between expiditing replacement parts and time to restore from backups we are expecting services to restore by roughly 6am PST, November 26th.
We are very aware this has been a huge problem for everyone, we apologize greatly and continue to work to restore service as quickly as possible.
UPDATES: 2:00pm PST 11/25/2015 - all failed hardware has been replaced and all logical data has finished restore process. NameServer (DNS) services are restored. As name servers were needed for the WebSites that were not affected, some sites are back online, however all email is still OFFLINE.
CRITICAL!!!: 6:48pm PST 11/25/2015 - a time drift/snapshot bug in the various backups has made full recovery impossible. We highly suggest if you are using an email client such as Thunderbird, Outlook, etc.. that you use the "Local Folders", "Personal Folders", etc.. to make a copy of any EMAIL in your IMAP inbox. A large number of existing emails in the IMAP inboxes have been affected by the outage, and as email is not something that we do daily backups of - we can not restore any email in your inbox. Please create a "Local or Personal Folder" and drag and drop all email from your IMAP inbox into a local folder stored on your machine. NOTE if you use POP3 you should already be storing data locally on your machine, and no action is required.
UPDATES: 11:30pm PST 11/25/2015 - integrity checks of the restored storage snapshots are still running, and still finding lots of errors due to the time drift of the snapshots. We suggest you locate copies of the static files (pictures, app software, etc) as you may need to re-upload some content to the webservers. Again - the database servers are not affected at all - so all posts, messages, purchase history, etc.. are not affected, but the forum software, or uploaded images may be affected - we'll know more once the integrity checks finish.
UPDATES: 4:00am PST 11/26/2016 - integrity checks finished - which ended up producing more questions as to how we arrived where we are; in short we have unusable backups, lots of filesystem corruption, and now sets of servers that even though they can now access working hardware refused to run at speeds faster than a computer from the late 1960s . As we have already been planning the next phase of our services to be built using Amazon AWS Cloud Servers, we have spun up several servers there and are moving critical services there as quickly as possible. We hate to say it - but you likely won't have your website back in the next 2 hours - unless you already have your site back now. Next update will be at 12 noon PST.
Sincerely,
Professional Edge LLC
800-208-5510
http://www.professionaledgellc.com
No Bueno my friends. No Bueno at all.
Folks:
To all the armchair IT gurus, Kestryll and the hosting provider don't have much experience in this field and can really use your help. Please put all of your suggestions in writing on the back of $100 bills. If one bill is not enough for your ideas, please use multiple bills and mail them all to him for review, evaluation, and consideration.
Until then, put the smartphone/tablet/laptop down, go outside your mom's basement, enjoy a breath of fresh air, clean your guns, go out for a walk, maybe go out to the range...
You get the idea...
Don't know, but this is Calguns we are talking about!
Seriously though, meno377, why don't you go to Amazon EC2 and check prices if you are that concerned about price.
A few questions that I know nobody here can answer:
-Why did they not have redundant hardware in place? No self-respecting computer company doesn't have a plan (and spare hardware) in place to prevent a hardware failure from taking out company operations. "they are trying to expedite replacement parts"... in other words, they only had 1 of whatever broke.
-If the storage disk array is what blew up, and they are "working to restore from backups", does that mean we lose hours, days, weeks, or months of data?
-Which IT guy there is getting canned?
I fully understand that hardware failures happen, I've experienced lots of them myself... but anyone who halfway knows what they're doing, would never experience a catastrophic hardware failure like this. Redundant systems should be in place, where one a hardware bank goes offline, catches fire, or whatever, the system kicks itself over to the redundant hardware and operations continue. Google, Amazon, Facebook, and many others never experience downtime, and I'm pretty sure it isn't because their hardware never fails.
Maybe time for CG to switch hosts?
Computer Engineer here, I agree 100%!
They are more than likely running Calguns.net on a virtual machine (on a hardware server with other clients virtuals) in a datacenter.
They should have had all the virtual machines on another set of "hardware machines" in a DIFFERENT data center on the East Coast so that the other "cluster" takes over if ANYTHING fails!
FOR PETE SAKE, THIS IS CALGUNS WE ARE TALKING ABOUT!! : )
Firmware being upgrade for alerts to be sent sounds odd. Why didn't they implement monitoring scripts that sends alerts when a drive drops from the raid? Seems they are making bs excuses to what happened. I bet a drive dropped from the raid it never got replace, and now that two drives from the array died the whole system went poof!
That was my thought too, it doesn't sound like an unfortunate series of coincidences, it sounds like they weren't diligent and are now paying the price. Coming up with excuses that are extremely unlikely, but we can't prove it didn't happen, to try to save face and not sound incompetent. Did they not realize that their system was incapable of warning them that a drive failure was detected? Does that mean that, until now, they've never head a drive failure? Heck, my server at work only has 4 disks, and I have to replace a dead one at least once every 2 years. And guess what, I even get an email when one goes bad
I've known Michael the owner of PE/GV for over a decade, he has at one point donated two years of hosting free back when Calguns was small and needed time to grow. He's been a Calgunner for most of the time Calguns has existed and has helped dozens of Calgunners and 2A orgs get web stores and sites off the ground.
Nobody has to come up with excuses, for over ten years I've gotten straight answers whenever I've had a problem.
I don't care how unlikely it is I don't need them to prove jack squat because I've never been given a reason to mistrust Michael or PE/GV.
I have been given many reasons to trust them over the last decade so when I get a call from my friend and owner of the hosting company to tell me 'This is what happened' I believe him that that is what happened.
