BitVPS: Rebooting Cal2 server tonight

[phungus]

Hey everyone, we're still having some trouble notifying everyone individually, and completely forgot to post here.

We've been having some issues with our California #2 node and are going to have to perform an emergency reboot tonight. We already posted about this hours ago in #bitvps on IRC but it should probably be said here too.

We'll reboot CAL2 at 09:00 PST on 01/11/13.

Sorry for the late notice here, we're still working on getting all our ducks lined up to make BitVPS the best hosting provider in the Bitcoin community.

Thanks all!

-phungus

[1714952881]

1714952881

[1714952881]

1714952881

[1714952881]

1714952881

[phungus]

... and, everything came back up cleanly. We had to perform a second reboot though because some packages weren't installed properly. I made sure that they were and now the box is up for good.

Thanks everyone!

-phungus

[phungus]

Well, I spoke too soon. We're still experiencing issues with various VPS's on the Cal2 node.

We are working to fix the issue as quickly as we can and apologize for the outage.

-phungus

[phungus]

We finally resolved all of the issues with the Cal2 node and everyone should be up and working again. Sorry for the mess.

-phungus

[mc_lovin]

We finally resolved all of the issues with the Cal2 node and everyone should be up and working again. Sorry for the mess.

-phungus

Except me?  Keep plugging away, been down for 12 hours now, this thread makes it seem like a seamless transition but really here is exactly how much downtime there has been:



..and counting!

SO GET MY SERVER BACK ONLINE.  You guys said you are 'rebooting the server' I never heard of a 12 hour reboot before.

[mc_lovin]

I should have woken up earlier.  Sorry for being a bit irate but I wake up from a slumber to see my server down for so long, and I have detailed records of exactly how long and it's never a pretty sight. 

So glad you guys were able to get me rollin' so quickly.  I didn't even finish my 1st cup of morning coffee yet (even tho it's 4PM, it's morning to me recovering from the week). 

Please.. minimize these outages!  Do experiments on offline stuff!

[phungus]

Hey Mc_Lovin, this was definitely no experiment. We were a little hamstrung with no access to the hosting provider's portal or support. It took many hours but we finally were able to brute force our way into the panel and we were also fortunate enough to get hold of RG. He was still in possession of certain pieces of information that we needed to right everything.

The node had been up for 158 days prior to our announcing the outage. We thought it prudent to apply security updates to the box and then reboot it to try and solve the issues we had been seeing (unable to ping google.com, for instance). We most certainly were *not* experimenting, just trying to deliver a stable and secure environment for everyone.

I apologize for the delays. I believe that we now posses all information and logins for ALL of the BitVPS nodes, finally, as of today, Saturday 01/13/13. This after we took control of BitVPS before Christmas. We are definitely trying to make everything smooth and deliver better support than ever before.

Thanks for hanging in there. If you feel like you would like reimbursement for the outage, please open a ticket and we'll work with you.

Best,
-phungus