Redis Down on Bitcoinica

[notme]

Connection refused - Unable to connect to Redis on redis-1.bitcoinica.net:6379

Wake up Zhoutong!

[teflone]

Here too..

[zhoutong]

The Redis server went out of memory due to huge queues of flood of orders.

I have rebooted and the site will be up in 1 minute.

Thank you for the notification and I apologize for the error!

[notme]

The Redis server went out of memory due to huge queues of flood of orders.

I have rebooted and the site will be up in 1 minute.

Thank you for the notification and I apologize for the error!

I see the site, but prices are way low compared to what I see from mtgox data.

[zhoutong]

The Redis server went out of memory due to huge queues of flood of orders.

I have rebooted and the site will be up in 1 minute.

Thank you for the notification and I apologize for the error!

I see the site, but prices are way low compared to what I see from mtgox data.

It was temporary and not tradable.

Cause confirmed: Recently Mt. Gox API had a lot of issues and our Redis server has accumulated all sorts of errors until the memory is full.

[notme]

The Redis server went out of memory due to huge queues of flood of orders.

I have rebooted and the site will be up in 1 minute.

Thank you for the notification and I apologize for the error!

I see the site, but prices are way low compared to what I see from mtgox data.

It was temporary and not tradable.

Cause confirmed: Recently Mt. Gox API had a lot of issues and our Redis server has accumulated all sorts of errors until the memory is full.

You should really have a monitoring system in place to notify you if memory usage is growing without bound.

[zhoutong]

The Redis server went out of memory due to huge queues of flood of orders.

I have rebooted and the site will be up in 1 minute.

Thank you for the notification and I apologize for the error!

I see the site, but prices are way low compared to what I see from mtgox data.

It was temporary and not tradable.

Cause confirmed: Recently Mt. Gox API had a lot of issues and our Redis server has accumulated all sorts of errors until the memory is full.

You should really have a monitoring system in place to notify you if memory usage is growing without bound.

I have new relic to alert me whenever the site is down or any server has issues. However, I'm in the airport waiting to board my plane. That's why I had such delays just now. So unfortunate.

[zhoutong]

Site completely down for 41 minutes. The worst outage ever!!!

[notme]

You might want to consider having staff on call at all times, as well as failover hardware.  Operating on a shoestring budget isn't going to cut it if you want people to continue to trust you with thousands of dollars.  I'm not upset about the outage, and you responded very well given the situation.  However, one must always work towards reliability when dealing with such large sums.

[zhoutong]

You might want to consider having staff on call at all times, as well as failover hardware.  Operating on a shoestring budget isn't going to cut it if you want people to continue to trust you with thousands of dollars.  I'm not upset about the outage, and you responded very well given the situation.  However, one must always work towards reliability when dealing with such large sums.

I agree! Bitcoinica is a mission critical project now. I'll consider your suggestion. Thanks!

[teflone]

You might want to consider having staff on call at all times, as well as failover hardware.  Operating on a shoestring budget isn't going to cut it if you want people to continue to trust you with thousands of dollars.  I'm not upset about the outage, and you responded very well given the situation.  However, one must always work towards reliability when dealing with such large sums.

I agree! Bitcoinica is a mission critical project now. I'll consider your suggestion. Thanks!

I gotta hand it to you Zhou..   your professionalism is something you should be proud of.

Good work, keep it up..