|
Title: Redis Down on Bitcoinica Post by: notme on January 12, 2012, 11:52:55 AM Connection refused - Unable to connect to Redis on redis-1.bitcoinica.net:6379
Wake up Zhoutong! Title: Re: Redis Down on Bitcoinica Post by: teflone on January 12, 2012, 11:54:58 AM Here too.. :(
Title: Re: Redis Down on Bitcoinica Post by: zhoutong on January 12, 2012, 12:27:39 PM The Redis server went out of memory due to huge queues of flood of orders.
I have rebooted and the site will be up in 1 minute. Thank you for the notification and I apologize for the error! Title: Re: Redis Down on Bitcoinica Post by: notme on January 12, 2012, 12:30:55 PM The Redis server went out of memory due to huge queues of flood of orders. I have rebooted and the site will be up in 1 minute. Thank you for the notification and I apologize for the error! I see the site, but prices are way low compared to what I see from mtgox data. Title: Re: Redis Down on Bitcoinica Post by: zhoutong on January 12, 2012, 12:40:39 PM The Redis server went out of memory due to huge queues of flood of orders. I have rebooted and the site will be up in 1 minute. Thank you for the notification and I apologize for the error! I see the site, but prices are way low compared to what I see from mtgox data. It was temporary and not tradable. Cause confirmed: Recently Mt. Gox API had a lot of issues and our Redis server has accumulated all sorts of errors until the memory is full. Title: Re: Redis Down on Bitcoinica Post by: notme on January 12, 2012, 12:41:35 PM The Redis server went out of memory due to huge queues of flood of orders. I have rebooted and the site will be up in 1 minute. Thank you for the notification and I apologize for the error! I see the site, but prices are way low compared to what I see from mtgox data. It was temporary and not tradable. Cause confirmed: Recently Mt. Gox API had a lot of issues and our Redis server has accumulated all sorts of errors until the memory is full. You should really have a monitoring system in place to notify you if memory usage is growing without bound. Title: Re: Redis Down on Bitcoinica Post by: zhoutong on January 12, 2012, 12:49:41 PM The Redis server went out of memory due to huge queues of flood of orders. I have rebooted and the site will be up in 1 minute. Thank you for the notification and I apologize for the error! I see the site, but prices are way low compared to what I see from mtgox data. It was temporary and not tradable. Cause confirmed: Recently Mt. Gox API had a lot of issues and our Redis server has accumulated all sorts of errors until the memory is full. You should really have a monitoring system in place to notify you if memory usage is growing without bound. I have new relic to alert me whenever the site is down or any server has issues. However, I'm in the airport waiting to board my plane. That's why I had such delays just now. So unfortunate. Title: Re: Redis Down on Bitcoinica Post by: zhoutong on January 12, 2012, 12:52:32 PM Site completely down for 41 minutes. The worst outage ever!!!
Title: Re: Redis Down on Bitcoinica Post by: notme on January 12, 2012, 12:54:15 PM You might want to consider having staff on call at all times, as well as failover hardware. Operating on a shoestring budget isn't going to cut it if you want people to continue to trust you with thousands of dollars. I'm not upset about the outage, and you responded very well given the situation. However, one must always work towards reliability when dealing with such large sums.
Title: Re: Redis Down on Bitcoinica Post by: zhoutong on January 12, 2012, 12:58:34 PM You might want to consider having staff on call at all times, as well as failover hardware. Operating on a shoestring budget isn't going to cut it if you want people to continue to trust you with thousands of dollars. I'm not upset about the outage, and you responded very well given the situation. However, one must always work towards reliability when dealing with such large sums. I agree! Bitcoinica is a mission critical project now. I'll consider your suggestion. Thanks! Title: Re: Redis Down on Bitcoinica Post by: teflone on January 12, 2012, 03:09:43 PM You might want to consider having staff on call at all times, as well as failover hardware. Operating on a shoestring budget isn't going to cut it if you want people to continue to trust you with thousands of dollars. I'm not upset about the outage, and you responded very well given the situation. However, one must always work towards reliability when dealing with such large sums. I agree! Bitcoinica is a mission critical project now. I'll consider your suggestion. Thanks! I gotta hand it to you Zhou.. your professionalism is something you should be proud of. Good work, keep it up.. |