Bitcoin Forum

Never was I so happy to see my favorite service being down :)
http://lw2.leowandersleb.de/tmp/maintenance.png

Slow mac mini? Slow? Your service always was amazingly fast :)
I love the scale that you are targeting for :)

Actually that is an old maintenance message which i forgot to update, the servers were switched a few months back. The old mac minis are being fired up again and some processes are being moved over (Bitcoind, Email server etc) as well as upgrading the firmware on the firewall and installing the latest mysql cluster version. Should be back fairly soon.

Ok, hmm … I was about to ask which programming language you used to serve so many users with so decent speed.
And as blockexplorer.com is down, too (at least when trying to find the balance of an address), I'm getting impatient now.

If you now move stuff to the mac mini again … I'm confused. Do you expect improved performance from that??? Kind of doesn't make sense to me.

Java


Spreading the workload around. Bitcoind resource usage is increasing, running in VMs is putting too much load on the main servers but a mac mini can run a full node without problem freeing up resources.

Can we get a pic of the blockchain.info server rack? Just curious as to what this all looks like.

I have asked the guys at the datacenter to take a photo.

-----

Couple of problems upgrading the database, ETA another hour.

Any updates?

Still waiting for a job to finish, should not be much longer.

-----

The database needs rebalancing as when the site was moved from two servers to 4 servers the existing tables were not repartitioned. This means the load is much higher on two of the servers rather than being distributed equally. The task is taking much longer to complete than I was expecting but it is important that it is done.

https://i.imgur.com/PqLBM.png

cool thanks :)

Close now?

Easy guys, they're working on it, right?

Database partitioning finished. Just need to make a full backup, run one alter table query, catch up bitcoind and the site can be brought back online.

You are the man, Piuk!

piuk, good to hear things are going well. I'm curious as to what caused the surge of problems though? I intended to rely on your receive api but now I'm having second thoughts. Was fairly spotty before you went down.

Ok back online, just catching up with the blockchain. Apologies for the downtime everyone.


dos attacks and high load in general coupled with the fact I have been away. The changes made today should help a lot with reliability.

ddos atacks astound me.  Don't little kiddies have better things to do with their time?

how good is this company?
http://www.blacklotus.net/ (http://www.blacklotus.net/)

I would have never tought to ask this here but hey why not... How do you find NDB to be? What kind of query/second can you handle? Are you keeping the whole database on ram or indexes only? I was always interested in it but always ended up with other solutions!

how long will it be down for?

I hear they have good moxie.

Here is a photo of the rack https://i.imgur.com/wMFG1.jpg. We do not have a full rack to ourselves yet.

3 mac mins (top) - 4 servers in one chassis (middle) - firewall & intrusion detection (bottom).


They are not really ddos attacks which a company like that would be able to help with. It is more just a few dozen tor ip addresses causing problems. For example one ip opening 1000's of websocket connections, or generating huge numbers of forwarding addresses.


It is entirely in ram. The site averages about 200-300 queries / second which isn't anything particularly special but cpu usage of ndb averages about 20%, considering this is decent 4 server cluster I feel like the load should be lower. However I haven't benchmarked it against any other database so I don't know if it really is an ndb problem or poor schema/query design. On the plus side it's pretty stable, I think before the recent maintenance it ran for 100 consecutive days without problems. In future I imagine moving to something like HBase/Cassandra but not for a while yet.