[20 BTC] Multithreaded Keep-alive Implementation in Bitcoind

[Jine]

Hi!

As a pool operator, I'm looking for a multi threaded keep-alive solution for bitcoind.
A brief description of the issue we're having (and i know BTCGuild is having) is described here:
https://github.com/bitcoin/bitcoin/issues/344

Async IO is already implemented here, could maybe give some hints.
https://github.com/bitcoin/bitcoin/pull/214 - But not fully functional.


20 BTC reward if we can fix this issues now:
A keep-alive (only a few connections between pushpoold and bitcoind) multi-threaded solution for bitcoind.

/ Jim

[1714802498]

1714802498

[1714802498]

1714802498

[1714802498]

1714802498

[1714802498]

1714802498

[Jine]

Someone at a Swedish forum found this link, should be something worth taking a look on:
http://codingplayground.blogspot.com/2008/07/boostasio-and-keep-alive.html

I'm no C++ dev tho..

/ Jim

[sakkaku]

If you run bitcoind and pushpoold on the same machine (or on the same lan) keep alive is a minimal improvement because opening TCP connections has next to no overhead locally compared to remote.

[Jine]

But we're seeing 20-30k TIME_WAIT connections due to this.
It's not the overhead, it's the amount of new sockets/connections.

This is a known problem for both us, eligius and btcguild (This is why they got multiple nodes with maximum ~500GH each)

[padrino]

But we're seeing 20-30k TIME_WAIT connections due to this.
It's not the overhead, it's the amount of new sockets/connections.

This is a known problem for both us, eligius and btcguild (This is why they got multiple nodes with maximum ~500GH each)

Did you end up sorting it out?

[Jine]

No solution yet, no.

[padrino]

No solution yet, no.

I have a history with socket programming and a couple hours to blow right now, I'll give it a look.. Offhand even without code fixes there may be some system level tweaks you can do to get things under control a bit more but I'll give the code a look now.

[Bloody Bell]

I have a history with socket programming

I just took a look. The Boost asio library is used, so to avoid replacing everything (and introduce a different style of coding), one should probably stick with that. The whole thing is running on a single thread, and as soon as a request is answered, closes the connection. Doesn't seem too efficient, indeed. I tried starting a new thread for each request, but there is some glitch somewhere. Unfortunately I am not too familiar with boost

[JoelKatz]

The problem description doesn't explain what the actual problem is. It says there are a large number of connections in TIME_WAIT state, but doesn't explain why that's a problem. I'd hate for someone to add keep alive and have it not actually solve the problem.

Since a DDoS attack can create arbitrarily large numbers of connection in the TIME_WAIT state, it's a much better solution to fix whatever is causing the large number of connections to be a problem. What is going wrong because of this? Fix that, and you'll get improved resistance to DDoS attacks for free.

[Bloody Bell]

The problem description doesn't explain what the actual problem is.

OP said that the connections are between the mining pool server and bitcoind. I suppose the rpc port of bitcoind isn't exposed to the internet, therefore can't be flooded. Also, most of the time it makes no sense to leave it open for anyone.

[JoelKatz]

The problem description doesn't explain what the actual problem is.

OP said that the connections are between the mining pool server and bitcoind. I suppose the rpc port of bitcoind isn't exposed to the internet, therefore can't be flooded. Also, most of the time it makes no sense to leave it open for anyone.

Ahh, okay. So forget the DDoS resistance stuff. But we still don't know why this is a problem.

If all the pool server needs is the 'getwork' output, it seems like a much better solution is to avoid polling altogether and have 'bitcoind' put this information in a file or write it continuously to a queue or something. (Assuming I'm understanding the problem correctly. I'm not 100% sure that I am.)

[Jine]

Lets get things straight.

It's an issue on localhost, for each and every getwork request pushpoold gets - it opens up a NEW socket against bitcoind, and issues getwork over that socket.
When it's finished and pushpoold got his work, the socket is left untouched and still opened.

We (and I know BTC Guild have) tweaked Linux a bit to reuse TIME_WAIT connections (using tcp_tw_recycle in /proc/...) which has helped - but it's not scalable.
As the load increases, the amount of new connections is overwhelming.

This was taken yesterday by me:

jine@bitcoins:~$ netstat -an | awk '/^tcp/ {A[$(NF)]++} END {for (I in A) {printf "%5d %s\n", A, I}}'
   22 FIN_WAIT2
   10 LISTEN
   58 SYN_RECV
  229 CLOSE_WAIT
21330 TIME_WAIT
 3899 ESTABLISHED
  282 LAST_ACK
   12 FIN_WAIT1

At that moment, we had 21.000+ TIME_WAIT sockets ready to be reused and/or closed due to timeout.
When that figure reaches 25k+ - the server starts to get hard to keep up, we're also hitting limits with num openfiles (nofiles in ulimit) which we increased from 1024 to 128.000+.

We're looking for a solution to get rid of "all" those TIME_WAIT connections (which 99.99999% comes from 127.0.0.1:8332 (bitcoind) against pushpoold at 127.0.0.01)
The best way I've come up with is implementing keep-alive support in bitcoind - hence this thread.

This will mean that bitcoind and pushpoold are only using a couple (multi threaded) keep-alive sockets for getwork and sending responses.
This will drastically lower the amount of open sockets (nofiles dropping to the bottom) and free both ports (limit of 65554 ports in TCP/IP afaik) and also make it more scalable.

Are everyone following now?

[Bloody Bell]

Are everyone following now?

I am, and this is what I read out from the first post too

On the other hand let's hope that libcurl can properly reuse the connections kept open, otherwise the new problem will be the overhead of creating/destroying the threads for each connection.

Not that if my bitcoind patch worked

[sakkaku]

Lets get things straight.

It's an issue on localhost, for each and every getwork request pushpoold gets - it opens up a NEW socket against bitcoind, and issues getwork over that socket.
When it's finished and pushpoold got his work, the socket is left untouched and still opened.

Has this been confirmed as a bitcoind problem?  Maybe it is pushpool that is misbehaving?  What about explicitly terminating the connection?  Adding keepalive isn't really a magical fix for that because pushpoold could decide it wants to open a new connection anyway.

[Jine]

pushpoold does NOT create a new socket explicitly for each connection, nor closing the previous one.
I got a PM with a "temporary" fix for this - make it explicitly close & open new connections.

We'll see how that goes, I'll try it out later on.

I don't know if this problem is confirmed to be bitcoind-only, but all larger pools based upon bitcoind+pushpoold are suffering of it AFAIK.
I'm gladly accepting comments, suggestions or other tips on this matter.

[Bloody Bell]

Has this been confirmed as a bitcoind problem?  Maybe it is pushpool that is misbehaving?  What about explicitly terminating the connection?  Adding keepalive isn't really a magical fix for that because pushpoold could decide it wants to open a new connection anyway.

You can try it by connecting to port 8332 with telnet: after one request has been serverd, bitcoind will close the connection. Even worse, as long as that connection is open it won't accept an other one. Everything is done on a single thread, you can see it in rpc.cpp,  at ThreadRPCServer2()

[JoelKatz]

I modified bitcoind both to implement keepalive properly and to fully multithread its RPC so that it can handle multiple RPC connections.

You can download a modified rpc.cpp here:
http://davids.webmaster.com/~davids/rpc.cpp

RPC performance is much snappier and it seems to work. But I am by no means a boost expert, so this is at your own risk. This is definitely 'quick and dirty, should clean up later' type code. Appreciation may be shown at this address: 1H3STBxuzEHZQQD4hkjVE22TWTazcZzeBw

[spiccioli]

But we're seeing 20-30k TIME_WAIT connections due to this.
It's not the overhead, it's the amount of new sockets/connections.

This is a known problem for both us, eligius and btcguild (This is why they got multiple nodes with maximum ~500GH each)

Jine,

maybe I'm saying something that is known to everybody here... but, I'll say anyway since I had a similar problem long ago (not bitcoin related, though).

Every socket spends some time in TIME_WAIT to correctly handle the closing of itself.

You can change the time it spends in TIME_WAIT which is set too long for a localhost only socket.

Per RFC every client waits 2*MSL (maximum segment life) before finally closing a socket, if you lower this time (default should be at least 120 seconds) to 10 seconds (for example) you should see that you have a lot less sockets waiting, and this should help you avoid reaching this 25K waiting sockets limit.

I don't know how to change MSL in Linux, though

Best regards.

spiccioli

[Jine]

I modified bitcoind both to implement keepalive properly and to fully multithread its RPC so that it can handle multiple RPC connections.

You can download a modified rpc.cpp here:
http://davids.webmaster.com/~davids/rpc.cpp

RPC performance is much snappier and it seems to work. But I am by no means a boost expert, so this is at your own risk. This is definitely 'quick and dirty, should clean up later' type code. Appreciation may be shown at this address: 1H3STBxuzEHZQQD4hkjVE22TWTazcZzeBw

At first sight i was like this:   
A bit later... 

Seg-faulted after a few seconds under the load. Not sure why, can't find any logs about it... :/

[Jine]

I don't know how to change MSL in Linux, though

I think it's possible by patching the kernel.
This will however not fix the issue completly.

I'm using tcp_max_tw_bucket (or something similar) atm - that limits TW-sockets to 10k max.
It works, but it does not solve the problem