[20 BTC] Multithreaded Keep-alive Implementation in Bitcoind

[eleuthria]

Just sent the first of possibly many donations.  Great efforts so far, and I'll be testing these on one of my servers tonight.

[1714026620]

1714026620

[1714026620]

1714026620

[1714026620]

1714026620

[1714026620]

1714026620

[1714026620]

1714026620

[1714026620]

1714026620

[simplecoin]

Having problems doing rpc calls from php now. any page using them just stalls out.... I'm guessing it's waiting on a connection close (that doesn't exist)...

[JoelKatz]

Having problems doing rpc calls from php now. any page using them just stalls out.... I'm guessing it's waiting on a connection close (that doesn't exist)...

It could be my bug or it could be your bug. I think it's my bug, and it only affects clients that either send HTTP/1.0 requests or send 'connection: close' headers. I'll fix it.

[simplecoin]

Having problems doing rpc calls from php now. any page using them just stalls out.... I'm guessing it's waiting on a connection close (that doesn't exist)...

It could be my bug or it could be your bug. I think it's my bug, and it only affects clients that either send HTTP/1.0 requests or send 'connection: close' headers. I'll fix it.

I'm using the standard php bitcoin interface from Mike Gogulski: http://www.nostate.com/3962/bitcoin-php-a-php-library-for-bitcoin/

[bluefirecorp]

My mate MIGHT have a solution -- he's dealt with having to have millions (literally) of connections to single servers/clusters before in his programming. He might of already built an API to deal with this.

[JoelKatz]

Okay, all builds have now been updated. (rpc.diff.txt, bitcoin-3diff.txt, rpc.cpp)

It now does keepalives correct if the client specified HTTP/1.1 and didn't ask for the connection to be closed or the client specified HTTP/1.0 and asked for the connection to be kept alive. If this doesn't fix your connection hanging issue, make sure you are either sending "HTTP/1.0" or "Connection: close".

[JoelKatz]

I'm using the standard php bitcoin interface from Mike Gogulski: http://www.nostate.com/3962/bitcoin-php-a-php-library-for-bitcoin/

That code expects that if it asks for HTTP 1.0 behavior, it will get correct HTTP 1.0 behavior. How unreasonable! My latest fix should do it.

[Jine]

We're pushing this live in 40min. As soon as we got connection to bitcoind again, I'm paying out the promised 20 BTC.
I seriously love you right now

EDIT: To what address would you like to receive the payout? Feel free to PM me.
Or do you have a BitLC account i could credit? Choice is yours

/ Jim

[simplecoin]

Okay, all builds have now been updated. (rpc.diff.txt, bitcoin-3diff.txt, rpc.cpp)

It now does keepalives correct if the client specified HTTP/1.1 and didn't ask for the connection to be closed or the client specified HTTP/1.0 and asked for the connection to be kept alive. If this doesn't fix your connection hanging issue, make sure you are either sending "HTTP/1.0" or "Connection: close".

That did it! Thanks!

[Jine]

We're up and running with the new patch.
Some fast stats:

root@bitcoins:/usr/local/sbin# netstat -an | grep 8332
tcp        0      0 127.0.0.1:8332          0.0.0.0:*               LISTEN
tcp        0    769 127.0.0.1:8332          127.0.0.1:42015         ESTABLISHED
tcp        0      0 127.0.0.1:42015         127.0.0.1:8332          ESTABLISHED

21:09:31 <@jine> damn, bitcoind is really fast
21:09:35 <@jine> getinfo takes about 0.01 sec now
21:09:39 <@jine> from 0.5s
21:09:45 <@jine> real    0m0.097s

21:04:56 <@jine> root@bitcoins:/usr/local/sbin# netstat -an | grep TIME | wc -l
21:04:56 <@jine> 833

By the way, here's the promised reward to our hero JoelKatz:
http://blockexplorer.com/tx/263dccc9f3899b4118a8f8568064ba2b417868ecff51329efad83b6446f22fed

I'm holding my breath and watching every stats live atm... We'll see if this run with our amount of connections - stable.
So far - It's rocking.

[ius]

Test reports, suggestions, and donations are all welcome.

Great job.

However (as mentioned before), I believe the proper way of solving the issue is by implementing keepalive on top of the asio [1] pull request (that's what Jeff suggested). I've looked into the issue reported there - turns out to be a trivial fix (send buffer goes out of scope, hence large tx'es fail).

Implementing keepalive on top of that should only take a few lines (header handling and timeout), and is probably a better merge candidate. Seems to work for me after fixing the asio patch and adding a quick hack to accept additional requests.

[1]: https://github.com/bitcoin/bitcoin/pull/214

[JoelKatz]

However (as mentioned before), I believe the proper way of solving the issue is by implementing keepalive on top of the asio [1] pull request (that's what Jeff suggested). I've looked into the issue reported there - turns out to be a trivial fix (send buffer goes out of scope, hence large tx'es fail).

I agree 100%. Using asio would be much better. My changes to implement the HTTP protocol properly with respect to keepalives should work fine with that code. (And my changes to rpc.cpp are definitely not mergable as they are.)

I'll see what I can do.

[Jine]

/me cries.

At 23:13 bitcoind died. I'm not sure why, can't find anything more then this:

Jun 28 23:09:01 bitcoins /USR/SBIN/CRON[31658]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$($
Jun 28 23:09:35 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:09:58 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:10:01 bitcoins /USR/SBIN/CRON[8590]: (jine) CMD (cd /var/www; php cron.php >> cron.log)
Jun 28 23:10:50 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:11:19 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:12:08 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:12:42 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host

sending: inv (109 bytes)
received: inv (37 bytes)
  got inventory: tx e9c02552aa536253625a  have
sending: addr (31 bytes)
received: inv (37 bytes)
  got inventory: tx 5a643f15315d475b8136  have
sending: inv (37 bytes)
received: addr (31 bytes)
sending: inv (37 bytes)
received: addr (31 bytes)
received: addr (31 bytes)
sending: addr (31 bytes)
received: addr (31 bytes)
received: inv (37 bytes)
  got inventory: tx e9c02552aa536253625a  have
sending: addr (31 bytes)
received: addr (61 bytes)
AddAddress(88.122.132.102:8333)
received: addr (31 bytes)
received: addr (31 bytes)











(I then restarted it)



Bitcoin version 0.3.23-beta
Default data directory /root/.bitcoin
Bound to port 8333
Loading addresses...
dbenv.open strLogDir=/root/.bitcoin/database strErrorFile=/root/.bitcoin/db.log
Loaded 440713 addresses
 addresses              2626ms
Loading block index...
LoadBlockIndex(): hashBestChain=000000000000072182e9  height=133738
 block index            5793ms
Loading wallet...
...

[JoelKatz]

Oh no!

You can run bitcoind in a script, instead of daemon mode. That way, if it dies, it will restart immediately and automatically. That's not great, but pain minimization is important. Basically:

while `true`; do bitcoind parameters go here; date >> /path/to/crash.log; done

If you do a 'ulimit -c unlimited' before running bitcoind, and make sure you aren't running a service that eats core files (like 'abrt'), you should be able to get a core dump you can analyze. Use 'gdb <executable> <path>' followed by the 'where' command. (Keeping hitting enter to get the full output, the last few lines are usually the most important.)

[JoelKatz]

I believe I found your bug. All builds have been updated. It occurs when a JSON function times out. We try to close the underlying stream to force the HTTP parser to time out, but that doesn't work for SSL, and even if you aren't using SSL, if you compile with SSL support, all your API streams are SSL streams with SSL off. With the bug, when an RPC connection times out in bitcoind and then later times out in the caller, we fault.

[Jine]

Compiled it and started it, seems to run fine so far... I'll keep you updated.

[Jine]

A couple of minutes ago bitcoind became unresponsive, used up 25-45% of the dedicated 8 cores (2xE5504 quad) and halted.

Code:

Mem:   3085764k total,  1642032k used,  1443732k free,   154552k buffers
Swap:   240632k total,        0k used,   240632k free,   955448k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7573 root      20   0 3052m 135m 7584 S   47  4.5  46:18.87 bitcoind
 7646 root      20   0 31180  23m 2116 S   10  0.8  20:21.11 pushpoold
  866 www-data  20   0 23204  14m 1692 S    4  0.5   2:06.18 nginx
  851 nobody    20   0 67752  23m  796 S    2  0.8  21:02.75 memcached
13181 www-data  20   0 25724 6512 3244 S    2  0.2   0:00.02 php5-fpm
13192 root      20   0  2468 1100  796 R    2  0.0   0:00.01 top

Nothing ordinary in dmesg / syslog nor netstat what i can see...
EDIT: This is with the ordinary rpc.cpp patch for keep-alive & threading - not the long-polling one.

[Jine]

I think this was from when bitcoind hung:

sending: addr (31 bytes)
sending: addr (31 bytes)
sending: addr (31 bytes)
sending: addr (31 bytes)
received: addr (61 bytes)
received: addr (31 bytes)
Error: pthread_create() returned 11










*restarted*




Bitcoin version 0.3.23-beta
Default data directory /root/.bitcoin
Bound to port 8333
Loading addresses...

[JoelKatz]

It sounds like you either ran out of threads or ran out of address space.

Do have your 'ulimit -u' value set to something unusually low? Or do you have lots of things other than pushpoold that also talk to bitcoind? Is the bitcoind a 32-bit executable or 64-bit? The 'file bitcoind' command will tell you.

[Jine]

root@bitcoins:/usr/local/sbin# file bitcoind
bitcoind: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped
(Running on a 32-bit debian 6 release, latest stable)

root@bitcoins:/usr/local/sbin# ulimit -u
unlimited

I do have some requests to bitcoind from PHP (php-fpm) but so many. A few requests per minute - MAX (payouts, some stats which are cached in memcache etc)

EDIT:

Code:

root@bitcoins:/usr/local/sbin# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 131072
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited