Bitcoin Forum
December 09, 2016, 02:12:11 AM *
News: Latest stable version of Bitcoin Core: 0.13.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: « 1 2 3 [4] 5 6 7 8 9 10 11 12 13 »  All
  Print  
Author Topic: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind  (Read 29393 times)
eleuthria
Legendary
*
Offline Offline

Activity: 1750


BTC Guild Owner


View Profile WWW
June 28, 2011, 04:20:22 PM
 #61

Just sent the first of possibly many donations.  Great efforts so far, and I'll be testing these on one of my servers tonight.

R.I.P. BTC Guild, 2011 - 2015.
BTC Guild Forum Thread
1481249531
Hero Member
*
Offline Offline

Posts: 1481249531

View Profile Personal Message (Offline)

Ignore
1481249531
Reply with quote  #2

1481249531
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
simplecoin
Sr. Member
****
Offline Offline

Activity: 406



View Profile WWW
June 28, 2011, 04:31:16 PM
 #62

Having problems doing rpc calls from php now. any page using them just stalls out.... I'm guessing it's waiting on a connection close (that doesn't exist)...

Donations: 1VjGJHPtLodwCFBDWsHJMdEhqRcRKdBQk
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
June 28, 2011, 04:34:10 PM
 #63

Having problems doing rpc calls from php now. any page using them just stalls out.... I'm guessing it's waiting on a connection close (that doesn't exist)...
It could be my bug or it could be your bug. I think it's my bug, and it only affects clients that either send HTTP/1.0 requests or send 'connection: close' headers. I'll fix it.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
simplecoin
Sr. Member
****
Offline Offline

Activity: 406



View Profile WWW
June 28, 2011, 04:37:16 PM
 #64

Having problems doing rpc calls from php now. any page using them just stalls out.... I'm guessing it's waiting on a connection close (that doesn't exist)...
It could be my bug or it could be your bug. I think it's my bug, and it only affects clients that either send HTTP/1.0 requests or send 'connection: close' headers. I'll fix it.

I'm using the standard php bitcoin interface from Mike Gogulski: http://www.nostate.com/3962/bitcoin-php-a-php-library-for-bitcoin/

Donations: 1VjGJHPtLodwCFBDWsHJMdEhqRcRKdBQk
bluefirecorp
Hero Member
*****
Offline Offline

Activity: 686


View Profile
June 28, 2011, 04:38:27 PM
 #65

My mate MIGHT have a solution -- he's dealt with having to have millions (literally) of connections to single servers/clusters before in his programming. He might of already built an API to deal with this.

.BitDice.               ▄▄███▄▄
           ▄▄██▀▀ ▄ ▀▀██▄▄
      ▄▄█ ▀▀  ▄▄█████▄▄  ▀▀ █▄▄
  ▄▄██▀▀     ▀▀ █████ ▀▀     ▀▀██▄▄
██▀▀ ▄▄██▀      ▀███▀      ▀██▄▄ ▀▀██
██  ████▄▄       ███       ▄▄████  ██
██  █▀▀████▄▄  ▄█████▄  ▄▄████▀▀█  ██
██  ▀     ▀▀▀███████████▀▀▀     ▀  ██
             ███████████
██  ▄     ▄▄▄███████████▄▄▄     ▄  ██
██  █▄▄████▀▀  ▀█████▀  ▀▀████▄▄█  ██
██  ████▀▀       ███       ▀▀████  ██
██▄▄ ▀▀██▄      ▄███▄      ▄██▀▀ ▄▄██
  ▀▀██▄▄     ▄▄ █████ ▄▄     ▄▄██▀▀
      ▀▀█ ▄▄  ▀▀█████▀▀  ▄▄ █▀▀
           ▀▀██▄▄ ▀ ▄▄██▀▀
               ▀▀███▀▀
        ▄▄███████▄▄
     ▄███████████████▄
    ████▀▀       ▀▀████
   ████▀           ▀████
   ████             ████
   ████ ▄▄▄▄▄▄▄▄▄▄▄ ████
▄█████████████████████████▄
██████████▀▀▀▀▀▀▀██████████
████                   ████
████                   ████
████                   ████
████                   ████
████                   ████
████▄                 ▄████
████████▄▄▄     ▄▄▄████████
  ▀▀▀█████████████████▀▀▀
        ▀▀▀█████▀▀▀
▄▄████████████████████████████████▄▄
██████████████████████████████████████
█████                            █████
█████                            █████
█████                            █████
█████                            █████
█████                     ▄▄▄▄▄▄▄▄▄▄
█████                   ▄█▀▀▀▀▀▀▀▀▀▀█▄
█████                   ██          ██
█████                   ██          ██
█████                   ██          ██
██████████████████▀▀███ ██          ██
 ████████████████▄  ▄██ ██          ██
   ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ ██          ██
             ██████████ ██          ██
           ▄███████████ ██████▀▀██████
          █████████████  ▀████▄▄████▀
[/]
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
June 28, 2011, 05:13:38 PM
 #66

Okay, all builds have now been updated. (rpc.diff.txt, bitcoin-3diff.txt, rpc.cpp)

It now does keepalives correct if the client specified HTTP/1.1 and didn't ask for the connection to be closed or the client specified HTTP/1.0 and asked for the connection to be kept alive. If this doesn't fix your connection hanging issue, make sure you are either sending "HTTP/1.0" or "Connection: close".

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
June 28, 2011, 06:16:42 PM
 #67

I'm using the standard php bitcoin interface from Mike Gogulski: http://www.nostate.com/3962/bitcoin-php-a-php-library-for-bitcoin/
That code expects that if it asks for HTTP 1.0 behavior, it will get correct HTTP 1.0 behavior. How unreasonable! My latest fix should do it.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
June 28, 2011, 06:21:42 PM
 #68

We're pushing this live in 40min. As soon as we got connection to bitcoind again, I'm paying out the promised 20 BTC.
I seriously love you right now Smiley

EDIT: To what address would you like to receive the payout? Feel free to PM me.
Or do you have a BitLC account i could credit? Choice is yours Smiley

/ Jim

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
simplecoin
Sr. Member
****
Offline Offline

Activity: 406



View Profile WWW
June 28, 2011, 07:07:36 PM
 #69

Okay, all builds have now been updated. (rpc.diff.txt, bitcoin-3diff.txt, rpc.cpp)

It now does keepalives correct if the client specified HTTP/1.1 and didn't ask for the connection to be closed or the client specified HTTP/1.0 and asked for the connection to be kept alive. If this doesn't fix your connection hanging issue, make sure you are either sending "HTTP/1.0" or "Connection: close".

That did it! Thanks!

Donations: 1VjGJHPtLodwCFBDWsHJMdEhqRcRKdBQk
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
June 28, 2011, 07:14:29 PM
 #70

We're up and running with the new patch.
Some fast stats:

Quote
root@bitcoins:/usr/local/sbin# netstat -an | grep 8332
tcp        0      0 127.0.0.1:8332          0.0.0.0:*               LISTEN
tcp        0    769 127.0.0.1:8332          127.0.0.1:42015         ESTABLISHED
tcp        0      0 127.0.0.1:42015         127.0.0.1:8332          ESTABLISHED

Quote
21:09:31 <@jine> damn, bitcoind is really fast
21:09:35 <@jine> getinfo takes about 0.01 sec now
21:09:39 <@jine> from 0.5s
21:09:45 <@jine> real    0m0.097s

Quote
21:04:56 <@jine> root@bitcoins:/usr/local/sbin# netstat -an | grep TIME | wc -l
21:04:56 <@jine> 833


By the way, here's the promised reward to our hero JoelKatz:
http://blockexplorer.com/tx/263dccc9f3899b4118a8f8568064ba2b417868ecff51329efad83b6446f22fed

I'm holding my breath and watching every stats live atm... We'll see if this run with our amount of connections - stable.
So far - It's rocking.

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
ius
Jr. Member
*
Offline Offline

Activity: 56


View Profile
June 28, 2011, 07:31:32 PM
 #71

Test reports, suggestions, and donations are all welcome.

Great job.

However (as mentioned before), I believe the proper way of solving the issue is by implementing keepalive on top of the asio [1] pull request (that's what Jeff suggested). I've looked into the issue reported there - turns out to be a trivial fix (send buffer goes out of scope, hence large tx'es fail).

Implementing keepalive on top of that should only take a few lines (header handling and timeout), and is probably a better merge candidate. Seems to work for me after fixing the asio patch and adding a quick hack to accept additional requests.

[1]: https://github.com/bitcoin/bitcoin/pull/214

PGP: 0xCC06E446 Bitcoin: 19kdfgW1KXQgV7SCLEPAojtHxN9xotGkGH
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
June 28, 2011, 07:46:10 PM
 #72

However (as mentioned before), I believe the proper way of solving the issue is by implementing keepalive on top of the asio [1] pull request (that's what Jeff suggested). I've looked into the issue reported there - turns out to be a trivial fix (send buffer goes out of scope, hence large tx'es fail).
I agree 100%. Using asio would be much better. My changes to implement the HTTP protocol properly with respect to keepalives should work fine with that code. (And my changes to rpc.cpp are definitely not mergable as they are.)

I'll see what I can do.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
June 28, 2011, 09:36:32 PM
 #73

/me cries.

At 23:13 bitcoind died. I'm not sure why, can't find anything more then this:
Quote
Jun 28 23:09:01 bitcoins /USR/SBIN/CRON[31658]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$($
Jun 28 23:09:35 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:09:58 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:10:01 bitcoins /USR/SBIN/CRON[8590]: (jine) CMD (cd /var/www; php cron.php >> cron.log)
Jun 28 23:10:50 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:11:19 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:12:08 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:12:42 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host

Quote
sending: inv (109 bytes)
received: inv (37 bytes)
  got inventory: tx e9c02552aa536253625a  have
sending: addr (31 bytes)
received: inv (37 bytes)
  got inventory: tx 5a643f15315d475b8136  have
sending: inv (37 bytes)
received: addr (31 bytes)
sending: inv (37 bytes)
received: addr (31 bytes)
received: addr (31 bytes)
sending: addr (31 bytes)
received: addr (31 bytes)
received: inv (37 bytes)
  got inventory: tx e9c02552aa536253625a  have
sending: addr (31 bytes)
received: addr (61 bytes)
AddAddress(88.122.132.102:8333)
received: addr (31 bytes)
received: addr (31 bytes)











(I then restarted it)



Bitcoin version 0.3.23-beta
Default data directory /root/.bitcoin
Bound to port 8333
Loading addresses...
dbenv.open strLogDir=/root/.bitcoin/database strErrorFile=/root/.bitcoin/db.log
Loaded 440713 addresses
 addresses              2626ms
Loading block index...
LoadBlockIndex(): hashBestChain=000000000000072182e9  height=133738
 block index            5793ms
Loading wallet...
...

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
June 28, 2011, 09:45:58 PM
 #74

Oh no!

You can run bitcoind in a script, instead of daemon mode. That way, if it dies, it will restart immediately and automatically. That's not great, but pain minimization is important. Basically:

while `true`; do bitcoind parameters go here; date >> /path/to/crash.log; done

If you do a 'ulimit -c unlimited' before running bitcoind, and make sure you aren't running a service that eats core files (like 'abrt'), you should be able to get a core dump you can analyze. Use 'gdb <executable> <path>' followed by the 'where' command. (Keeping hitting enter to get the full output, the last few lines are usually the most important.)

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
June 28, 2011, 10:29:37 PM
 #75

I believe I found your bug. All builds have been updated. It occurs when a JSON function times out. We try to close the underlying stream to force the HTTP parser to time out, but that doesn't work for SSL, and even if you aren't using SSL, if you compile with SSL support, all your API streams are SSL streams with SSL off. With the bug, when an RPC connection times out in bitcoind and then later times out in the caller, we fault.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
June 29, 2011, 12:04:29 AM
 #76

Compiled it and started it, seems to run fine so far... I'll keep you updated.

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
June 29, 2011, 01:36:42 AM
 #77

A couple of minutes ago bitcoind became unresponsive, used up 25-45% of the dedicated 8 cores (2xE5504 quad) and halted.

Code:
Mem:   3085764k total,  1642032k used,  1443732k free,   154552k buffers
Swap:   240632k total,        0k used,   240632k free,   955448k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7573 root      20   0 3052m 135m 7584 S   47  4.5  46:18.87 bitcoind
 7646 root      20   0 31180  23m 2116 S   10  0.8  20:21.11 pushpoold
  866 www-data  20   0 23204  14m 1692 S    4  0.5   2:06.18 nginx
  851 nobody    20   0 67752  23m  796 S    2  0.8  21:02.75 memcached
13181 www-data  20   0 25724 6512 3244 S    2  0.2   0:00.02 php5-fpm
13192 root      20   0  2468 1100  796 R    2  0.0   0:00.01 top

Nothing ordinary in dmesg / syslog nor netstat what i can see...
EDIT: This is with the ordinary rpc.cpp patch for keep-alive & threading - not the long-polling one.

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
June 29, 2011, 02:29:23 AM
 #78

I think this was from when bitcoind hung:

Quote
sending: addr (31 bytes)
sending: addr (31 bytes)
sending: addr (31 bytes)
sending: addr (31 bytes)
received: addr (61 bytes)
received: addr (31 bytes)
Error: pthread_create() returned 11










*restarted*




Bitcoin version 0.3.23-beta
Default data directory /root/.bitcoin
Bound to port 8333
Loading addresses...

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
June 29, 2011, 02:46:55 AM
 #79

It sounds like you either ran out of threads or ran out of address space. Sad

Do have your 'ulimit -u' value set to something unusually low? Or do you have lots of things other than pushpoold that also talk to bitcoind? Is the bitcoind a 32-bit executable or 64-bit? The 'file bitcoind' command will tell you.


I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
June 29, 2011, 02:48:52 AM
 #80

Quote
root@bitcoins:/usr/local/sbin# file bitcoind
bitcoind: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped
(Running on a 32-bit debian 6 release, latest stable)

root@bitcoins:/usr/local/sbin# ulimit -u
unlimited

I do have some requests to bitcoind from PHP (php-fpm) but so many. A few requests per minute - MAX (payouts, some stats which are cached in memcache etc)

EDIT:
Code:
root@bitcoins:/usr/local/sbin# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 131072
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
Pages: « 1 2 3 [4] 5 6 7 8 9 10 11 12 13 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!