eleuthria
Legendary
Offline
Activity: 1750
Merit: 1007
|
|
June 28, 2011, 04:20:22 PM |
|
Just sent the first of possibly many donations. Great efforts so far, and I'll be testing these on one of my servers tonight.
|
RIP BTC Guild, April 2011 - June 2015
|
|
|
simplecoin
|
|
June 28, 2011, 04:31:16 PM |
|
Having problems doing rpc calls from php now. any page using them just stalls out.... I'm guessing it's waiting on a connection close (that doesn't exist)...
|
Donations: 1VjGJHPtLodwCFBDWsHJMdEhqRcRKdBQk
|
|
|
JoelKatz
Legendary
Offline
Activity: 1596
Merit: 1012
Democracy is vulnerable to a 51% attack.
|
|
June 28, 2011, 04:34:10 PM |
|
Having problems doing rpc calls from php now. any page using them just stalls out.... I'm guessing it's waiting on a connection close (that doesn't exist)...
It could be my bug or it could be your bug. I think it's my bug, and it only affects clients that either send HTTP/1.0 requests or send 'connection: close' headers. I'll fix it.
|
I am an employee of Ripple. Follow me on Twitter @JoelKatz 1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
|
|
|
simplecoin
|
|
June 28, 2011, 04:37:16 PM |
|
Having problems doing rpc calls from php now. any page using them just stalls out.... I'm guessing it's waiting on a connection close (that doesn't exist)...
It could be my bug or it could be your bug. I think it's my bug, and it only affects clients that either send HTTP/1.0 requests or send 'connection: close' headers. I'll fix it. I'm using the standard php bitcoin interface from Mike Gogulski: http://www.nostate.com/3962/bitcoin-php-a-php-library-for-bitcoin/
|
Donations: 1VjGJHPtLodwCFBDWsHJMdEhqRcRKdBQk
|
|
|
bluefirecorp
Legendary
Offline
Activity: 882
Merit: 1000
|
|
June 28, 2011, 04:38:27 PM |
|
My mate MIGHT have a solution -- he's dealt with having to have millions (literally) of connections to single servers/clusters before in his programming. He might of already built an API to deal with this.
|
|
|
|
JoelKatz
Legendary
Offline
Activity: 1596
Merit: 1012
Democracy is vulnerable to a 51% attack.
|
|
June 28, 2011, 05:13:38 PM |
|
Okay, all builds have now been updated. (rpc.diff.txt, bitcoin-3diff.txt, rpc.cpp)
It now does keepalives correct if the client specified HTTP/1.1 and didn't ask for the connection to be closed or the client specified HTTP/1.0 and asked for the connection to be kept alive. If this doesn't fix your connection hanging issue, make sure you are either sending "HTTP/1.0" or "Connection: close".
|
I am an employee of Ripple. Follow me on Twitter @JoelKatz 1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
|
|
|
JoelKatz
Legendary
Offline
Activity: 1596
Merit: 1012
Democracy is vulnerable to a 51% attack.
|
|
June 28, 2011, 06:16:42 PM |
|
That code expects that if it asks for HTTP 1.0 behavior, it will get correct HTTP 1.0 behavior. How unreasonable! My latest fix should do it.
|
I am an employee of Ripple. Follow me on Twitter @JoelKatz 1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
|
|
|
Jine (OP)
|
|
June 28, 2011, 06:21:42 PM |
|
We're pushing this live in 40min. As soon as we got connection to bitcoind again, I'm paying out the promised 20 BTC. I seriously love you right now EDIT: To what address would you like to receive the payout? Feel free to PM me. Or do you have a BitLC account i could credit? Choice is yours / Jim
|
Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
|
|
|
simplecoin
|
|
June 28, 2011, 07:07:36 PM |
|
Okay, all builds have now been updated. (rpc.diff.txt, bitcoin-3diff.txt, rpc.cpp)
It now does keepalives correct if the client specified HTTP/1.1 and didn't ask for the connection to be closed or the client specified HTTP/1.0 and asked for the connection to be kept alive. If this doesn't fix your connection hanging issue, make sure you are either sending "HTTP/1.0" or "Connection: close".
That did it! Thanks!
|
Donations: 1VjGJHPtLodwCFBDWsHJMdEhqRcRKdBQk
|
|
|
Jine (OP)
|
|
June 28, 2011, 07:14:29 PM |
|
We're up and running with the new patch. Some fast stats: root@bitcoins:/usr/local/sbin# netstat -an | grep 8332 tcp 0 0 127.0.0.1:8332 0.0.0.0:* LISTEN tcp 0 769 127.0.0.1:8332 127.0.0.1:42015 ESTABLISHED tcp 0 0 127.0.0.1:42015 127.0.0.1:8332 ESTABLISHED 21:09:31 <@jine> damn, bitcoind is really fast 21:09:35 <@jine> getinfo takes about 0.01 sec now 21:09:39 <@jine> from 0.5s 21:09:45 <@jine> real 0m0.097s
21:04:56 <@jine> root@bitcoins:/usr/local/sbin# netstat -an | grep TIME | wc -l 21:04:56 <@jine> 833
By the way, here's the promised reward to our hero JoelKatz: http://blockexplorer.com/tx/263dccc9f3899b4118a8f8568064ba2b417868ecff51329efad83b6446f22fedI'm holding my breath and watching every stats live atm... We'll see if this run with our amount of connections - stable. So far - It's rocking.
|
Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
|
|
|
ius
Newbie
Offline
Activity: 56
Merit: 0
|
|
June 28, 2011, 07:31:32 PM |
|
Test reports, suggestions, and donations are all welcome.
Great job. However (as mentioned before), I believe the proper way of solving the issue is by implementing keepalive on top of the asio [1] pull request (that's what Jeff suggested). I've looked into the issue reported there - turns out to be a trivial fix (send buffer goes out of scope, hence large tx'es fail). Implementing keepalive on top of that should only take a few lines (header handling and timeout), and is probably a better merge candidate. Seems to work for me after fixing the asio patch and adding a quick hack to accept additional requests. [1]: https://github.com/bitcoin/bitcoin/pull/214
|
|
|
|
JoelKatz
Legendary
Offline
Activity: 1596
Merit: 1012
Democracy is vulnerable to a 51% attack.
|
|
June 28, 2011, 07:46:10 PM |
|
However (as mentioned before), I believe the proper way of solving the issue is by implementing keepalive on top of the asio [1] pull request (that's what Jeff suggested). I've looked into the issue reported there - turns out to be a trivial fix (send buffer goes out of scope, hence large tx'es fail).
I agree 100%. Using asio would be much better. My changes to implement the HTTP protocol properly with respect to keepalives should work fine with that code. (And my changes to rpc.cpp are definitely not mergable as they are.) I'll see what I can do.
|
I am an employee of Ripple. Follow me on Twitter @JoelKatz 1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
|
|
|
Jine (OP)
|
|
June 28, 2011, 09:36:32 PM |
|
/me cries.At 23:13 bitcoind died. I'm not sure why, can't find anything more then this: Jun 28 23:09:01 bitcoins /USR/SBIN/CRON[31658]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$($ Jun 28 23:09:35 bitcoins pushpoold[988]: mysql pwdb query failed at fetch Jun 28 23:09:58 bitcoins pushpoold[988]: mysql pwdb query failed at fetch Jun 28 23:10:01 bitcoins /USR/SBIN/CRON[8590]: (jine) CMD (cd /var/www; php cron.php >> cron.log) Jun 28 23:10:50 bitcoins pushpoold[988]: mysql pwdb query failed at fetch Jun 28 23:11:19 bitcoins pushpoold[988]: mysql pwdb query failed at fetch Jun 28 23:12:08 bitcoins pushpoold[988]: mysql pwdb query failed at fetch Jun 28 23:12:42 bitcoins pushpoold[988]: mysql pwdb query failed at fetch Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
sending: inv (109 bytes) received: inv (37 bytes) got inventory: tx e9c02552aa536253625a have sending: addr (31 bytes) received: inv (37 bytes) got inventory: tx 5a643f15315d475b8136 have sending: inv (37 bytes) received: addr (31 bytes) sending: inv (37 bytes) received: addr (31 bytes) received: addr (31 bytes) sending: addr (31 bytes) received: addr (31 bytes) received: inv (37 bytes) got inventory: tx e9c02552aa536253625a have sending: addr (31 bytes) received: addr (61 bytes) AddAddress(88.122.132.102:8333) received: addr (31 bytes) received: addr (31 bytes)
(I then restarted it)
Bitcoin version 0.3.23-beta Default data directory /root/.bitcoin Bound to port 8333 Loading addresses... dbenv.open strLogDir=/root/.bitcoin/database strErrorFile=/root/.bitcoin/db.log Loaded 440713 addresses addresses 2626ms Loading block index... LoadBlockIndex(): hashBestChain=000000000000072182e9 height=133738 block index 5793ms Loading wallet... ...
|
Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
|
|
|
JoelKatz
Legendary
Offline
Activity: 1596
Merit: 1012
Democracy is vulnerable to a 51% attack.
|
|
June 28, 2011, 09:45:58 PM Last edit: June 28, 2011, 10:00:58 PM by JoelKatz |
|
Oh no!
You can run bitcoind in a script, instead of daemon mode. That way, if it dies, it will restart immediately and automatically. That's not great, but pain minimization is important. Basically:
while `true`; do bitcoind parameters go here; date >> /path/to/crash.log; done
If you do a 'ulimit -c unlimited' before running bitcoind, and make sure you aren't running a service that eats core files (like 'abrt'), you should be able to get a core dump you can analyze. Use 'gdb <executable> <path>' followed by the 'where' command. (Keeping hitting enter to get the full output, the last few lines are usually the most important.)
|
I am an employee of Ripple. Follow me on Twitter @JoelKatz 1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
|
|
|
JoelKatz
Legendary
Offline
Activity: 1596
Merit: 1012
Democracy is vulnerable to a 51% attack.
|
|
June 28, 2011, 10:29:37 PM |
|
I believe I found your bug. All builds have been updated. It occurs when a JSON function times out. We try to close the underlying stream to force the HTTP parser to time out, but that doesn't work for SSL, and even if you aren't using SSL, if you compile with SSL support, all your API streams are SSL streams with SSL off. With the bug, when an RPC connection times out in bitcoind and then later times out in the caller, we fault.
|
I am an employee of Ripple. Follow me on Twitter @JoelKatz 1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
|
|
|
Jine (OP)
|
|
June 29, 2011, 12:04:29 AM |
|
Compiled it and started it, seems to run fine so far... I'll keep you updated.
|
Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
|
|
|
Jine (OP)
|
|
June 29, 2011, 01:36:42 AM |
|
A couple of minutes ago bitcoind became unresponsive, used up 25-45% of the dedicated 8 cores (2xE5504 quad) and halted. Mem: 3085764k total, 1642032k used, 1443732k free, 154552k buffers Swap: 240632k total, 0k used, 240632k free, 955448k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7573 root 20 0 3052m 135m 7584 S 47 4.5 46:18.87 bitcoind 7646 root 20 0 31180 23m 2116 S 10 0.8 20:21.11 pushpoold 866 www-data 20 0 23204 14m 1692 S 4 0.5 2:06.18 nginx 851 nobody 20 0 67752 23m 796 S 2 0.8 21:02.75 memcached 13181 www-data 20 0 25724 6512 3244 S 2 0.2 0:00.02 php5-fpm 13192 root 20 0 2468 1100 796 R 2 0.0 0:00.01 top
Nothing ordinary in dmesg / syslog nor netstat what i can see... EDIT: This is with the ordinary rpc.cpp patch for keep-alive & threading - not the long-polling one.
|
Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
|
|
|
Jine (OP)
|
|
June 29, 2011, 02:29:23 AM |
|
I think this was from when bitcoind hung: sending: addr (31 bytes) sending: addr (31 bytes) sending: addr (31 bytes) sending: addr (31 bytes) received: addr (61 bytes) received: addr (31 bytes) Error: pthread_create() returned 11
*restarted*
Bitcoin version 0.3.23-beta Default data directory /root/.bitcoin Bound to port 8333 Loading addresses...
|
Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
|
|
|
JoelKatz
Legendary
Offline
Activity: 1596
Merit: 1012
Democracy is vulnerable to a 51% attack.
|
|
June 29, 2011, 02:46:55 AM |
|
It sounds like you either ran out of threads or ran out of address space. Do have your 'ulimit -u' value set to something unusually low? Or do you have lots of things other than pushpoold that also talk to bitcoind? Is the bitcoind a 32-bit executable or 64-bit? The 'file bitcoind' command will tell you.
|
I am an employee of Ripple. Follow me on Twitter @JoelKatz 1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
|
|
|
Jine (OP)
|
|
June 29, 2011, 02:48:52 AM |
|
root@bitcoins:/usr/local/sbin# file bitcoind bitcoind: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped (Running on a 32-bit debian 6 release, latest stable)
root@bitcoins:/usr/local/sbin# ulimit -u unlimited
I do have some requests to bitcoind from PHP (php-fpm) but so many. A few requests per minute - MAX (payouts, some stats which are cached in memcache etc) EDIT: root@bitcoins:/usr/local/sbin# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 131072 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
|
Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
|
|
|
|