Bitcoin Forum

Bitcoin => Project Development => Topic started by: Jine on June 25, 2011, 10:46:05 PM



Title: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 25, 2011, 10:46:05 PM
Hi!

As a pool operator, I'm looking for a multi threaded keep-alive solution for bitcoind.
A brief description of the issue we're having (and i know BTCGuild is having) is described here:
https://github.com/bitcoin/bitcoin/issues/344

Async IO is already implemented here, could maybe give some hints.
https://github.com/bitcoin/bitcoin/pull/214 - But not fully functional.


20 BTC reward if we can fix this issues now:
A keep-alive (only a few connections between pushpoold and bitcoind) multi-threaded solution for bitcoind.

/ Jim


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 26, 2011, 12:33:40 AM
Someone at a Swedish forum found this link, should be something worth taking a look on:
http://codingplayground.blogspot.com/2008/07/boostasio-and-keep-alive.html

I'm no C++ dev tho.. :(

/ Jim


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: sakkaku on June 26, 2011, 12:54:15 AM
If you run bitcoind and pushpoold on the same machine (or on the same lan) keep alive is a minimal improvement because opening TCP connections has next to no overhead locally compared to remote.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 26, 2011, 01:16:39 AM
But we're seeing 20-30k TIME_WAIT connections due to this.
It's not the overhead, it's the amount of new sockets/connections.

This is a known problem for both us, eligius and btcguild (This is why they got multiple nodes with maximum ~500GH each)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: padrino on June 26, 2011, 12:28:37 PM
But we're seeing 20-30k TIME_WAIT connections due to this.
It's not the overhead, it's the amount of new sockets/connections.

This is a known problem for both us, eligius and btcguild (This is why they got multiple nodes with maximum ~500GH each)

Did you end up sorting it out?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 26, 2011, 04:04:42 PM
No solution yet, no.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: padrino on June 26, 2011, 04:11:01 PM
No solution yet, no.

I have a history with socket programming and a couple hours to blow right now, I'll give it a look.. Offhand even without code fixes there may be some system level tweaks you can do to get things under control a bit more but I'll give the code a look now.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Bloody Bell on June 26, 2011, 06:02:19 PM
I have a history with socket programming

I just took a look. The Boost asio library is used, so to avoid replacing everything (and introduce a different style of coding), one should probably stick with that. The whole thing is running on a single thread, and as soon as a request is answered, closes the connection. Doesn't seem too efficient, indeed. I tried starting a new thread for each request, but there is some glitch somewhere. Unfortunately I am not too familiar with boost :(



Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 26, 2011, 06:19:52 PM
The problem description doesn't explain what the actual problem is. It says there are a large number of connections in TIME_WAIT state, but doesn't explain why that's a problem. I'd hate for someone to add keep alive and have it not actually solve the problem.

Since a DDoS attack can create arbitrarily large numbers of connection in the TIME_WAIT state, it's a much better solution to fix whatever is causing the large number of connections to be a problem. What is going wrong because of this? Fix that, and you'll get improved resistance to DDoS attacks for free.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Bloody Bell on June 26, 2011, 06:41:07 PM
The problem description doesn't explain what the actual problem is.
OP said that the connections are between the mining pool server and bitcoind. I suppose the rpc port of bitcoind isn't exposed to the internet, therefore can't be flooded. Also, most of the time it makes no sense to leave it open for anyone.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 26, 2011, 06:54:54 PM
The problem description doesn't explain what the actual problem is.
OP said that the connections are between the mining pool server and bitcoind. I suppose the rpc port of bitcoind isn't exposed to the internet, therefore can't be flooded. Also, most of the time it makes no sense to leave it open for anyone.
Ahh, okay. So forget the DDoS resistance stuff. But we still don't know why this is a problem.

If all the pool server needs is the 'getwork' output, it seems like a much better solution is to avoid polling altogether and have 'bitcoind' put this information in a file or write it continuously to a queue or something. (Assuming I'm understanding the problem correctly. I'm not 100% sure that I am.)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 26, 2011, 07:56:02 PM
Lets get things straight.

It's an issue on localhost, for each and every getwork request pushpoold gets - it opens up a NEW socket against bitcoind, and issues getwork over that socket.
When it's finished and pushpoold got his work, the socket is left untouched and still opened.

We (and I know BTC Guild have) tweaked Linux a bit to reuse TIME_WAIT connections (using tcp_tw_recycle in /proc/...) which has helped - but it's not scalable.
As the load increases, the amount of new connections is overwhelming.

This was taken yesterday by me:
Quote
jine@bitcoins:~$ netstat -an | awk '/^tcp/ {A[$(NF)]++} END {for (I in A) {printf "%5d %s\n", A, I}}'
   22 FIN_WAIT2
   10 LISTEN
   58 SYN_RECV
  229 CLOSE_WAIT
21330 TIME_WAIT
 3899 ESTABLISHED
  282 LAST_ACK
   12 FIN_WAIT1

At that moment, we had 21.000+ TIME_WAIT sockets ready to be reused and/or closed due to timeout.
When that figure reaches 25k+ - the server starts to get hard to keep up, we're also hitting limits with num openfiles (nofiles in ulimit) which we increased from 1024 to 128.000+.

We're looking for a solution to get rid of "all" those TIME_WAIT connections (which 99.99999% comes from 127.0.0.1:8332 (bitcoind) against pushpoold at 127.0.0.01)
The best way I've come up with is implementing keep-alive support in bitcoind - hence this thread.

This will mean that bitcoind and pushpoold are only using a couple (multi threaded) keep-alive sockets for getwork and sending responses.
This will drastically lower the amount of open sockets (nofiles dropping to the bottom) and free both ports (limit of 65554 ports in TCP/IP afaik) and also make it more scalable.

Are everyone following now?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Bloody Bell on June 26, 2011, 07:59:35 PM
Are everyone following now?

I am, and this is what I read out from the first post too :)

On the other hand let's hope that libcurl can properly reuse the connections kept open, otherwise the new problem will be the overhead of creating/destroying the threads for each connection.

Not that if my bitcoind patch worked :D


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: sakkaku on June 26, 2011, 08:05:15 PM
Lets get things straight.

It's an issue on localhost, for each and every getwork request pushpoold gets - it opens up a NEW socket against bitcoind, and issues getwork over that socket.
When it's finished and pushpoold got his work, the socket is left untouched and still opened.


Has this been confirmed as a bitcoind problem?  Maybe it is pushpool that is misbehaving?  What about explicitly terminating the connection?  Adding keepalive isn't really a magical fix for that because pushpoold could decide it wants to open a new connection anyway.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 26, 2011, 08:31:13 PM
pushpoold does NOT create a new socket explicitly for each connection, nor closing the previous one.
I got a PM with a "temporary" fix for this - make it explicitly close & open new connections.

We'll see how that goes, I'll try it out later on.

I don't know if this problem is confirmed to be bitcoind-only, but all larger pools based upon bitcoind+pushpoold are suffering of it AFAIK.
I'm gladly accepting comments, suggestions or other tips on this matter.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Bloody Bell on June 26, 2011, 08:40:27 PM
Has this been confirmed as a bitcoind problem?  Maybe it is pushpool that is misbehaving?  What about explicitly terminating the connection?  Adding keepalive isn't really a magical fix for that because pushpoold could decide it wants to open a new connection anyway.

You can try it by connecting to port 8332 with telnet: after one request has been serverd, bitcoind will close the connection. Even worse, as long as that connection is open it won't accept an other one. Everything is done on a single thread, you can see it in rpc.cpp,  at ThreadRPCServer2()


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 26, 2011, 09:19:05 PM
I modified bitcoind both to implement keepalive properly and to fully multithread its RPC so that it can handle multiple RPC connections.

You can download a modified rpc.cpp here:
http://davids.webmaster.com/~davids/rpc.cpp

RPC performance is much snappier and it seems to work. But I am by no means a boost expert, so this is at your own risk. This is definitely 'quick and dirty, should clean up later' type code. Appreciation may be shown at this address: 1H3STBxuzEHZQQD4hkjVE22TWTazcZzeBw


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: spiccioli on June 26, 2011, 09:44:09 PM
But we're seeing 20-30k TIME_WAIT connections due to this.
It's not the overhead, it's the amount of new sockets/connections.

This is a known problem for both us, eligius and btcguild (This is why they got multiple nodes with maximum ~500GH each)

Jine,

maybe I'm saying something that is known to everybody here... but, I'll say anyway since I had a similar problem long ago (not bitcoin related, though).

Every socket spends some time in TIME_WAIT to correctly handle the closing of itself.

You can change the time it spends in TIME_WAIT which is set too long for a localhost only socket.

Per RFC every client waits 2*MSL (maximum segment life) before finally closing a socket, if you lower this time (default should be at least 120 seconds) to 10 seconds (for example) you should see that you have a lot less sockets waiting, and this should help you avoid reaching this 25K waiting sockets limit.

I don't know how to change MSL in Linux, though :)

Best regards.

spiccioli


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 26, 2011, 09:57:20 PM
I modified bitcoind both to implement keepalive properly and to fully multithread its RPC so that it can handle multiple RPC connections.

You can download a modified rpc.cpp here:
http://davids.webmaster.com/~davids/rpc.cpp

RPC performance is much snappier and it seems to work. But I am by no means a boost expert, so this is at your own risk. This is definitely 'quick and dirty, should clean up later' type code. Appreciation may be shown at this address: 1H3STBxuzEHZQQD4hkjVE22TWTazcZzeBw

At first sight i was like this:  :o :o :o :o :o  :D :D :D :D
A bit later...  :'(

Seg-faulted after a few seconds under the load. Not sure why, can't find any logs about it... :/


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 26, 2011, 09:59:19 PM
I don't know how to change MSL in Linux, though :)


I think it's possible by patching the kernel.
This will however not fix the issue completly.

I'm using tcp_max_tw_bucket (or something similar) atm - that limits TW-sockets to 10k max.
It works, but it does not solve the problem


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: spiccioli on June 26, 2011, 10:20:49 PM
I don't know how to change MSL in Linux, though :)


I think it's possible by patching the kernel.
This will however not fix the issue completly.

I'm using tcp_max_tw_bucket (or something similar) atm - that limits TW-sockets to 10k max.
It works, but it does not solve the problem


Jine,

I've google around a little, tcp_max_tw_bucket puts a limit (default 180000) on the maximum number of sockets in TIME_WAIT, if this limit is exceeded sockets entering TIME_WAIT state are closed without waiting and a warning issued... but first you have to reach the limit.

tcp_fin_timeout, or net.ipv4.tcp_fin_timeout, on the other hand, seems to be what you need.

Lowering it (it could be 60 seconds right now) you should be able to limit  TIME_WAIT sockets to a lower number without/before reaching the bucket limit.

I mean, if you set it to 3 second, you'll end up having as many TIME_WAIT sockets as your incoming connections in a 3 seconds period, so your 25K sockets should go down to 1500-2000.

spiccioli.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 26, 2011, 10:27:05 PM
At first sight i was like this:  :o :o :o :o :o  :D :D :D :D
A bit later...  :'(

Seg-faulted after a few seconds under the load. Not sure why, can't find any logs about it... :/
Sorry to hear that. I'll audit my changes for the kinds of things that typically cause seg faults.

Update: Found the problem. Fixing it now.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 26, 2011, 10:30:38 PM
I mean, if you set it to 3 second, you'll end up having as many TIME_WAIT sockets as your incoming connections in a 3 seconds period, so your 25K sockets should go down to 1500-2000.

tcp_fin_timeout is already set to 3s, forgot to mention that in my previous post.
These settings are currently used (since a week back), and we still have the problems described before


Quote
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 3 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 30 > /proc/sys/net/ipv4/tcp_keepalive_intvl
echo 5 > /proc/sys/net/ipv4/tcp_keepalive_probes
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_all
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
echo 1 > /proc/sys/net/ipv4/tcp_syncookies


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 26, 2011, 10:31:21 PM
Sorry to hear that. I'll audit my changes for the kinds of things that typically cause seg faults.

Thanks, please do!
If we solve this permanently, I'm sending 20 BTC your way...


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 26, 2011, 10:41:44 PM
Okay, new build is up. It passes a pretty aggressive stress test, but I'll keep stressing it, just in case. Thanks for trying it.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: phorensic on June 26, 2011, 11:34:14 PM
You guys might find the comments at the bottom of this pushpool commit interesting as well: https://github.com/shanew/pushpool/commit/ef4f7261a839e628bfa24a988055a23fe442e5d2 (https://github.com/shanew/pushpool/commit/ef4f7261a839e628bfa24a988055a23fe442e5d2)  He mentions changing from TCP sockets to unix domain sockets for the communications between pushpoold and bitcoind.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 27, 2011, 12:03:50 AM
Okay, new build is up. It passes a pretty aggressive stress test, but I'll keep stressing it, just in case. Thanks for trying it.

Same URL? Will try it out in a moment.

You guys might find the comments at the bottom of this pushpool commit interesting as well: https://github.com/shanew/pushpool/commit/ef4f7261a839e628bfa24a988055a23fe442e5d2 (https://github.com/shanew/pushpool/commit/ef4f7261a839e628bfa24a988055a23fe442e5d2)  He mentions changing from TCP sockets to unix domain sockets for the communications between pushpoold and bitcoind.

Thats actually better, BUT it would require modifications to both bitcoind and pushpoold.
This was the previous main target, but i couldn't find anyone able to implement it.

It would be really. really. awesome. Offering another 10 BTC for such an implementation.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 27, 2011, 12:15:53 AM
Same URL? Will try it out in a moment.
Yes, same URL.

Quote
Thats actually better, BUT it would require modifications to both bitcoind and pushpoold.
This was the previous main target, but i couldn't find anyone able to implement it.
It's better in some ways and worse in others. Alone, it won't help the fact that bitcoind can't respond while it's waiting for another connection to respond. But it would eliminate the TIME_WAIT states that pile up because UNIX domain sockets don't have them. You'd need a multi-listening patch like mine to listen on a UNIX-domain socket and a TCP socket at the same time (or you could call select, but that's ugly). I haven't seen the code on the other side yet, so I don't know what's involved in making it use a UNIX domain socket.

Of course, if you have multi-threaded listening, keep alives, and UNIX domain sockets, that should *really* solve the problem.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: hamdi on June 27, 2011, 12:17:44 AM
best would be to remove the network way from bitcoind to pushpoold.

on one hand shitty to force both running on one machine, but the network bottleneck would be gone by sharing that data via ram instead of network


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 27, 2011, 12:18:27 AM
best would be to remove the network way from bitcoind to pushpoold.

on one hand shitty to force both running on one machine, but the network bottleneck would be gone by sharing that data via ram instead of network
It is shared by RAM. Connections between two processes on the same machine don't actually flow over any network. It just emulates a network, and with that comes both good things and bad things.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 27, 2011, 01:01:47 AM
Same URL? Will try it out in a moment.
Yes, same URL.

Quote
Thats actually better, BUT it would require modifications to both bitcoind and pushpoold.
This was the previous main target, but i couldn't find anyone able to implement it.
It's better in some ways and worse in others. Alone, it won't help the fact that bitcoind can't respond while it's waiting for another connection to respond. But it would eliminate the TIME_WAIT states that pile up because UNIX domain sockets don't have them. You'd need a multi-listening patch like mine to listen on a UNIX-domain socket and a TCP socket at the same time (or you could call select, but that's ugly). I haven't seen the code on the other side yet, so I don't know what's involved in making it use a UNIX domain socket.

Of course, if you have multi-threaded listening, keep alives, and UNIX domain sockets, that should *really* solve the problem.


Unfortunately, i could not get this version to work in our production environment either.
The strange thing is that this DO work if i run it separately... Downloaded nightly build of blockchain, tried it out on port 1337 and it started, i could issue getwork-requests to it and it seems to have reused the socket to.

On the other hand, when i replace our bitcoind with this patched version - it starts, runs for 4 seconds and then silently dies. I was able to issue one getinfo requests from it's client - but then it died.

Straaaaange... but i really think we're onto something here.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 27, 2011, 03:13:11 AM
It must be a problem that only appears under load or under some particular combination of requests. I'll try to do some more troubleshooting. I don't have much time left today, but I should have a few hours tomorrow that I can dedicate.

Update1: Do you compile bitcoind with any non-standard settings? And do you start it with any command line flags other than '-daemon'? Do you enable RPC over SSL?

Update2: I made a few cleanups and fixed a few very minor issues. But I can't replicate your issue, which means either I solved it or it requires something unique to your setup to replicate. If you get a chance to try my latest (same place) please do. If you can compile it with '-g' (I believe that's provided by default), make sure not to strip the executable, do a 'ulimit -c unlimited' before you execute it, and if it crashes, run 'gdb' on the core file like this: 'gdb /path/to/bitcoind /path/to/core.filename' and then message me the output of a 'where' command. (You may have to hit 'enter' a few times to get the full output. It'll be the last few lines that will be the most helpful.)

As soon as you have the core file, you can restart the original bitcoind. You don't need to have any downtime. Just make sure to pass 'gdb' the path to the bitcoind executable that generate the core file.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 27, 2011, 08:40:34 AM
There's another new version up. I realized the JSON code wasn't re-entrant, which creates a problem when you try to use it in more than one thread. Unfortunately, the 'getblock' code isn't re-entrant, and the simplest way to deal with that is to wrap all the RPC handlers in a big mutex. It doesn't seem to have any effect on performance though, so I'll leave it that way because it's much safer.

This version makes the JSON code re-entrant, but single threads calls to do the actual RPC. This is slightly less than optimum, but in all of my tests it made no difference. Multi-threading the actual RPC calls would carry a significant risk that some part of that code would break for no significant benefit and unless the 'getblock' code was pessimized for the most common case, it wouldn't benefit anyway. (Plus, there would have to be invasive changes to the code that handles when you successfully find a block, and that scares me because it's so critical and so hard to test.)

Please test this version. It should solve the problem.

I also have a version with UNIX domain sockets available if anyone's interested (it's not up at the moment, but PM me if you want it). It's very ugly right now because I haven't had time to polish it, but it does work. It supports a
'-unixsocket=<filename>' option. The protocol is a single line query and a single line response, no headers, no authentication (so put the socket in a directory only the authorized user can access). There is also code to issue RPC calls over the UNIX-domain socket, so you can see how to do it and see that it works. The biggest issue with it right now is that if you make any errors, it just closes the socket. You can issue multiple requests over a single connection though, and of course there are no stale socket issues.

The biggest ugliness is that I couldn't figure out how to bind a basic_istream to a local::stream_protocol::socket. So I had to use a 'receive' call instead of 'getline'. If anyone knows how to do that, I'd appreciate a PM. (I always meant to learn boost.)

In truth, none of these are the right solution. I have some ideas for the 'right' solution (bitcoind should push changes to the mining controller so it doesn't have to poll), and I'll try to get them thought out and proposed as modifications to the official source. (Think of it as extending long polling back one more link in the chain.)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: NANO on June 28, 2011, 01:40:23 AM
JoelKatz seems to be a Hero...


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: ius on June 28, 2011, 02:17:25 AM
In truth, none of these are the right solution. I have some ideas for the 'right' solution (bitcoind should push changes to the mining controller so it doesn't have to poll), and I'll try to get them thought out and proposed as modifications to the official source. (Think of it as extending long polling back one more link in the chain.)

Even so it would make much more sense to do it properly (so we end up having a useful pull request against bitcoin); the asio route appears to be the way to go (patch is there, yet bugged) instead of spawning multiple threads.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 02:23:34 AM
Even so it would make much more sense to do it properly (so we end up having a useful pull request against bitcoin); the asio route appears to be the way to go (patch is there, yet bugged) instead of spawning multiple threads.
Oh, do you know where the patch is? There's a good chance I could debug it. I've been meaning to learn about boost anyway. (My day job involves high-performance, multi-threaded TCP server code.)

Asio won't actually gain you much over my patch. The main advantage of asio is when you have large numbers of connections, most of which aren't very active or when large numbers of them become active at the same time. You would want it in a mining controller. Think about the large number of connections, the long periods of inactivity, and the sudden burst when a new block comes out. Without asio, you have to have a context switch for each connection. With asio, you do not.

That said, there's basically no downside, and it's also the right thing to do.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 28, 2011, 02:25:13 AM
Lets try this out right now. Compiling the code as we speak.
Give you an update and/or coredump in a while :)

The "right way" seems pretty awesome to me... :)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 02:32:08 AM
The "right way" seems pretty awesome to me... :)
Okay. If I get the existing asio work for bitcoind, I'll work on that. I've grabbed the source code to the mining daemon and am looking at how it implements long polling.

Update: It looks like pushpoold already has a way to do this, with blkmond. So the only issue is the connection buildup, which I think I've already fixed.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 28, 2011, 03:24:28 AM
Something goes wrong when running your current patched bitcoind in our prod. environment.
Now the process just hung, couldn't get any coredump out of it or similar :(

Suggestions? I'll try this one in my personal dev-environment and see if i can replicate the issue.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 03:42:08 AM
Suggestions? I'll try this one in my personal dev-environment and see if i can replicate the issue.
I'm guessing that there's some path that doesn't release the RPC mutex. I put up a new build that might fix it, but it's hard to be sure since I can't replicate the problem.

Update: If that fails, I can just pull the multi-threading stuff and only fix keep-alives. That's a no-brainer (two lines change in trivial ways) and is very unlikely to cause any problems.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 28, 2011, 06:56:23 AM
Suggestions? I'll try this one in my personal dev-environment and see if i can replicate the issue.
I'm guessing that there's some path that doesn't release the RPC mutex. I put up a new build that might fix it, but it's hard to be sure since I can't replicate the problem.

Update: If that fails, I can just pull the multi-threading stuff and only fix keep-alives. That's a no-brainer (two lines change in trivial ways) and is very unlikely to cause any problems.

Tried out the latest version in my dev-environment - couldn't get the keep-alive to work tho.
It seems to create a new connection for each request.

Tried with the following script...

Code:
jine@srv:~/bitcoin-0.3.23/src/src$ cat test.sh
while [ 1 ]
do
        curl --keepalive-time 60 --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getwork", "params": [] }' -H 'content-type: text/plain;' http://test:test@127.0.0.1:8332/
done

Directly against bitcoind that is, but i think curl should handle keep-alives well...
Any suggestions? :/ Could NOT get it to crash tho. Tried multiple times with huge number of requests etc... (not tried pushpoold yet tho)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 08:30:46 AM
If you start a new curl process for each request, there is no way it can reuse the connection. You'd have to ask a single instance of curl to make multiple requests.

If you try this:
   curl --keepalive-time 60 --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getwork", "params": [] }'
 -H 'content-type: text/plain;' "http://test:test@127.0.0.1:8332/[00-99]"

You'll see it retrieve 100 times without using 100 connections.

Without my patch: 100 connections, 5 seconds.
With my patch: 1 connection, .067 seconds.
They were all 'access denied' though. I didn't send a password. I'll retest with correct credentials.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 08:43:19 AM
Thanks for the idea of using curl to test. I put up a new build. I didn't realize I was still sending 'Connection: close' in the success replies. Doh!

Now I ran this test:

curl -u MyUser:MyPass --keepalive --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getwork", "params": [] }' -H 'content-type: text/plain;' "http://test:test@127.0.0.1:8332/[00-99]"

Without my patch, 100 connections, .2 seconds.
With my patch, 1 connection, .09 seconds.

The [00-99] makes it retrieve 100 URLs -- bitcoind ignores the URL.

The only change between the version you have and my latest is this:

@@ -1571,7 +1571,7 @@ string HTTPReply(int nStatus, const stri
     return strprintf(
             "HTTP/1.1 %d %s\r\n"
             "Date: %s\r\n"
-            "Connection: close\r\n"
+            "Connection: keep-alive\r\n"
             "Content-Length: %d\r\n"
             "Content-Type: application/json\r\n"
             "Server: bitcoin-json-rpc/%s\r\n"


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Caesium on June 28, 2011, 10:06:59 AM
Looks good! Trying it with a small private pool and no errors to report so far.

Pushpool is happily re-using connections and work is still being dished out - TIME_WAITs have disappeared :)

For anyone struggling to get this to build, by the way, use 0.3.22, not .23 or git. Any chance of a diff rather than the full file JoelKatz?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 10:08:43 AM
Looks good! Trying it with a small private pool and no errors to report so far.

Pushpool is happily re-using connections and work is still being dished out - TIME_WAITs have disappeared :)

For anyone struggling to get this to build, by the way, use 0.3.22, not .23 or git. Any chance of a diff rather than the full file JoelKatz?
Glad to hear it. Unfortunately, the changes are really invasive. It will be hard to produce a clean diff. I'll see what I can do.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 10:14:05 AM
Actually, it proved to be really easy! Here's a diff:
http://davids.webmaster.com/~davids/rpc.diff.txt

If it doesn't patch cleanly against some version you need to patch against, let me know. I should be able to rebase it without too much trouble. By the way, I was using 0.3.23.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 28, 2011, 10:22:10 AM
I think i just discovered the issue with dying bitcoind, read the log below :)

I this when bitcoind unexpectedly  died, and i tried to restart it
Quote
11:44:15 <@jine> EXCEPTION: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_6system12sys tem_errorEEEEE
11:44:15 <@jine> Address already in use
11:44:15 <@jine> bitcoin in ThreadRPCServer()
11:44:15 <@jine> terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >'

What actually happend:
Quote
11:48:12 <@jine> I can't start bitcoind while 3000+ connections tries to connect to :8332
(This is due to the fact that pushpoold is using up all ports, including tcp/8332 to try to connect to bitcoind)

11:48:44 <@jine> When 3000+ people tries to connect to pushpoold
11:48:52 <@jine> I cant stop it on the "correct" way
11:48:57 <+cereal7802> so i take it your going to kill pushpool, try the patched bitcoind again, then restart push?
11:49:11 <@jine> Have to kill it with; kill -9 (forcefully)
(So all my previous test have been unsuccuessfull due to pushpoold still running, still using up sockets, which makes bitcoind not start / dying while initializing)

Will try that in a moment, but it looks promising :)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 10:29:51 AM
By default, temporary ports are allocated from 32768 - 61000, so 8332 shouldn't ever conflict (unless you changed that to increase the range). This is controlled by /proc/sys/net/ipv4/ip_local_port_range.

If you increased the range to try to prevent running out of ports, you can blacklist 8332. Use
/proc/sys/net/ipv4/ip_local_reserved_ports.

It may help to add a 'brake' to pushpoold to stop it from doing whatever it's doing that's causing the issue. (Perhaps if a connection to bitcoind fails, do not attempt to reconnect for two seconds. That's pretty easy to hack in.) But I doubt it's that it's using a needed port.



Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 28, 2011, 10:32:57 AM
We have tweaked ip_local_port_range ;)

Quote
root@bitcoins:/usr/src/bitcoin-0.3.23/src/src# cat /proc/sys/net/ipv4/ip_local_port_range
1024    65535

EDIT: This to allow more sockets to be open at once, due to the amount of connections we're getting :)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 10:33:57 AM
We have tweaked ip_local_port_range ;)

Quote
root@bitcoins:/usr/src/bitcoin-0.3.23/src/src# cat /proc/sys/net/ipv4/ip_local_port_range
1024    65535
Ahh, you need to blacklist critical ports then!

See starting from line 604 here:
http://www.mjmwired.net/kernel/Documentation/networking/ip-sysctl.txt


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 28, 2011, 10:34:21 AM
Or modify it back to 32-65k and apply your patch ;)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 10:35:02 AM
Or modify it back to 32-65k and apply your patch ;)
Even better!


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Caesium on June 28, 2011, 10:47:04 AM
Sent you a little something Joel for your efforts :)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 10:54:31 AM
Sent you a little something Joel for your efforts :)
Thanks. It is much appreciated.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: eleuthria on June 28, 2011, 02:14:23 PM
Checking in after getting linked back here this morning.  Is the patch now working in a high volume production environment?  If so I'll have to throw some BTC at JoelKatz as well.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 03:02:16 PM
Checking in after getting linked back here this morning.  Is the patch now working in a high volume production environment?  If so I'll have to throw some BTC at JoelKatz as well.
Donations will go to prioritizing my next BitCoin-related project, optimizing bitcoind for pooled mining in a few other ways.

First, native long-polling support in bitcoind, eliminating the need for blkmond and speeding up switching to a new block. You'll be able to pass a pidfile to bitcoind, and it will automatically send a SIGUSR1 to the pool manager when a new block is found.

Second, aggressively acquiring and maintaining more connections. The current client actually goes out of its way to keep fewer connections. But if you have more connections, you win two ways. First, you find out about new blocks faster, meaning you won't waste time working on a shorter chain. Second, your found blocks will get out the fastest, increasing the odds that your block will stay in the chain.

Third, optimizing the code paths that miners hit heavily. This includes, for example, more rapidly getting a found block out to all connected clients. (This actually may already be close to optimal. I'm not sure there's much room for improvement here, but it won't hurt to look.)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: simplecoin on June 28, 2011, 03:31:33 PM
running the diff against .23 I get 10 out of 10 hunks failed.....

does this only work against .22?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 03:36:01 PM
running the diff against .23 I get 10 out of 10 hunks failed.....

does this only work against .22?
I made it against .23, and I just checked it against a clean .23 and it applied fine. Make sure you're in the src/src directory. "patch < rpc.diff" should work.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: simplecoin on June 28, 2011, 03:39:00 PM
hmm, I was... tried with -p0, retrying.

Update: copied to src/src and renamed to rpc.diff. patch < diff worked fine. Thanks!

compiling now.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 03:54:28 PM
A preliminary combined diff is now available.
http://davids.webmaster.com/~davids/bitcoin-3diff.txt

Features:

1) RPC enhancements to support JSON parsing in a thread, concurrent handling of multiple RPC connections, and working HTTP keepalives.

2) Native long-polling support. New option "-pollpidfile=/path/to/pidfile". Pass in the path to your pool manager's PID file. (By default with pushpoold, it's "/var/run/pushpoold.pid".) This is faster than blkmond and eliminates the need to run it. (In fact, DO NOT use blkmond with this option as you'll wind up telling your miners to abort valid runs.)

3) More aggressive connection management. Outbound connections are made in their own thread and the client tries more aggressively to get more of them. This should mean both that you find out about new blocks faster and you get your own blocks out faster. This is most noticeable when you just restart bitcoind, as it tries very aggressively on startup to re-establish connectivity. But the benefits should accrue all the time -- you'll be working on a longer chain faster and others will be working on extending your blocks faster.

The changes to 'net.cpp' should not be used on machines with limited net connectivity as more connections will mean more traffic. Currently, these changes are not very tunable. (I plan to add a '-hub' option that will enable these features to make you more of a 'hub' of the BitCoin network.)

Test reports, suggestions, and donations are all welcome.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: eleuthria on June 28, 2011, 04:20:22 PM
Just sent the first of possibly many donations.  Great efforts so far, and I'll be testing these on one of my servers tonight.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: simplecoin on June 28, 2011, 04:31:16 PM
Having problems doing rpc calls from php now. any page using them just stalls out.... I'm guessing it's waiting on a connection close (that doesn't exist)...


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 04:34:10 PM
Having problems doing rpc calls from php now. any page using them just stalls out.... I'm guessing it's waiting on a connection close (that doesn't exist)...
It could be my bug or it could be your bug. I think it's my bug, and it only affects clients that either send HTTP/1.0 requests or send 'connection: close' headers. I'll fix it.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: simplecoin on June 28, 2011, 04:37:16 PM
Having problems doing rpc calls from php now. any page using them just stalls out.... I'm guessing it's waiting on a connection close (that doesn't exist)...
It could be my bug or it could be your bug. I think it's my bug, and it only affects clients that either send HTTP/1.0 requests or send 'connection: close' headers. I'll fix it.

I'm using the standard php bitcoin interface from Mike Gogulski: http://www.nostate.com/3962/bitcoin-php-a-php-library-for-bitcoin/


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: bluefirecorp on June 28, 2011, 04:38:27 PM
My mate MIGHT have a solution -- he's dealt with having to have millions (literally) of connections to single servers/clusters before in his programming. He might of already built an API to deal with this.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 05:13:38 PM
Okay, all builds have now been updated. (rpc.diff.txt, bitcoin-3diff.txt, rpc.cpp)

It now does keepalives correct if the client specified HTTP/1.1 and didn't ask for the connection to be closed or the client specified HTTP/1.0 and asked for the connection to be kept alive. If this doesn't fix your connection hanging issue, make sure you are either sending "HTTP/1.0" or "Connection: close".


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 06:16:42 PM
I'm using the standard php bitcoin interface from Mike Gogulski: http://www.nostate.com/3962/bitcoin-php-a-php-library-for-bitcoin/
That code expects that if it asks for HTTP 1.0 behavior, it will get correct HTTP 1.0 behavior. How unreasonable! My latest fix should do it.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 28, 2011, 06:21:42 PM
We're pushing this live in 40min. As soon as we got connection to bitcoind again, I'm paying out the promised 20 BTC.
I seriously love you right now :)

EDIT: To what address would you like to receive the payout? Feel free to PM me.
Or do you have a BitLC account i could credit? Choice is yours :)

/ Jim


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: simplecoin on June 28, 2011, 07:07:36 PM
Okay, all builds have now been updated. (rpc.diff.txt, bitcoin-3diff.txt, rpc.cpp)

It now does keepalives correct if the client specified HTTP/1.1 and didn't ask for the connection to be closed or the client specified HTTP/1.0 and asked for the connection to be kept alive. If this doesn't fix your connection hanging issue, make sure you are either sending "HTTP/1.0" or "Connection: close".

That did it! Thanks!


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 28, 2011, 07:14:29 PM
We're up and running with the new patch.
Some fast stats:

Quote
root@bitcoins:/usr/local/sbin# netstat -an | grep 8332
tcp        0      0 127.0.0.1:8332          0.0.0.0:*               LISTEN
tcp        0    769 127.0.0.1:8332          127.0.0.1:42015         ESTABLISHED
tcp        0      0 127.0.0.1:42015         127.0.0.1:8332          ESTABLISHED

Quote
21:09:31 <@jine> damn, bitcoind is really fast
21:09:35 <@jine> getinfo takes about 0.01 sec now
21:09:39 <@jine> from 0.5s
21:09:45 <@jine> real    0m0.097s

Quote
21:04:56 <@jine> root@bitcoins:/usr/local/sbin# netstat -an | grep TIME | wc -l
21:04:56 <@jine> 833


By the way, here's the promised reward to our hero JoelKatz:
http://blockexplorer.com/tx/263dccc9f3899b4118a8f8568064ba2b417868ecff51329efad83b6446f22fed

I'm holding my breath and watching every stats live atm... We'll see if this run with our amount of connections - stable.
So far - It's rocking.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: ius on June 28, 2011, 07:31:32 PM
Test reports, suggestions, and donations are all welcome.

Great job.

However (as mentioned before), I believe the proper way of solving the issue is by implementing keepalive on top of the asio [1] pull request (that's what Jeff suggested). I've looked into the issue reported there - turns out to be a trivial fix (send buffer goes out of scope, hence large tx'es fail).

Implementing keepalive on top of that should only take a few lines (header handling and timeout), and is probably a better merge candidate. Seems to work for me after fixing the asio patch and adding a quick hack to accept additional requests.

[1]: https://github.com/bitcoin/bitcoin/pull/214


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 07:46:10 PM
However (as mentioned before), I believe the proper way of solving the issue is by implementing keepalive on top of the asio [1] pull request (that's what Jeff suggested). I've looked into the issue reported there - turns out to be a trivial fix (send buffer goes out of scope, hence large tx'es fail).
I agree 100%. Using asio would be much better. My changes to implement the HTTP protocol properly with respect to keepalives should work fine with that code. (And my changes to rpc.cpp are definitely not mergable as they are.)

I'll see what I can do.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 28, 2011, 09:36:32 PM
/me cries.

At 23:13 bitcoind died. I'm not sure why, can't find anything more then this:
Quote
Jun 28 23:09:01 bitcoins /USR/SBIN/CRON[31658]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$($
Jun 28 23:09:35 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:09:58 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:10:01 bitcoins /USR/SBIN/CRON[8590]: (jine) CMD (cd /var/www; php cron.php >> cron.log)
Jun 28 23:10:50 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:11:19 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:12:08 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:12:42 bitcoins pushpoold[988]: mysql pwdb query failed at fetch
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host
Jun 28 23:13:02 bitcoins pushpoold[988]: HTTP request failed: couldn't connect to host

Quote
sending: inv (109 bytes)
received: inv (37 bytes)
  got inventory: tx e9c02552aa536253625a  have
sending: addr (31 bytes)
received: inv (37 bytes)
  got inventory: tx 5a643f15315d475b8136  have
sending: inv (37 bytes)
received: addr (31 bytes)
sending: inv (37 bytes)
received: addr (31 bytes)
received: addr (31 bytes)
sending: addr (31 bytes)
received: addr (31 bytes)
received: inv (37 bytes)
  got inventory: tx e9c02552aa536253625a  have
sending: addr (31 bytes)
received: addr (61 bytes)
AddAddress(88.122.132.102:8333)
received: addr (31 bytes)
received: addr (31 bytes)











(I then restarted it)



Bitcoin version 0.3.23-beta
Default data directory /root/.bitcoin
Bound to port 8333
Loading addresses...
dbenv.open strLogDir=/root/.bitcoin/database strErrorFile=/root/.bitcoin/db.log
Loaded 440713 addresses
 addresses              2626ms
Loading block index...
LoadBlockIndex(): hashBestChain=000000000000072182e9  height=133738
 block index            5793ms
Loading wallet...
...


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 09:45:58 PM
Oh no!

You can run bitcoind in a script, instead of daemon mode. That way, if it dies, it will restart immediately and automatically. That's not great, but pain minimization is important. Basically:

while `true`; do bitcoind parameters go here; date >> /path/to/crash.log; done

If you do a 'ulimit -c unlimited' before running bitcoind, and make sure you aren't running a service that eats core files (like 'abrt'), you should be able to get a core dump you can analyze. Use 'gdb <executable> <path>' followed by the 'where' command. (Keeping hitting enter to get the full output, the last few lines are usually the most important.)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 28, 2011, 10:29:37 PM
I believe I found your bug. All builds have been updated. It occurs when a JSON function times out. We try to close the underlying stream to force the HTTP parser to time out, but that doesn't work for SSL, and even if you aren't using SSL, if you compile with SSL support, all your API streams are SSL streams with SSL off. With the bug, when an RPC connection times out in bitcoind and then later times out in the caller, we fault.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 29, 2011, 12:04:29 AM
Compiled it and started it, seems to run fine so far... I'll keep you updated.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 29, 2011, 01:36:42 AM
A couple of minutes ago bitcoind became unresponsive, used up 25-45% of the dedicated 8 cores (2xE5504 quad) and halted.

Code:
Mem:   3085764k total,  1642032k used,  1443732k free,   154552k buffers
Swap:   240632k total,        0k used,   240632k free,   955448k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7573 root      20   0 3052m 135m 7584 S   47  4.5  46:18.87 bitcoind
 7646 root      20   0 31180  23m 2116 S   10  0.8  20:21.11 pushpoold
  866 www-data  20   0 23204  14m 1692 S    4  0.5   2:06.18 nginx
  851 nobody    20   0 67752  23m  796 S    2  0.8  21:02.75 memcached
13181 www-data  20   0 25724 6512 3244 S    2  0.2   0:00.02 php5-fpm
13192 root      20   0  2468 1100  796 R    2  0.0   0:00.01 top

Nothing ordinary in dmesg / syslog nor netstat what i can see...
EDIT: This is with the ordinary rpc.cpp patch for keep-alive & threading - not the long-polling one.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 29, 2011, 02:29:23 AM
I think this was from when bitcoind hung:

Quote
sending: addr (31 bytes)
sending: addr (31 bytes)
sending: addr (31 bytes)
sending: addr (31 bytes)
received: addr (61 bytes)
received: addr (31 bytes)
Error: pthread_create() returned 11










*restarted*




Bitcoin version 0.3.23-beta
Default data directory /root/.bitcoin
Bound to port 8333
Loading addresses...


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 29, 2011, 02:46:55 AM
It sounds like you either ran out of threads or ran out of address space. :(

Do have your 'ulimit -u' value set to something unusually low? Or do you have lots of things other than pushpoold that also talk to bitcoind? Is the bitcoind a 32-bit executable or 64-bit? The 'file bitcoind' command will tell you.



Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 29, 2011, 02:48:52 AM
Quote
root@bitcoins:/usr/local/sbin# file bitcoind
bitcoind: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped
(Running on a 32-bit debian 6 release, latest stable)

root@bitcoins:/usr/local/sbin# ulimit -u
unlimited

I do have some requests to bitcoind from PHP (php-fpm) but so many. A few requests per minute - MAX (payouts, some stats which are cached in memcache etc)

EDIT:
Code:
root@bitcoins:/usr/local/sbin# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 131072
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 29, 2011, 02:52:23 AM
It's a resource leak, and it's not in my changes! It's in the 'CreateThread' function. I guess I use that more heavily than anything else, so I exposed the resource leak. Here's the fix:

--- orig/util.h 2011-06-28 08:28:03.070006598 -0700
+++ new/util.h  2011-06-28 19:51:51.186449938 -0700
@@ -624,7 +624,10 @@ inline pthread_t CreateThread(void(*pfn)
         return (pthread_t)0;
     }
     if (!fWantHandle)
+    {
+        pthread_detach(hthread);
         return (pthread_t)-1;
+    }
     return hthread;
 }

Sorry about the rocky road. This one was just bad luck though. This change should be merged with the official client -- it's an obviously-correct, obviously-safe change.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 29, 2011, 03:10:22 AM
Compiled it and pushed it live now.
No segfault or errors in the beginning anyway ;)

We'll see in a few hours. I'll post here if anything happens.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on June 29, 2011, 03:30:13 AM
Forgot to mention, thanks a lot for your hard work.  :o
We'll remember this in the future, that you can be sure of.



Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 29, 2011, 08:04:16 AM
I have a new version out at:
http://davids.webmaster.com/~davids/bitcoin-3diff.txt

It has the turbo RPC changes, the new hub mode to reduce stale/lost blocks, and native long polling support. Please read the documentation at the top of the file.

CAUTION: This code was just finished and has not been well-tested. Test results are appreciated. Bug reports and success reports are equally welcome.

Update: Version 0.4 is now up. It fixes a bug that could cause bitcoind to hang when directed to shutdown.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: simplecoin on June 29, 2011, 06:10:46 PM
I have a new version out at:
http://davids.webmaster.com/~davids/bitcoin-3diff.txt

It has the turbo RPC changes, the new hub mode to reduce stale/lost blocks, and native long polling support. Please read the documentation at the top of the file.

CAUTION: This code was just finished and has not been well-tested. Test results are appreciated. Bug reports and success reports are equally welcome.

Update: Version 0.4 is now up. It fixes a bug that could cause bitcoind to hang when directed to shutdown.

Currently testing v0.5 on live server.... (a whopping 3 users since relaunch). Will let you know how it turns out.

So far so good. LP notifications seem faster than a certain unnamed big pool ;) Also it seems to favor my pool during new getwork causing my pool to get a higher share of hashrates than said big pool when using hashkill.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: eleuthria on June 29, 2011, 06:22:20 PM
800 GH/sec and counting on the BTC Guild stress test.
UPDATE:  900 and counting.

BTC Guild, where test runs are done on production servers!


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: eleuthria on June 29, 2011, 06:45:48 PM
1 TH/sec is where idles started coming back for my DE server, which is an i7-920 system.  I believe the new bottleneck is the getwork processing.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on June 29, 2011, 06:52:49 PM
Thanks for the testing and the report.

I believe your diagnosis is probably right. I've been trying to figure out why that latency is so high. It turns out that when processing a significant change, like when a new block has been found, the code has to do a whole bunch of stuff. Coincidentally, this is exactly the same time every client needs new work. I believe the mechanism is contention on the cs_main mutex, which 'getwork' needs in this case because it calls CreateNewBlock. I'll do some head scratching.

One way to fix this might be to pre-compute the information 'getwork' needs so we don't have to wait for a 'getwork' request to do it. That would allow 'getwork' to run without touching the cs_main mutex. I'll need to instrument the code to confirm that's the issue before putting the effort into a change that might fix something that's not actually a problem.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on July 01, 2011, 04:41:04 AM
We're running the latest 0.5 patch since last night - and it seems to work just fine.
Although - a couple of users have complained about higher stale-count.

Is that something you are aware of that could be a sideeffect of the modifications?

LP seems to work just fine tho.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 01, 2011, 05:00:31 AM
We're running the latest 0.5 patch since last night - and it seems to work just fine.
Although - a couple of users have complained about higher stale-count.

Is that something you are aware of that could be a sideeffect of the modifications?

LP seems to work just fine tho.
I'm not really quite sure. Two questions:

1) Are you using any of the hub modes? If so, which level?

2) Are you using my long polling or are you using blkmond?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on July 01, 2011, 05:17:55 AM
Hi!

This should answer both of those:
Code:
BITCOIND_OPTS="-conf=/etc/bitcoin.conf -pid=/var/run/bitcoind.pid -hub=2 -pollpidfile=/var/run/pushpoold.pid -daemon"


In other words: hub level 2, and yes - built in LP.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 01, 2011, 06:25:12 AM
Awesome. I'm glad all of my code bits are getting "real use". Let me know if people are still seeing a higher percentage of stale blocks. It's possible the daemon just needed some time to 'settle in' as it makes connections and so on. My next set of changes may fix this anyway, it's designed specifically to reduce latency in 'getwork', particularly around reorganizations and transaction list changes.

Make sure your CPU has the juice to run at "-hub=2". Note that with a multi-core CPU, things can be deceptive. Bitcoind isn't very well multi-threaded -- if one core is maxed out, there's a good chance bitcoind is CPU limited. My next changes after the changes I'm working on now are expected to be better multi-threading of the code that handles I/O.

You did remember not to run blkmond, right? It will conflict with native long polling.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on July 01, 2011, 03:42:10 PM
Hi!

We're running the "master node" upon a 2*Intel Xeon E5504 (8 cores in total);
http://data.fuskbugg.se/skalman02/4e0deb49dc5b9_tits.png

So far it looks good for me. And no, no blkmond.
But i think we're still experiencing a bit of lag / stales when LP is pushed, due to huge amounts of connections I'd think.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: phorensic on July 01, 2011, 03:56:24 PM
Great progress!  One question, are we tracking this fork on github?  I didn't see the patches on the network graph there.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 01, 2011, 06:16:33 PM
Great progress!  One question, are we tracking this fork on github?  I didn't see the patches on the network graph there.
No, I'm not doing that ... yet!


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: MrSam on July 04, 2011, 10:12:57 PM
hmm, where can i get v0.5 ?

I tried 0.4 yesterday but wont work, bitcoind crashes after pushpool sends many
'HTTP request failed: The requested URL returned error: 500' and opened like 20k sockets in seconds


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 04, 2011, 10:17:16 PM
hmm, where can i get v0.5 ?

I tried 0.4 yesterday but wont work, bitcoind crashes after pushpool sends many
'HTTP request failed: The requested URL returned error: 500' and opened like 20k sockets in seconds

http://davids.webmaster.com/~davids/bitcoin-4diff.txt

I'm starting to suspect the newer issues are coming from pushpool, but I'm not 100% sure yet. I'm not going to have much time today, but I hope to investigate in more detail tomorrow.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: MrSam on July 05, 2011, 05:39:10 PM
.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: MrSam on July 05, 2011, 11:41:07 PM
Joel,

I wanted to thank you ! The patch works really well now ..

The only issue that remains is that the USR1 signal has a bit of a delay.

pushpool debug:
[2011-07-05 23:36:1.710434] USR1 signal received, flushing LP waiters
if i check blockexplorer, that block was found at 2011-07-05 23:35:31)

[2011-07-05 23:33:36.427621] USR1 signal received, flushing LP waiters
real time 2011-07-05 23:33:24

[2011-07-05 23:30:36.337827] USR1 signal received, flushing LP waiters
real time 2011-07-05 23:30:20

When i test this on my mining rig at home, the rejected blocks are just those few seconds between the found block and the sig.





Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 06, 2011, 03:50:54 AM
Thanks for that report. I'll check the code and see if I can get the SIGUSR1 to be sent faster. I think that will solve half the problem though. While pushpool will issue fewer blocks that will turn into stale shares, bitcoind will still be processing the new block when all the new 'getwork' requests start piling in, and you'll get miner idles instead.

I have a fix for this that has other benefits as well, but the first time I tried to implement it, it was a disaster. I think I know the right way to do it now, having run out of wrong ways to do it already. What I'd like to do is pre-compute the skeleton 'getwork' block when a new block or new transaction comes in, rather than waiting to hijack the next 'getwork' request to do it. But to make this work, the locking has to be exactly right. The funny thing is, in my day job ten years ago, this was exactly the type optimization I specialized in. And now I can't seem to quite get it right.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: somebadger on July 06, 2011, 04:22:40 AM
i just merged your diff with the latest from the official bitcoin git,

https://github.com/somebadger/bitcoin4pools (https://github.com/somebadger/bitcoin4pools)

I also added the following to --help

Code:
  -pollpidfile=<file>      Pushpoold Pid file for long polling support
  -hub=<n>      Hub mode for pool operators

other than some manual intervention and fixing CReserveKey::CReserveKey(CWallet*) syntax for the latest build it wasnt too much of a headache. enough though that i'm keeping it in git now :P

ps. i hate manually doing .rej's lol


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 06, 2011, 04:50:22 AM
If you get the chance, a pull request for the two resource leak fixes would be nice.

That would be this change to net.cpp:

     // Send and receive from sockets, accept connections
-    pthread_t hThreadSocketHandler = CreateThread(ThreadSocketHandler, NULL, true);
+    CreateThread(ThreadSocketHandler, NULL);

And this change to util.h:

         return (pthread_t)0;
     }
     if (!fWantHandle)
+    {
+        pthread_detach(hthread);
         return (pthread_t)-1;
+    }


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: somebadger on July 06, 2011, 05:08:34 AM
If you get the chance, a pull request for the two resource leak fixes would be nice.

That would be this change to net.cpp:

     // Send and receive from sockets, accept connections
-    pthread_t hThreadSocketHandler = CreateThread(ThreadSocketHandler, NULL, true);
+    CreateThread(ThreadSocketHandler, NULL);

And this change to util.h:

         return (pthread_t)0;
     }
     if (!fWantHandle)
+    {
+        pthread_detach(hthread);
         return (pthread_t)-1;
+    }


Patched and submitted pull request
https://github.com/bitcoin/bitcoin/pull/385


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 06, 2011, 05:33:42 AM
Thanks.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: somebadger on July 06, 2011, 06:50:37 AM
No probs ;)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: TeraPool on July 06, 2011, 06:55:34 AM
i just merged your diff with the latest from the official bitcoin git,

https://github.com/somebadger/bitcoin (https://github.com/somebadger/bitcoin)

I also added the following to --help

Code:
  -pollpidfile=<file>      Pushpoold Pid file for long polling support
  -hub=<n>      Hub mode for pool operators

other than some manual intervention and fixing CReserveKey::CReserveKey(CWallet*) syntax for the latest build it wasnt too much of a headache. enough though that i'm keeping it in git now :P

ps. i hate manually doing .rej's lol

QUESTION

JoelKatz's optimizations here will probably never make it into the official bitcoin github account, will they? Because they typically only make sense for pool operators or am I wrong?

The patches I believe will definitely make it in because that can benefit everybody. (Great work!)

So if anyone downloads the latest "v0.3.24rc3" or later from somebadger's github account, then it will contain JoelKatz's latest "diff" as seen here http://davids.webmaster.com/~davids/bitcoin-3diff.txt ???

Will somebadger's fork always contain the new pool related code? Just wondering if I should just make my own github fork of bitcoin just for my own pool so I can keep track of this and any future similar changes.

Great work JoelKatz. Thank you for that contribution to the community!


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: somebadger on July 06, 2011, 07:13:19 AM
QUESTION

JoelKatz's optimizations here will probably never make it into the official bitcoin github account, will they? Because they typically only make sense for pool operators or am I wrong?

The patches I believe will definitely make it in because that can benefit everybody. (Great work!)

So if anyone downloads the latest "v0.3.24rc3" or later from somebadger's github account, then it will contain JoelKatz's latest "diff" as seen here http://davids.webmaster.com/~davids/bitcoin-3diff.txt ???

Will somebadger's fork always contain the new pool related code? Just wondering if I should just make my own github fork of bitcoin just for my own pool so I can keep track of this and any future similar changes.

Great work JoelKatz. Thank you for that contribution to the community!

My plan is to keep the fork upto date with any improvements for pool operators, i'll also be adding bitcoin pulls that help pool performance

ps. its patched based on http://davids.webmaster.com/~davids/bitcoin-4diff.txt and modifications to make it apply to the latest build


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 06, 2011, 07:16:52 AM
JoelKatz's optimizations here will probably never make it into the official bitcoin github account, will they? Because they typically only make sense for pool operators or am I wrong?
Some of them may not, some of them might. Once I'm convinced this code is stable, I can work on getting some of them into the main distribution. I think the hub mode changes and native long polling could get in. The RPC turbocharge and keepalive fixes could.

Some of the stuff in the latest diff might get in too. Stashing the RPC user/pass, avoiding re-processing partial blocks, faster base64 decode, and faster bin2hex all might get in. (Change 3diff to 4diff to see them.)

Some code should never get in. The JSON bypass code is awful. Perhaps a change to using Jansson is possible. But that's a pretty big change.

Quote
The patches I believe will definitely make it in because that can benefit everybody. (Great work!)
...
Great work JoelKatz. Thank you for that contribution to the community!
Thanks.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: MrSam on July 06, 2011, 10:09:06 AM
Thanks for that report. I'll check the code and see if I can get the SIGUSR1 to be sent faster. I think that will solve half the problem though. While pushpool will issue fewer blocks that will turn into stale shares, bitcoind will still be processing the new block when all the new 'getwork' requests start piling in, and you'll get miner idles instead.

I have a fix for this that has other benefits as well, but the first time I tried to implement it, it was a disaster. I think I know the right way to do it now, having run out of wrong ways to do it already. What I'd like to do is pre-compute the skeleton 'getwork' block when a new block or new transaction comes in, rather than waiting to hijack the next 'getwork' request to do it. But to make this work, the locking has to be exactly right. The funny thing is, in my day job ten years ago, this was exactly the type optimization I specialized in. And now I can't seem to quite get it right.


I solved it with what i call the "pushpool feelgood patch". This patch accepts solutions that he knows are from the prev block, and marks them that way in the database rather then rejecting them to the miner.

This way the users have 0 rejected shares, and they feel good!

Thanks again for all your work, and i'll be following all of your updates :)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on July 06, 2011, 10:15:09 AM
So you're actually lying to your users, about their amount of shares?  :o
Or are you paying for those invalid shares as well?

/ Jim

EDIT: I feel rather bad about that, to be honest.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: MrSam on July 06, 2011, 10:19:53 AM
So you're actually lying to your users, about their amount of shares?  :o
Or are you paying for those invalid shares as well?


Well the idea of pushpool shares at difficulty 1 is that it serves as proof of work, and i think that it is a proof of work, it's not their fault that they submit work that has become invalid.

So yes, i accept this type 'recentstale' for payout.

Why feel bad about this ? proof of work is proof of work ?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: talpan on July 06, 2011, 10:22:24 AM
Hello,

I've patched the official .23 with patch 4.
so far so good, no errors during compiling.
But when i start bitcoind the cpu spikes after 40-50s through the roof, without using it.

regards, talpan


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on July 06, 2011, 10:28:12 AM
So yes, i accept this type 'recentstale' for payout.

So i would be able to resend the same stale-share, over and over again? Or just keep sending stales - it doesn't help the pool at ALL, but i get paid more? :D
No seriously, I'm disappointed.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: MrSam on July 06, 2011, 10:33:45 AM
So yes, i accept this type 'recentstale' for payout.

So i would be able to resend the same stale-share, over and over again? Or just keep sending stales - it doesn't help the pool at ALL, but i get paid more? :D
No seriously, I'm disappointed.

I never said that i accept stales .. only those specific recentstales ..

Happy disappointment day !

https://www.triplemining.com - Probably the best pool in the world


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 06, 2011, 10:36:29 AM
Hello,

I've patched the official .23 with patch 4.
so far so good, no errors during compiling.
But when i start bitcoind the cpu spikes after 40-50s through the roof, without using it.

regards, talpan
Hmm, that's pretty strange. Can you confirm you're using version 0.95?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: talpan on July 06, 2011, 11:47:06 AM
Hi,

Sorry I made a mistake, I'm using a custom .23 with a little patch.
Without this patch and with the patch from everythings fine.
But I can't figure out whats wrong with this patch:

Code:
diff -up orig/rpc.cpp new/rpc.cpp
--- orig/rpc.cpp
+++ new/rpc.cpp
@@ -184,6 +184,149 @@
 }
 
 
+Value BlockToValue(CBlock &block)
+{
+    Object obj;
+    obj.push_back(Pair("hash", block.GetHash().ToString().c_str()));
+    obj.push_back(Pair("version", block.nVersion));
+    obj.push_back(Pair("prev_block", block.hashPrevBlock.ToString().c_str()));
+    obj.push_back(Pair("mrkl_root", block.hashMerkleRoot.ToString().c_str()));
+    obj.push_back(Pair("time", (uint64_t)block.nTime));
+    obj.push_back(Pair("bits", (uint64_t)block.nBits));
+    obj.push_back(Pair("nonce", (uint64_t)block.nNonce));
+    obj.push_back(Pair("n_tx", (int)block.vtx.size()));
+    obj.push_back(Pair("size", (int)::GetSerializeSize(block, SER_NETWORK)));
+
+    Array tx;
+    for (int i = 0; i < block.vtx.size(); i++) {
+    Object txobj;
+
+ txobj.push_back(Pair("hash", block.vtx[i].GetHash().ToString().c_str()));
+ txobj.push_back(Pair("version", block.vtx[i].nVersion));
+ txobj.push_back(Pair("lock_time", (uint64_t)block.vtx[i].nLockTime));
+ txobj.push_back(Pair("size",
+ (int)::GetSerializeSize(block.vtx[i], SER_NETWORK)));
+
+ Array tx_vin;
+ for (int j = 0; j < block.vtx[i].vin.size(); j++) {
+     Object vino;
+
+     Object vino_outpt;
+
+     vino_outpt.push_back(Pair("hash",
+     block.vtx[i].vin[j].prevout.hash.ToString().c_str()));
+     vino_outpt.push_back(Pair("n", (uint64_t)block.vtx[i].vin[j].prevout.n));
+
+     vino.push_back(Pair("prev_out", vino_outpt));
+
+     if (block.vtx[i].vin[j].prevout.IsNull())
+     vino.push_back(Pair("coinbase", HexStr(
+ block.vtx[i].vin[j].scriptSig.begin(),
+ block.vtx[i].vin[j].scriptSig.end(), false).c_str()));
+     else
+     vino.push_back(Pair("scriptSig",
+ block.vtx[i].vin[j].scriptSig.ToString().c_str()));
+     if (block.vtx[i].vin[j].nSequence != UINT_MAX)
+     vino.push_back(Pair("sequence", (uint64_t)block.vtx[i].vin[j].nSequence));
+
+     tx_vin.push_back(vino);
+ }
+
+ Array tx_vout;
+ for (int j = 0; j < block.vtx[i].vout.size(); j++) {
+     Object vouto;
+
+     vouto.push_back(Pair("value",
+     (double)block.vtx[i].vout[j].nValue / (double)COIN));
+     vouto.push_back(Pair("scriptPubKey",
+ block.vtx[i].vout[j].scriptPubKey.ToString().c_str()));
+
+     tx_vout.push_back(vouto);
+ }
+
+ txobj.push_back(Pair("in", tx_vin));
+ txobj.push_back(Pair("out", tx_vout));
+
+ tx.push_back(txobj);
+    }
+
+    obj.push_back(Pair("tx", tx));
+
+    Array mrkl;
+    for (int i = 0; i < block.vMerkleTree.size(); i++)
+    mrkl.push_back(block.vMerkleTree[i].ToString().c_str());
+
+    obj.push_back(Pair("mrkl_tree", mrkl));
+
+    return obj;
+}
+
+
+Value getblockbycount(const Array& params, bool fHelp)
+{
+    if (fHelp || params.size() != 1)
+        throw runtime_error(
+            "getblockbycount height\n"
+            "Dumps the block existing at specified height");
+
+    int64 height = params[0].get_int64();
+    if (height > nBestHeight)
+        throw runtime_error(
+            "getblockbycount height\n"
+            "Dumps the block existing at specified height");
+
+    string blkname = strprintf("blk%d", height);
+
+    CBlockIndex* pindex;
+    bool found = false;
+
+    for (map<uint256, CBlockIndex*>::iterator mi = mapBlockIndex.begin();
+         mi != mapBlockIndex.end(); ++mi)
+    {
+    pindex = (*mi).second;
+ if ((pindex->nHeight == height) && (pindex->IsInMainChain())) {
+ found = true;
+ break;
+ }
+    }
+
+    if (!found)
+        throw runtime_error(
+            "getblockbycount height\n"
+            "Dumps the block existing at specified height");
+
+    CBlock block;
+    block.ReadFromDisk(pindex);
+    block.BuildMerkleTree();
+
+    return BlockToValue(block);
+}
+
+
+Value getblockbyhash(const Array& params, bool fHelp)
+{
+    if (fHelp || params.size() != 1)
+        throw runtime_error(
+            "getblockbyhash hash\n"
+            "Dumps the block with specified hash");
+
+    uint256 hash;
+    hash.SetHex(params[0].get_str());
+
+    map<uint256, CBlockIndex*>::iterator mi = mapBlockIndex.find(hash);
+    if (mi == mapBlockIndex.end())
+        throw JSONRPCError(-18, "hash not found");
+
+    CBlockIndex* pindex = (*mi).second;
+
+    CBlock block;
+    block.ReadFromDisk(pindex);
+    block.BuildMerkleTree();
+
+    return BlockToValue(block);
+}
+
+
 Value getconnectioncount(const Array& params, bool fHelp)
 {
     if (fHelp || params.size() != 0)
@@ -1423,6 +1566,8 @@
 {
     make_pair("help",                  &help),
     make_pair("stop",                  &stop),
+    make_pair("getblockbycount",       &getblockbycount),
+    make_pair("getblockbyhash",        &getblockbyhash),
     make_pair("getblockcount",         &getblockcount),
     make_pair("getblocknumber",        &getblocknumber),
     make_pair("getconnectioncount",    &getconnectioncount),
@@ -2113,6 +2258,7 @@
         if (strMethod == "listtransactions"       && n > 1) ConvertTo<boost::int64_t>(params[1]);
         if (strMethod == "listtransactions"       && n > 2) ConvertTo<boost::int64_t>(params[2]);
         if (strMethod == "listaccounts"           && n > 0) ConvertTo<boost::int64_t>(params[0]);
+        if (strMethod == "getblockbycount"        && n > 0) ConvertTo<boost::int64_t>(params[0]);
         if (strMethod == "sendmany"               && n > 1)
         {
             string s = params[1].get_str();

regards, talpan


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 06, 2011, 11:58:02 AM
That patch looks safe to me. I don't think that explains your issue.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: talpan on July 06, 2011, 01:40:26 PM
I can't find the problem.
I've compiled it many times from scratch and every time the same result.
Can you test it on your end?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Matt Corallo on July 06, 2011, 02:09:54 PM
Some of them may not, some of them might. Once I'm convinced this code is stable, I can work on getting some of them into the main distribution. I think the hub mode changes and native long polling could get in. The RPC turbocharge and keepalive fixes could.

Some of the stuff in the latest diff might get in too. Stashing the RPC user/pass, avoiding re-processing partial blocks, faster base64 decode, and faster bin2hex all might get in. (Change 3diff to 4diff to see them.)

Some code should never get in. The JSON bypass code is awful. Perhaps a change to using Jansson is possible. But that's a pretty big change.
1) Micro-optimizations
Havent had much of a chance to really look through these, but some of them look pull-able others maybe not, but I haven't looked too closely.

2) Native Long Polling Support
This one I'm not sure about, its a useful feature to be sure, but I'm not sure about pulling yet.  When the client codebase gets cleaned up a ton, features like this could very well be pulled, but IMHO not so much yet.

3) A fix from Luke Dash Jr.
Almost certainly can be pulled.

4) Hub Mode
No, this is really not a good change for several reasons.  A. the percent of nodes accepting incoming connections is already low enough, encouraging people to start making a ridiculous number of outgoing connections and filling up the few nodes who do accept incoming connection's connections slots is not a good idea at all.  B. Not even pool operators should be using this, a pool bitcoind should be accepting incoming connections, and if you do that, you will get plenty of connections really fast.  If you want to make sure your blocks are more likely to get accepted quicker, find out the IPs of other large miner nodes and addnode those instead of just blindly making a ton of connections as this patch does.

5) RPC Turbocharge
Absolutely, there is already a asio patch for this waiting as a pull request, so some version of asio/keepalive support/threaded RPC will absolutely be merged at some point in the not-too-distant future.

6) Resource Leak Fix
Absolutely, thanks somebadger for the pull.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 06, 2011, 02:32:36 PM
No, this is really not a good change for several reasons.  A. the percent of nodes accepting incoming connections is already low enough, encouraging people to start making a ridiculous number of outgoing connections and filling up the few nodes who do accept incoming connection's connections slots is not a good idea at all.
That's not a point I had considered. I think I may make some adjustments.

Quote
B. Not even pool operators should be using this, a pool bitcoind should be accepting incoming connections, and if you do that, you will get plenty of connections really fast.
Yes, but from lots of small nodes. That doesn't help you very much.

Quote
If you want to make sure your blocks are more likely to get accepted quicker, find out the IPs of other large miner nodes and addnode those instead of just blindly making a ton of connections as this patch does.
Won't those generally be the nodes that are accepting incoming connections?

Thanks for the feedback. You've given me a few ideas for ways to make this work better.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: MrSam on July 06, 2011, 02:34:51 PM

Quote
If you want to make sure your blocks are more likely to get accepted quicker, find out the IPs of other large miner nodes and addnode those instead of just blindly making a ton of connections as this patch does.
Won't those generally be the nodes that are accepting incoming connections?

I tend to use https://en.bitcoin.it/wiki/Fallback_Nodes , but the wikibot doesnt seem to update it.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Matt Corallo on July 06, 2011, 02:39:26 PM
B. Not even pool operators should be using this, a pool bitcoind should be accepting incoming connections, and if you do that, you will get plenty of connections really fast.
Yes, but from lots of small nodes. That doesn't help you very much.
True, but those nodes are probably only connected to other large nodes who listen, making two hops is really very fast anyway.

If you want to make sure your blocks are more likely to get accepted quicker, find out the IPs of other large miner nodes and addnode those instead of just blindly making a ton of connections as this patch does.
Won't those generally be the nodes that are accepting incoming connections?
Yep, but if you blindly make a ton of connections, you are still fairly unlikely to make connections to other miners, better to just addnode the best ones and make 10 connections instead of making 30 and only getting a couple large miners.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 06, 2011, 02:42:42 PM
I can't find the problem.
I've compiled it many times from scratch and every time the same result.
Can you test it on your end?
Are you saying it works fine with just my patches but if you add the 'getblock' patches, then you have problems? Are you using the 'getblock*' RPC calls that the patch adds? It looks to me like they access mapBlockIndex without holding the appropriate lock. Can you try this change: (And if this fixes it, you may want to share this fix with whoever wrote those patches!)

--- patch/rpc.cpp       2011-07-06 07:39:21.647871060 -0700
+++ rpc.cpp     2011-07-06 07:41:38.033681011 -0700
@@ -277,9 +277,13 @@ Value getblockbycount(const Array& param
 
     string blkname = strprintf("blk%d", height);
 
+    CBlock block;
     CBlockIndex* pindex;
     bool found = false;
 
+    CRITICAL_BLOCK(cs_main)
+    {
+
     for (map<uint256, CBlockIndex*>::iterator mi = mapBlockIndex.begin();
          mi != mapBlockIndex.end(); ++mi)
     {
@@ -295,9 +299,9 @@ Value getblockbycount(const Array& param
             "getblockbycount height\n"
             "Dumps the block existing at specified height");
 
-    CBlock block;
     block.ReadFromDisk(pindex);
     block.BuildMerkleTree();
+    }
 
     return BlockToValue(block);
 }
@@ -312,6 +316,10 @@ Value getblockbyhash(const Array& params
 
     uint256 hash;
     hash.SetHex(params[0].get_str());
+    CBlock block;
+
+    CRITICAL_BLOCK(cs_main)
+    {
 
     map<uint256, CBlockIndex*>::iterator mi = mapBlockIndex.find(hash);
     if (mi == mapBlockIndex.end())
@@ -319,9 +327,9 @@ Value getblockbyhash(const Array& params
 
     CBlockIndex* pindex = (*mi).second;
 
-    CBlock block;
     block.ReadFromDisk(pindex);
     block.BuildMerkleTree();
+    }
 
     return BlockToValue(block);
 }


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: talpan on July 06, 2011, 05:19:17 PM
Thanks a lot for your help, so far it is all running perfect.
But I think the problem was somewhere else, after a little bit more testing I got an exception.
It had something to do with libboost, I did reinstall that and the problem was solved.

I'll send you small donation(haven't that much, yet ;)) in the next days.

And the patch is from jgarzik, I'll make a pull with your patch and of course your credits of course.

regards, talpan


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: somebadger on July 06, 2011, 11:13:00 PM
fyi: renamed the repository to bitcoin4pools, new url is:

https://github.com/somebadger/bitcoin4pools (https://github.com/somebadger/bitcoin4pools)

Any addition suggestions are welcome, will start a new thread later today and reference this thread and give credit to JoelKatz :)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 06, 2011, 11:57:26 PM

I have some updates available at:
http://davids.webmaster.com/~davids/updates.diff.txt

This includes a performance improvement for ToHex, some style cleanups, some tuning changes to hub mode to reduce outbound connections, and a huge change to 'getwork'.

The change to 'getwork' is one I attempted before with not so good results, but I've rethought the way it's implemented, and this is much simpler. The idea is to update the 'skeleton block' used to build 'getwork' requests when a new transaction or block is received rather than waiting for the next 'getwork' request to come along to do it. The reason this makes a difference is that it means the 'getwork' request won't have to wait for the code handling the new block/transaction to release the 'cs_main' lock. Testing shows this really smooths out the 'spikes' in the 'getwork' response time.

However, this is a risky change. It is invasive to not just the code to get a work unit but also the code to confirm a valid block. Some people may wish to apply just the ToHex change and the cleanups.



Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: MrSam on July 07, 2011, 12:36:52 AM
I'll run it on testnet to see what it gives


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: TeraPool on July 07, 2011, 04:15:20 AM
fyi: renamed the repository to bitcoin4pools, new url is:

https://github.com/somebadger/bitcoin4pools (https://github.com/somebadger/bitcoin4pools)

Any addition suggestions are welcome, will start a new thread later today and reference this thread and give credit to JoelKatz :)

Thanks for that!

Please make a post of said new through in this post, as I am watching this post for some time now as I assume many other pool creators are.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Caesium on July 07, 2011, 10:13:50 AM
JoelKatz,

Managing to get segfaults when I send getwork to bitcoind with this set of patches now, backtrace below:

Code:
(gdb) bt
#0  0x00007f710a6e8394 in pthread_mutex_lock ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x0000000000447103 in boost::interprocess::interprocess_recursive_mutex::lock (this=<value optimized out>)
    at /usr/include/boost/interprocess/sync/posix/interprocess_recursive_mutex.hpp:53
#2  0x0000000000477d98 in Enter (this=0x0, nIndex=@0x9c1968, keypool=...)
    at util.h:245
#3  CCriticalBlock (this=0x0, nIndex=@0x9c1968, keypool=...) at util.h:260
#4  CWallet::ReserveKeyFromKeyPool (this=0x0, nIndex=@0x9c1968, keypool=...)
    at wallet.cpp:1041
#5  0x000000000047889c in CReserveKey::GetReservedKey (this=0x9c1960)
    at wallet.cpp:1117
#6  0x0000000000458562 in CreateNewBlock (reservekey=...) at main.cpp:2713
#7  0x000000000048f630 in GetWorkBlock (
    pmidstate=0x7f70feff9ef0 "C\377\376p\177", pdata=0x7f70feff9e30 "#",
    phash1=0x7f70feff9eb0 "\250\223\f\003", hashTarget=...) at rpc.cpp:1359
#8  0x000000000048fca2 in FastGetWork (id=...) at rpc.cpp:1461
#9  0x000000000049d03b in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:2031
#10 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#11 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
---Type <return> to continue, or q <return> to quit---
    at rpc.cpp:1981
#12 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#13 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#14 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#15 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#16 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#17 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#18 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#19 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#20 0x00007f710a6e5d8c in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#21 0x00007f7109c9004d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#22 0x0000000000000000 in ?? ()
(gdb)

This happening both with a fresh blockchain downloaded and a new wallet.

Let me know what I can give you to help debug :)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 07, 2011, 03:56:12 PM

What build is this exactly?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on July 07, 2011, 03:59:25 PM
wallet.cpp is in the github version - no the stable one.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on July 07, 2011, 04:21:28 PM

I have some updates available at:
http://davids.webmaster.com/~davids/updates.diff.txt

This includes a performance improvement for ToHex, some style cleanups, some tuning changes to hub mode to reduce outbound connections, and a huge change to 'getwork'.

The change to 'getwork' is one I attempted before with not so good results, but I've rethought the way it's implemented, and this is much simpler. The idea is to update the 'skeleton block' used to build 'getwork' requests when a new transaction or block is received rather than waiting for the next 'getwork' request to come along to do it. The reason this makes a difference is that it means the 'getwork' request won't have to wait for the code handling the new block/transaction to release the 'cs_main' lock. Testing shows this really smooths out the 'spikes' in the 'getwork' response time.

However, this is a risky change. It is invasive to not just the code to get a work unit but also the code to confirm a valid block. Some people may wish to apply just the ToHex change and the cleanups.



Quote
jine@bitcoins:~$ uptime
 18:20:27 up 9 days,  6:45,  3 users,  load average: 0.00, 0.19, 0.22

I say no more.

We had some serious issues a while ago with a lot of disconnects, so i took a shot and applied this....
0.00 load.. seriously? :P :D


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on July 07, 2011, 04:23:11 PM
... and now it froze.

Reverted back to old bitcoind... :(

EDIT:
Quote
received: addr (31 bytes)
received: inv (37 bytes)
  got inventory: tx f9e2f6b050644cc94341  have
sending: addr (31 bytes)
received: addr (31 bytes)
sending: inv (37 bytes)
sending: addr (31 bytes)
sending: inv (37 bytes)
sending: inv (37 bytes)
received: addr (31 bytes)
received: addr (61 bytes)
sending: addr (31 bytes)
sending: inv (37 bytes)
sending: inv (37 bytes)
BitcoinMiner:
proof-of-work found
  hash: 00000000000004f00d620b3a2fd5e4ba19043e2053e9f4060a3795d2d00c5155
target: 0000000000000abbcf0000000000000000000000000000000000000000000000
CBlock(hash=00000000000004f00d62, ver=1, hashPrevBlock=0000000000000a6772c7, hashMerkleRoot=7e05d00288, nTime=1310055675, nBits=1a0abbcf, nNonce=568783248, vtx=37)
  CTransaction(hash=14b69a7cab, ver=1, vin.size=1, vout.size=1, nLockTime=0)
    CTxIn(COutPoint(0000000000, -1), coinbase 04cfbb0a1a035fc601)
    CTxOut(nValue=50.07110000, scriptPubKey=044dcd4ee79a23daccd12df3450a23)
  CTransaction(hash=da5496d248, ver=1, vin.size=16, vout.size=2, nLockTime=0)
    CTxIn(COutPoint(9c55619ba7, 1), scriptSig=3045022100f6c19dcd06577d)
    CTxIn(COutPoint(8d7f3d9c1d, 1), scriptSig=304602210095f7f99816769e)
    CTxIn(COutPoint(3255f7943d, 0), scriptSig=304602210089bbaa23e5be3c)
    CTxIn(COutPoint(a8be29ad98, 1), scriptSig=3045022100bab4ddaaf7da9c)
    CTxIn(COutPoint(abdc5e1a70, 1), scriptSig=30450220442f40e8b65ef407)

... new block?

    CTxOut(nValue=0.01000000, scriptPubKey=OP_DUP OP_HASH160 91d45c1f79fa)
  CTransaction(hash=a51043e11b, ver=1, vin.size=2, vout.size=2, nLockTime=0)
    CTxIn(COutPoint(6701c1b7a5, 0), scriptSig=30450220224680e1db9d648d)
    CTxIn(COutPoint(60cc6580e0, 0), scriptSig=304402207cb98fb1870604cd)
    CTxOut(nValue=0.00775252, scriptPubKey=OP_DUP OP_HASH160 c1e8b98c8c3d)
    CTxOut(nValue=0.02000000, scriptPubKey=OP_DUP OP_HASH160 401ce258491c)
  vMerkleTree: 14b69a7cab da5496d248 035078318f ea17127588 c0c344418b eef1e852f6 649507f180 ac905c95d4 0842b073a4 305372759d 9a6e764565 6794e2e09c d76296138f fb2fbfb31b 827e418d7f b142831de3 cd499e996d 5b43802900 68971974ef 29a1b4054b 1ca1ee01f1 f7abc8ab40 9f5768fc54 47394$
07/07/11 16:21 generated 50.0711
keypool keep 10145
AddToWallet 14b69a7cab  new
AddToWallet 68971974ef  update
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
accepted connection 178.47.108.93:60078
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo

* was unresponsive, so i runned ./bitcoind stop*


ThreadRPCServer method=stop
DBFlush(false)
addr.dat refcount=0
addr.dat flush
ThreadSocketHandler exiting
ThreadOpenConnections exiting
ThreadRPCServer exiting
blkindex.dat refcount=1
wallet.dat refcount=25
StopNode()







* restarted *





Bitcoin version 0.3.23-beta
Default data directory /root/.bitcoin


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 07, 2011, 05:13:31 PM
Getting the locking right on this patch (the one to precompute the skeleton block) is proving to be a pain.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on July 07, 2011, 05:22:47 PM
Is it any other way to do it? Possibly by not requesting work at all? Generating it in pushpoold perhaps?
I don't have any other ideas.

Both me and Eleuthria is talking about suicide on IRC (not serious tho... yet ;P) because of all the issues.. (Not only bitcoind-related, but server/ddos/stability/stupid users etc)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 07, 2011, 05:27:21 PM
I've been there with the DDoS attacks, I know how awful that is.

Giel de Nijs suggested a binary protocol between bitcoind and pushpool. This would eliminate all the HTTP/JSON/hex/base64 nonsense entirely. It would also permit pushpool to request work units in blocks of, say, 10, reducing the handoffs between the two programs. This would mean long polling could be implemented in a saner way than a signal as well.

On the bright side, my current secret project will make all mining pools obsolete anyway and drastically reduce the feasibility of the 51% attack so there's some hope. ;)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Caesium on July 07, 2011, 06:47:37 PM

What build is this exactly?


It's a git clone of bitcoin4pools as posted a while back:
https://github.com/somebadger/bitcoin4pools

As of commit https://github.com/somebadger/bitcoin4pools/commit/fbfd3ecbd322edd2c478135fb43e63b8ad4edeab

Which I notice happens to mention CReserveKey, which is also in the backtrace..


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Zagitta on July 07, 2011, 11:21:06 PM
Is it any other way to do it? Possibly by not requesting work at all? Generating it in pushpoold perhaps?
I don't have any other ideas.

Both me and Eleuthria is talking about suicide on IRC (not serious tho... yet ;P) because of all the issues.. (Not only bitcoind-related, but server/ddos/stability/stupid users etc)

That's an idea i've had myself and i'm trying to implement it in my own backend however i still haven't quit understood what exactly the data is calculated from (besides that the previous block is involved somehow) but once i get it, it should be pretty trival to implement considering the code for handling the block chain is already ported to C#: http://code.google.com/p/bitcoinsharp/

So if anyone got a simple explenation or links that explains the getwork process please share :)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: somebadger on July 11, 2011, 01:29:54 AM

What build is this exactly?


It's a git clone of bitcoin4pools as posted a while back:
https://github.com/somebadger/bitcoin4pools

As of commit https://github.com/somebadger/bitcoin4pools/commit/fbfd3ecbd322edd2c478135fb43e63b8ad4edeab

Which I notice happens to mention CReserveKey, which is also in the backtrace..

I have updated the repo, reverted back to .23 with patches and updates, plus --help additions.
.24 had some issues will get back on those once i get some time.
https://github.com/somebadger/bitcoin4pools/commits/v0.3.23-pool (https://github.com/somebadger/bitcoin4pools/commits/v0.3.23-pool)

ps. running now for 22hrs without a glitch :)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Caesium on July 12, 2011, 04:47:19 PM
Looks good here too, cheers :)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: m0mchil on July 12, 2011, 08:38:55 PM
Even if too late in this thread, I'd like to make some comments. Partly because I feel some guilt over the notorious 'getwork' RPC call.

It was made with only one purpose - to experiment with mining outside of Satoshi's code. I never imagined it will feed heavy loaded servers in the way it is doing it now. Sadly, it made pooled mining possible and at the same time allowed de-democratization of the mining process. I believe this can be fixed.

Right now, classic 'getwork' is re-processing all transactions whenever there are new ones and 60 seconds have passed or when there is new block(1). Worse, because each worker needs its own hash space, the Merkle tree is recalculated entirely with each request(2). When the size of transaction pool (unconfirmed transactions) gets really large, this becomes unfeasible. There were episodes with significant numbers of spam transactions which proved this.

Some months ago I made https://github.com/m0mchil/bitcoin/tree/poolmode (https://github.com/m0mchil/bitcoin/tree/poolmode)

About (1), transaction processing was moved out of the RPC thread (to main.cpp, ProcessTransactions) to make 'getwork' always return as fast as possible. For (2) UpdateMerkleTree was introduced to allow only specific branch to be rebuilt (specifically the first one). Bitcoind was creating new thread handle for each connection accepted (I guess it was needed for connection timeout guard) - this was removed (see rpc.cpp, around 'boost::thread api_caller') because with pools the server is always used locally, by trusted process(es). Even still single threaded, 'getwork' performance improved drastically.

But it is time for a new scheme. I see Gavin's monitorX (https://github.com/gavinandresen/bitcoin-git/tree/monitorreceived) patch as a good candidate. We need something like 'monitorTransactionPool' to push whenever there is change in the set of transactions currently ready to be included in a block.

Also, pools should be changed to allow miners to just prove they included pool's coin base in the block they solve. This is possible by sending the transaction with pool's address in it and the next Merkle branch. Then miners will have complete control over which transactions to include and which block chain to build on.

I am intending to have this implemented soon.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 12, 2011, 09:19:47 PM
Also, pools should be changed to allow miners to just prove they included pool's coin base in the block they solve. This is possible by sending the transaction with pool's address in it and the next Merkle branch. Then miners will have complete control over which transactions to include and which block chain to build on.
This is a great idea, however, it requires changes to *everything*. Bitcoin will need changes, pool managers will need changes, and mining programs will need changes.

I'm not sure how the proof of work flow would go though. If the pool only gets the address and the branch, it cannot submit the block to the public chain. Are you expecting the miner to do that alone? Or are you suggesting different paths for solved block versus share found?

This will force pool managers to rely on the miners to work on useful branches, and it will mean that if a miner has a poor connection, he may be submitting useless shares for which he will get full credit.

I think this needs some subtle changes to address those issues. But I think the idea of letting the miners decide which transactions to include is a great one. And avoiding recomputing the entire merkle tree on each 'getwork' call will eliminate the biggest CPU sucker left in 'getwork' after my changes.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: m0mchil on July 12, 2011, 09:41:10 PM

I'm not sure how the proof of work flow would go though. If the pool only gets the address and the branch, it cannot submit the block to the public chain. Are you expecting the miner to do that alone? Or are you suggesting different paths for solved block versus share found?

This will force pool managers to rely on the miners to work on useful branches, and it will mean that if a miner has a poor connection, he may be submitting useless shares for which he will get full credit.

This is true even now - miners can fail to submit the precious one true solution due to many reasons. Even on purpose. And yes, I expect the miner to submit his blocks.

Quote
... And avoiding recomputing the entire merkle tree on each 'getwork' call will eliminate the biggest CPU sucker left in 'getwork' after my changes.

Feel free to pull this in your code. My patch is tested against pushpoold and seems to work fine.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 12, 2011, 09:58:11 PM
Something else occurred to me: If we make these changes, we can allow the miner to decide how much he wants to pool mine and how much he wants to solo mine. Instead of defining each share submitted as 1, we can define each share submitted as the fraction of the revenue for the block sent to the pool's account. So a miner could split the money 50/50 and get half shares from the pool but make 25BTC if he mines the block himself, or anywhere in between.

In fact, he could even mine for more than one pool at a time, accumulating fractional shares from each pool.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 14, 2011, 11:27:37 AM
Is anyone else having problems with somebadgers git pull of this?

I had a bunch of socket recv error 104 errors in the debug.log and bitcoind was not serving any work this morning.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 14, 2011, 11:53:55 PM
I'm also seeing this in the console log now and again:

[2011-07-14 23:53:5.926549] JSON-RPC call failed: (null)
[2011-07-14 23:53:5.926566] submit_work json_rpc_call failed


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 15, 2011, 02:42:20 AM
Locked up again...

Somebadger: Your git is definitely not stable, sadly.  I'm trying now without any -hub command line, we'll see what happens there.



Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: phorensic on July 15, 2011, 03:33:14 AM
I think it's because somebadger is patching against 0.3.24 and JoelKatz started with 0.3.23.  Correct me if I'm wrong.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 15, 2011, 04:20:09 AM
It's my understanding Somebadger backed off to .23 in the git repository.  Is that not correct?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: btcmonkey on July 15, 2011, 05:49:41 PM
Hello,

I grabbed bitcoin v0.3.23 and applied bitcoin-4diff.txt and updates.diff.txt.

This is on 64bit CentOS 5.5.

I am now generating bitcoind cores every hour on the hour.  I am fairly new to troubleshooting this type of thing, but a backtrace shows:

Code:
Core was generated by `/home/bitcoin/bitcoind -testnet -conf=/etc/bitcoin.conf -daemon -pollpidfile=/v'.
Program terminated with signal 6, Aborted.
#0  0x00000039f9e30265 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00000039f9e30265 in raise () from /lib64/libc.so.6
#1  0x00000039f9e31d10 in abort () from /lib64/libc.so.6
#2  0x00000039fd2bed14 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3  0x00000039fd2bce16 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x00000039fd2bce43 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5  0x00000039fd2bcec5 in __cxa_rethrow () from /usr/lib64/libstdc++.so.6
#6  0x000000000040a40a in PrintException (pex=<value optimized out>, pszThread=<value optimized out>) at util.cpp:659
#7  0x00000000004adbee in ThreadRPCServer (parg=0x0) at rpc.cpp:1897
#8  0x00000039faa0673d in start_thread () from /lib64/libpthread.so.0
#9  0x00000039f9ed44bd in clone () from /lib64/libc.so.6

my bitcoind command line is:

/usr/local/sbin/bitcoind -conf=/etc/bitcoin.conf -daemon -pollpidfile=/var/run/pushpoold/pushpoold.pid

with bitcoind.conf only containing rpc user and pass.

Does anyone have any suggestions as to what I can do to resolve this issue?

Thanks much,
   btcmonkey



Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 15, 2011, 06:04:12 PM
Can you paste the code from your 'rpc.cpp' file around line 1897 (say five lines before and five after). And please identify exactly which line is 1897.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 15, 2011, 09:14:46 PM
Anyone got any ideas what's going on on my end?

Can't really run a backtrace since it's not crashing exactly.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: btcmonkey on July 16, 2011, 12:20:20 AM
Can you paste the code from your 'rpc.cpp' file around line 1897 (say five lines before and five after). And please identify exactly which line is 1897.


Here is a snip from rpc.cpp:
Code:

void ThreadRPCServer(void* parg)
{
    IMPLEMENT_RANDOMIZE_STACK(ThreadRPCServer(parg));
    try
    {
        vnThreadsRunning[4]++;
        ThreadRPCServer2(parg);
        vnThreadsRunning[4]--;
    }
    catch (std::exception& e) {
        vnThreadsRunning[4]--;
        PrintException(&e, "ThreadRPCServer()");
    } catch (...) {
        vnThreadsRunning[4]--;
        PrintException(NULL, "ThreadRPCServer()");
    }
    printf("ThreadRPCServer exiting\n");
}

Line 1897 is: PrintException(&e, "ThreadRPCServer()");



Just to verify that I am doing things right, to create this file I:

Code:
git clone http://github.com/bitcoin/bitcoin/ davids
cd davids
git checkout v0.3.23
cd src
patch  < ~/src/bitcoin/bitcoin-4diff.txt
patch  < ~/src/bitcoin/updates.diff.txt
then built bitcoind normally.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 16, 2011, 02:45:01 AM
This is one of the annoying things about C++ exception handling. The exception was caught, and that obscured the information needed to find the code that generated it. :(

All we can tell from that is that the RPC code threw an exception. This could be for reasons that really aren't the code's fault, such as running out of memory at a critical point, or (more likely) they could be due to bugs in the code.

One thing you can try -- use the 'up' command until you get to level 7, the 'ThreadRPCServer' call. And type 'print e'. If for some reason that doesn't work, you can try level 6, 'PrintException' and the command 'print pszMessage'.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 16, 2011, 04:26:08 AM
I put in some debug code, why is this failing?

[2011-07-16 04:22:57.656676] JSON-RPC call failed: (null)
[2011-07-16 04:22:57.656694] submit_work json_rpc_call failed
[2011-07-16 04:22:57.656701] curl: P*, srv.rpc_url: http://127.0.0.1:8332/, srv.rpc_userpass: xxx:xxx, s: {"method": "getwork", "params": [ "000000017389ec1cfb159619a8182312807a8d7a77041d1d835f6377000008f00000000020988cf 8191cab5a828a8bd44b73c7310ab418f1c469f9d836d66d8dd6cf139c4e2112071a0abbcf6a64c9 c400000000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000000000" ], "id":1}

The above submit_work call is what's failing with JoelKatz patch and update applied.  it works fine without the patch.  This comes from the submit_work() function in msg.c



Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 16, 2011, 04:48:54 AM
I put in some debug code, why is this failing?

[2011-07-16 04:22:57.656676] JSON-RPC call failed: (null)
[2011-07-16 04:22:57.656694] submit_work json_rpc_call failed
[2011-07-16 04:22:57.656701] curl: P*, srv.rpc_url: http://127.0.0.1:8332/, srv.rpc_userpass: xxx:xxx, s: {"method": "getwork", "params": [ "000000017389ec1cfb159619a8182312807a8d7a77041d1d835f6377000008f00000000020988cf 8191cab5a828a8bd44b73c7310ab418f1c469f9d836d66d8dd6cf139c4e2112071a0abbcf6a64c9 c400000000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000000000" ], "id":1}

The above submit_work call is what's failing with JoelKatz patch and update applied.  it works fine without the patch.  This comes from the submit_work() function in msg.c

Nice catch!!!! This is a bug in the code that processes a found block. Fortunately, we are correctly processing the block, so you are getting credit for it. However, we bungle getting the information back to the caller and return nothing. Here's the fix:

--- rpc.cpp~    2011-07-10 04:37:16.000000000 -0700
+++ rpc.cpp     2011-07-15 21:47:58.097050116 -0700
@@ -1461,7 +1461,7 @@ Value getwork(const Array& params, bool
             npblock->vtx[0].vin[0].scriptSig = CScript() << pblock->nBits << CBigNum(nExtraNonce);
             npblock->hashMerkleRoot = npblock->BuildMerkleTree();
 
-            Value ret = CheckWork(npblock, reservekey);
+            ret = CheckWork(npblock, reservekey);
         }
         return ret;
     }



Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 16, 2011, 04:59:15 AM
Made the changes, I will see how it goes.

I get this when doing a kill on bitcoind BTW:

terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >'
  what():  mutex: Invalid argument

It looks like bitcoind is not exiting cleanly when it gets a SIG to die.



Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 16, 2011, 05:09:07 AM
Made the changes, I will see how it goes.

I get this when doing a kill on bitcoind BTW:

terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >'
  what():  mutex: Invalid argument

It looks like bitcoind is not exiting cleanly when it gets a SIG to die.
The most likely issue is in this code:

 { // CAUTION: Raising the delay will slow connection accept
     boost::posix_time::time_duration wait_duration = boost::posix_time::millisec(250);
     boost::unique_lock<boost::mutex> lock(mWorkNotification);
     if(!fWorkFound)
         cvWorkNotification.timed_wait(lock, wait_duration); // ** HERE **
 }

I wonder if I have to enclose that in a try/catch block or it doesn't release the mutex if the timed wait is interrupted. I'll try to track it down.

You can safely replace that entire block of code with 'Sleep(250);' if you want. Network latency (the time between when you get a block or transaction and the time you can pass it to neighboring nodes) will be slightly higher -- but still better than without the hub patches. But if nothing else, it would tell me if that block is causing the issue.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: btcmonkey on July 16, 2011, 05:14:49 AM
This is one of the annoying things about C++ exception handling. The exception was caught, and that obscured the information needed to find the code that generated it. :(

All we can tell from that is that the RPC code threw an exception. This could be for reasons that really aren't the code's fault, such as running out of memory at a critical point, or (more likely) they could be due to bugs in the code.

One thing you can try -- use the 'up' command until you get to level 7, the 'ThreadRPCServer' call. And type 'print e'. If for some reason that doesn't work, you can try level 6, 'PrintException' and the command 'print pszMessage'.


So it turns out I made a stupid mistake.  As part of a test some days ago, I had created a cron job that stopped and restarted bitcoind on an hourly basis.  It appears that I wasn't waiting long enough for the bitcoind stop command to close out bitcoind.  The 'trying to start bitcoind when it was in the midst of still shutting down' seems to have caused the cores.

I'm sorry for wasting your time, but really appreciate what you have taught me about debugging/troubleshooting these issues.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 16, 2011, 05:42:15 AM
I'm sorry for wasting your time, but really appreciate what you have taught me about debugging/troubleshooting these issues.
No problem. That's par for the course. I've made more stupid mistakes that have wasted other people's times than I can count.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 16, 2011, 09:12:45 PM
Well, I tried a fresh copy of .23 and applied the update and diff4, still same problem as somebadgers repository.  System will basically stop responding after ~12 hours and bitcoind needs to be restarted and everything picks right back up.

How can I trouble shoot this?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 16, 2011, 09:15:14 PM
Well, I tried a fresh copy of .23 and applied the update and diff4, still same problem as somebadgers repository.  System will basically stop responding after ~12 hours and bitcoind needs to be restarted and everything picks right back up.
What is this update?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 16, 2011, 09:19:10 PM
Your updates from post http://forum.bitcoin.org/index.php?topic=22585.msg334074#msg334074


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 16, 2011, 09:46:14 PM
Your updates from post http://forum.bitcoin.org/index.php?topic=22585.msg334074#msg334074
Ahh, okay. I think there's a locking bug in those updates. I'm working on two things:

1) A new 4-diff based on 0.3.24 that includes all the updates that are believed to be safe.

2) A new diff based on 0.3.24 that includes even the getwork pre-compute update that is suspected to be responsible for deadlocks, but hopefully with the deadlock issue fixed.

1 should be ready soon. I still haven't done final auditing of the diff and testing to make sure it works.
I just put up a preview here:
http://davids.webmaster.com/~davids/preview.diff

2 may take a bit longer.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 16, 2011, 10:10:09 PM
Cool, I will try it out and see what happens.  You might consider removing the upnp junk in the diff as well, since it's an extra dependency that is a) a pain in the ass and b) completely useless in this application.  I remove it from the makefile before compiling.

Just to clarify, is this preview a potential fix for the lockup issue I'm experiencing or is that part of number 2?

When you say ready soon do you mean today or within a couple days?  I don't want to start digging around in the code  and making a nuisance of myself if there's going to be some changes within the next few hours as far as that goes.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 16, 2011, 10:12:13 PM
When you say ready soon do you mean today or within a couple days?  I don't want to start digging around in the code  and making a nuisance of myself if there's going to be some changes within the next few hours as far as that goes.
1 should be ready today. All that's left is a final audit and testing to make sure I can't break it. There are unlikely to be any significant changes and there's a good chance there will be no changes at all. Testing and reports are very helpful, so don't worry about making a nuisance of yourself. "It worked for me" is extremely helpful because it helps me get closer to the confidence level needed for release. "It didn't work for me" is extremely helpful because I hate to find issues after I release.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 16, 2011, 10:19:01 PM
Ok, well, with your preview.diff, bitcoind won't start.  Here is debug.log errors:

Bitcoin version 0.3.24-beta
Default data directory /home/xxx/.bitcoin
Bound to port 8333
Loading addresses...
dbenv.open strLogDir=/home/xxx/.bitcoin/database strErrorFile=/home/xxx/.bitcoin/db.log
Loaded 356873 addresses
 addresses              1032ms
Loading block index...
LoadBlockIndex(): hashBestChain=00000000000002b99ddf  height=136616
 block index            1914ms
Loading wallet...
nFileVersion = 32400
fGenerateBitcoins = 0
nTransactionFee = 0
addrIncoming = 255.255.255.255:8333
fMinimizeToTray = 0
fMinimizeOnClose = 0
fUseProxy = 0
addrProxy = 127.0.0.1:9050
 wallet                  118ms
Done loading
mapBlockIndex.size() = 136636
nBestHeight = 136616
mapKeys.size() = 172
setKeyPool.size() = 101
mapPubKeys.size() = 172
mapWallet.size() = 190
mapAddressBook.size() = 2
Loading addresses from DNS seeds (could take a while)
AddAddress(84.49.174.161:8333)
48 addresses found from DNS seeds
sending: version (85 bytes)
ThreadRPCServer started
ipv4 eth0: x.x.x.x
addrLocalHost = x.x.x.x:8333
ThreadSocketHandler started
ThreadIRCSeed started
ThreadOpenConnections started
ThreadMessageHandler started
trying connection 67.172.181.225:8333 lastseen=-0.1hrs lasttry=-364126.2hrs
IRC :irc.lechat.ir NOTICE AUTH :*** Looking up your hostname...
connected 67.172.181.225:8333
sending: version (85 bytes)



bitcoind exits at this point


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 16, 2011, 10:21:41 PM
I am going to pull a fresh copy of .24 and reapply just to be sure I didn't do something odd - but the patch applied just fine.


EDIT - whoops... I guess you are patching against the tar and not the repo?  The above report is off the tar and it applied fine.  Just tried against the repo and it didn't like that.



Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 16, 2011, 10:31:45 PM
Ok, confirmed that a fresh copy of the .24 tar file with your preview gives the above result.  Here's another debug.log output from the fresh copy:

Bitcoin version 0.3.24-beta
Default data directory /home/xxx/.bitcoin
Bound to port 8333
Loading addresses...
dbenv.open strLogDir=/home/xxx/.bitcoin/database strErrorFile=/home/xxx/.bitcoin/db.log
Loaded 357138 addresses
 addresses               942ms
Loading block index...
LoadBlockIndex(): hashBestChain=0000000000000a1fe384  height=136619
 block index            1925ms
Loading wallet...
nFileVersion = 32400
fGenerateBitcoins = 0
nTransactionFee = 0
addrIncoming = 255.255.255.255:8333
fMinimizeToTray = 0
fMinimizeOnClose = 0
fUseProxy = 0
addrProxy = 127.0.0.1:9050
 wallet                  175ms
Done loading
mapBlockIndex.size() = 136639
nBestHeight = 136619
mapKeys.size() = 172
setKeyPool.size() = 101
mapPubKeys.size() = 172
mapWallet.size() = 190
mapAddressBook.size() = 2
Loading addresses from DNS seeds (could take a while)
AddAddress(62.155.236.249:8333)
AddAddress(109.75.176.193:8333)
AddAddress(174.120.185.74:8333)
AddAddress(69.163.132.101:8333)
AddAddress(178.79.147.99:8333)
AddAddress(91.85.220.84:8333)
48 addresses found from DNS seeds
sending: version (85 bytes)
ThreadRPCServer started
ipv4 eth0: x.x.x.x
addrLocalHost = x.x.x.x:8333
ThreadSocketHandler started
ThreadIRCSeed started
ThreadOpenConnections started
ThreadMessageHandler started
IRC :pelican.heliacal.net NOTICE AUTH :*** Looking up your hostname...
IRC :pelican.heliacal.net NOTICE AUTH :*** Couldn't look up your hostname
IRC SENDING: NICK x265309750^M
IRC SENDING: USER x265309750 8 * : x265309750^M
trying connection 213.111.82.119:8333 lastseen=-0.2hrs lasttry=-364126.5hrs
IRC :pelican.heliacal.net 001 x265309750 :Welcome to the LFNet Internet Relay Chat Network x265309750
IRC :pelican.heliacal.net 002 x265309750 :Your host is pelican.heliacal.net[173.246.103.92/6667], running version hybri
d-7.2.3
IRC :pelican.heliacal.net 003 x265309750 :This server was created Jun 28 2011 at 14:26:11
IRC :pelican.heliacal.net 004 x265309750 pelican.heliacal.net hybrid-7.2.3 CDGabcdfgiklnorsuwxyz biklmnopstveI bkloveI
connected 213.111.82.119:8333
sending: version (85 bytes)


Bitcoind exits at this point.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 16, 2011, 10:31:49 PM
It is against the 0.3.24 release. I just made a small, but very critical, fix. So redownload it. (That's likely the problem in your post above.)

I've finished my own testing and auditing. So if the latest works for you, I'm ready to release it.

Performance is about 3,000 getwork's per second on my test machine (a Core 2 Quad Q9550 running 64-bit Linux).


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 16, 2011, 10:44:40 PM
That fixed it!

Just to clarify, is this preview a potential fix for the lockup issue I'm experiencing or is that part of number 2?



Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 16, 2011, 10:46:38 PM
Just to clarify, is this preview a potential fix for the lockup issue I'm experiencing or is that part of number 2?
This does not contain the optimization that I believe is responsible for the lockup issue. That optimization was not contained in 3diff or 4diff but was an additional optimization (introduced in the 'update' patch) specifically to reduce the 'hiccup' in 'getwork' responses when a new transaction hits the network.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: backburn on July 17, 2011, 06:01:08 AM
Patch at http://davids.webmaster.com/~davids/preview.diff compiles perfectly on the 0.3.24 release Ubuntu x64 10.04 + 10.10. Was working properly with load on test net and I have pushed to live with about 100 GHash. About an hour in, so far so good. Thank you so much for all your hard work Joel!

Code:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
19263 pool      20   0 4262m 139m 9396 S    2  2.5   2:24.12 bitcoind-24prev

Yummy!


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 17, 2011, 06:17:53 AM
Patch at http://davids.webmaster.com/~davids/preview.diff compiles perfectly on the 0.3.24 release Ubuntu x64 10.04 + 10.10. Was working properly with load on test net and I have pushed to live with about 100 GHash. About an hour in, so far so good. Thank you so much for all your hard work Joel!
Thanks for the success report. The very first preview had a serious bug that would make it crash immediately. The final release had, other than that fix, only cosmetic changes (like fixing extra spaces and such). So you can stick with the build you've got, since it obviously doesn't have the 'makes it crash immediately' bug.

Do you know offhand how that CPU usage compares with what you were seeing before?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: backburn on July 17, 2011, 07:18:12 AM
Do you know offhand how that CPU usage compares with what you were seeing before?

For a basis, this server is using an Intel Q8400. Looks to be around half the usage at "idle". During a LP; it hits about 40-50% usage. Bunches better than stock bitcoind, and about 10% less than the diff-4 v.95.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 17, 2011, 07:24:44 AM
Do you know offhand how that CPU usage compares with what you were seeing before?
For a basis, this server is using an Intel Q8400. Looks to be around half the usage at "idle". During a LP; it hits about 40-50% usage. Bunches better than stock bitcoind, and about 10% less than the diff-4 v.95.
Great to hear. Thanks.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: jollyjim on July 17, 2011, 07:46:34 AM
I've been testing and debugging the patches against the 0.3.23 version.  I think I've got most of the problems figured out but there is still a critical problem that's made me lose out on 3 blocks so far.

First, the fast getwork parsing doesn't work if the client doesn't send the id without quotes.  This will allow it to send the id with or without quotes:

ThreadRPCServer3 in rpc.cpp:
Code:
        if ((strRequest.find("\"getwork\"")!=std::string::npos) && strRequest.find("[]")!=std::string::npos)
            std::string id;
            size_t p = strRequest.find("\"id\":");
            if (p != std::string::npos)
            {
                size_t ep = strRequest.find(" ", p+5), e;
                if((e=strRequest.find(",", p+5))<ep) ep = e;
                if((e=strRequest.find("}", p+5))<ep) ep = e;
                id = strRequest.substr(p+5, ep-p-5);
            }

Second, the locking problems seen are mostly due to the mutex being non-recursive.  Changing mGetWork to recursive_mutex instead of mutex, along with the iterators for it, should fix the problem.

Third, the CheckWork done in getwork assigns a value that should be returned but to a variable in the wrong scope.  This has been addressed earlier.

Some of the locking seems to be off.  An example would be:

CommitTransaction in main.cpp:
Code:
        // Broadcast
        if (!wtxNew.AcceptToMemoryPool())
        {
            // This must not fail. The transaction has already been signed and recorded.
            printf("CommitTransaction() : Error: Transaction not valid");
            return false;
        }
        wtxNew.RelayWalletTransaction();
+        SyncGetWork(4);
    }
-    SyncGetWork(4);

The comment for SyncGetWork says cs_main must be held but before being called, but that's not the case with the patch before the above change.

Another one would be:

AddToBlockIndex in main.cpp:
Code:
    if (pindexNew == pindexBest)
    {
        // Notify UI to display prev block's coinbase if it was ours
        static uint256 hashPrevBestCoinBase;
-        CRITICAL_BLOCK(cs_mapWallet)
+        CRITICAL_BLOCK(cs_mapWallet) {
            vWalletUpdated.push_back(hashPrevBestCoinBase);
+            hashPrevBestCoinBase = vtx[0].GetHash();
+        }
-         hashPrevBestCoinBase = vtx[0].GetHash();
    }

This one may or may not matter depending on how many threads call it (I believe only one thread runs this function but I'm not 100% sure).  However, this related to crashes I've been seeing right after writing out the block and before sending out the proof of work.  The call to vtx[0].GetHash causes a crash in the Serialize function due to the assertion that nSize >= 0.  I believe it's due to vtx[0] being corrupted but the event is rare at such high difficulties.  I don't see the problem with the test network but it happens on the live one.

One thought I had is that the live one has more transactions and appending enough of them would trigger the crash.  If this is the case, one cause might be the changes JoelKatz has made that'd allow vtx to get updated elsewhere (from another client calling getwork?).  Another one may be vtx is a vector and whenever it resizes, the structures related to each object inside vtx isn't correctly copied over (I didn't see any copy constructors).  I was able to get a block through when I commented the above out so that might be a sign that I'm on the right track but I'd like to see it get a few more blocks before I can rule out that it wasn't just a random event (initially, crashes on the test network were random and I've yet to see any on it since the fixes mentioned earlier).

These high difficulty levels are making it really troublesome to debug locking/race conditions.  It'd be nice if one of the big pools try out various things to see if they also experience the crashes (and possibly end up not getting credit for the block) and try out various fixes to see if it fixes the problem.  I haven't seen anyone mention these crashes so for all I know, it's just some compiler/machine issue.  But then again, it doesn't seem like anyone's tried it with the fixes that doesn't lock up.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: backburn on July 17, 2011, 09:37:43 AM
We had both bitcoin-4diff.txt (.95)+ updates.txt +  0.3.23 and just 4diff (.95) + 0.3.23 in production for about a week @ 100+ GH/s each. Both produced blocks and we had no invalids. Neither crashed and ran un-interrupted under load for the week long test. On a side note, it seemed just using the 4diff and no updates we generated more blocks. But that really doesn't mean anything.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: backburn on July 17, 2011, 09:47:19 AM
Oh I forgot to add. We generated our first live block with the 4-diff preview code. So all seems to be working.

There are some awesome crash test pilots miners in our irc channel that are eager to test new features on our public test pool. So if there is anything we can try to break for you, please ask!


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Furyan on July 17, 2011, 02:19:07 PM
I've applied the patch to 0.3.24 and rebuilt, running now apparently without problems.

I'm doing a heavier load test today so I will let you know how that goes.

Can you explain the "hub" mode a little better?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 17, 2011, 02:26:11 PM
Can you explain the "hub" mode a little better?
The bitcoin code makes few outbound connections and makes them very slowly. It tried to make the connections to numerically diverse IP addresses. And it has a few design decisions that makes it waste CPU under certain circumstances involving having larger numbers of connections.

The hub modes are aimed at two things. First, for mining controllers, they allow the client to get a larger mix of connections and to reconnect itself to more of the network more rapidly on a restart. Second, for people who want to run bitcoind nodes to provide the network with a reduced diameter, they make it possible to more easily run nodes with many hundreds of connections.

It is still a work in progress. The ultimate goal is to improve network reliability and reduce network diameter and propagation delay by greater interconnectedness between well-connected nodes.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Matt Corallo on July 17, 2011, 02:30:35 PM
Can you explain the "hub" mode a little better?
Don't use it...period.  Hub mode creates a ton more outgoing connections in a much more aggressive manner than stock bitcoin.  This is very bad for the network as the number of nodes accepting incoming connections is relatively low and filling all of their slots that you can is really quite terrible.  That said, hub mode is an attempt at solving actual problems, namely the "islanding" of nodes.  This is largely caused by a large number of people on the network running old nodes, specifically those older than version 0.3.24 (including 0.3.23).  This causes nodes to not correctly forward recent blocks.  If you are running 0.3.23 or older (without having applied https://github.com/bitcoin/bitcoin/commit/497317453422611a077f7f195eb193d3bb597a9c) you are doing the network a great disservice and causing problems for others.
If you are running 0.3.24 and are, like some, seeing islanding and not being able to download the latest blocks, hub mode theoretically solves the issue in a very brute-force way, to the detriment of others.  The proper solution here is to -addnode the nodes of other large pools/miners/fallback nodes who are virtually guaranteed to have the latest blocks.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 17, 2011, 02:40:44 PM
Can you explain the "hub" mode a little better?
Don't use it...period.  Hub mode creates a ton more outgoing connections in a much more aggressive manner than stock bitcoin.  This is very bad for the network as the number of nodes accepting incoming connections is relatively low and filling all of their slots that you can is really quite terrible.
This was a valid criticism against the first hub mode patches. I was not aware of this issue and have changed the patches in such a way that I believe they help solve this problem rather than making it worse.

The newer versions decrease the number of outgoing connections made over the initial versions. In the largest hub mode now, the outgoing connections is capped at 96. In exchange, a much larger number of inbound connections (over 500 instead of 125) is accepted, so on balance the network gains in the number of connections it can accept.

I 100% agree that nodes that are behind NAT or cannot accept inbound connections should never ever run in hub mode. Otherwise, I don't think this particular criticism is valid anymore. In my opinion, people who run well-maintained, stable clients on the latest build in the largest hub mode with good network connectivity are doing the bitcoin network a favor.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Matt Corallo on July 17, 2011, 04:13:55 PM
The newer versions decrease the number of outgoing connections made over the initial versions. In the largest hub mode now, the outgoing connections is capped at 96. In exchange, a much larger number of inbound connections (over 500 instead of 125) is accepted, so on balance the network gains in the number of connections it can accept.
I wasnt aware you had dropped the outgoing cap, but 96 is still way, way, way too much.  If you want more outgoing connections, fine, but maybe make 10 or 20, not 96.  Accepting more incoming is great, and that should be encouraged. However, accepting more incoming than you make outgoing is not a solution, nor does it add more to the network than it takes away.  Keep in mind that nodes attempt to keep a diverse connection pool (something that could be implemented better, though) so accepting 1000 connections does not make up for making 100 connections because you are never going to get that many connections and because you are taking 100 slots on nodes in different /24s.  Again, let me just point out that though there is a valid argument for having something to help you get a better connection to more up-to-date nodes, hub mode is probably the worst way to fix the problem possible.  
As a side note, if you are going to encourage people to patch 0.3.23, please, please include https://github.com/bitcoin/bitcoin/commit/497317453422611a077f7f195eb193d3bb597a9c as without it, you really aren't helping the network at all. Sorry, for some reason I still thought this was based on .23, not sure why I thought that.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 17, 2011, 04:31:41 PM
I wasnt aware you had dropped the outgoing cap, but 96 is still way, way, way too much.  If you want more outgoing connections, fine, but maybe make 10 or 20, not 96.  Accepting more incoming is great, and that should be encouraged. However, accepting more incoming than you make outgoing is not a solution, nor does it add more to the network than it takes away.  Keep in mind that nodes attempt to keep a diverse connection pool (something that could be implemented better, though) so accepting 1000 connections does not make up for making 100 connections because you are never going to get that many connections and because you are taking 100 slots on nodes in different /24s.
I don't think this argument is valid. It seems valid if you think about one node, but think about 100 nodes on different /24s. Those 100 nodes running in hub mode are adding 500 plus connection slots each and in total they are doing so on 100 different /24s.

Of course any one node can't add IP diversity. But so long as each node adds more capacity than it takes, there should be a net gain in IP diversity.

It was based on 0.3.23 until just recently. I absolutely agree that running in hub mode without that patch is a problem.

On the next version of the hub mode patches, I will reduce the number of outgoing connections further. In hub mode 2, recommended for mining controllers, I will reduce the outbound connections from 48 to 32. I feel this is needed because otherwise there is too much of a delay on a restart before the node has an adequate view of the network. I will reduce hub mode 3 from 96 to 32 as well. Hub mode 3 is intended for stable network nodes, and there is no particular reason to need a quick network view.

 const unsigned HubModes[4][4]=
 { // outbound connections, total connections, IP mask, multithreaded connect
  {   8, 125, 0x0000ffff, 0 }, // Normal mode
- {  32, 200, 0x0000ffff, 0 }, // Small hub mode
- {  48, 384, 0x00ffffff, 1 }, // Medium hub mode
- {  96, 640, 0xffffffff, 1 }  // Large hub mode
+ {  16, 200, 0x0000ffff, 0 }, // Small hub mode
+ {  32, 384, 0x00ffffff, 1 }, // Medium hub mode
+ {  32, 640, 0x00ffffff, 1 }  // Large hub mode
 };

Update: This change is in the current bitcoin-4diff that is up on my web server.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Matt Corallo on July 17, 2011, 04:38:17 PM
On the next version of the hub mode patches, I will reduce the number of outgoing connections further. In hub mode 2, recommended for mining controllers, I will reduce the output connections from 48 to 32. I feel this is needed because otherwise there is too much of a delay on a restart before the node has an adequate view of the network. I will reduce hub mode 3 from 96 to 32 as well. Hub mode 3 is intended for stable network nodes, and there is no particular reason to need a quick network view.
32 seems like a reasonable number of outgoing connections.  Just keep in mind (or I guess others who haven't upgraded should keep in mind) that the reason it takes so long for nodes to get a good connection is not because of a poor network, it is because nodes have not upgraded to 0.3.24.  If you connect to a pre-.24 node and are more than a day or two out of date with the blockchain, you will be disconnected before you are able to get updated, which causes so many of the recent problems.  Seriously people, when it is said that you NEED TO UPGRADE TO 0.3.24, there is a reason behind it.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 17, 2011, 04:47:56 PM
Thanks for helping me make these patches better. If hope I didn't come across as hostile. I know we both want the same thing -- a network than can remain stable and reliable even with exponential growth.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Matt Corallo on July 17, 2011, 04:55:08 PM
If hope I didn't come across as hostile.
Oh absolutely not, it is going to solving a legitimate problem.  Hopefully I didn't come across as hostile...
All the work you are doing here is great, really hope we can get the asio/threaded RPC into mainline sometime soon (hopefully for a 4.1)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 17, 2011, 10:46:02 PM
Actually, now that I think about it, reducing the outbound connections hub make will weaken the network. My goal was to get a richly interconnected mesh of high-capacity hubs. And I guess I was thinking "we don't need to connect, they'll connect to us eventually". But this is not true -- if neither side is willing to make many outbound connections, then two hubs won't be likely to connect to each other. But if hubs make lots of outbound connections, then hubs will consume the inbound capacity of non-hubs, reducing the diversity of the network.

I've been thinking about a workaround. What I'm thinking is to have a NODE_HUB bit to indicate a large hub (with the same propagation/storage semantics as NODE_NETWORK). That way, hubs will know that nodes that set that bit are also capable of handling large numbers of incoming connections.

Hubs will first follow the normal mechanism for making outbound connections. But then they will preferentially make extra outbound connections strictly to other hubs to which they are not connected to improve network interconnection density and speed network propagation.

Once we attain a decent number hubs, the net effect will be that each hub will have some 32 outbound connections mostly to other hubs, some 32 inbound connections from other hubs, and be able to support about 512 incoming connections from non-hubs.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Matt Corallo on July 17, 2011, 11:14:56 PM
Yep, that is absolutely something that should be done, and its been tossed around in one form or another quite a few times (typically under the supernode/client or sub-node name).  You get a ton of supernodes which act like nodes do today, except that the client doesn't upgrade itself into a supernode unless it sees that it has good uptime, accept incoming connections, etc similar to the way gtk-gnutella does it.  At that point whether or not hubs need more outgoing connections can be decided.  I have a feeling there will be a small enough number of hubs/supernodes that any more than 8 would be needless, but that depends on any number of factors that can't really be decided now.
See https://forum.bitcoin.org/index.php?topic=12286.0 (https://forum.bitcoin.org/index.php?topic=12286.0) and http://forum.bitcoin.org/index.php?topic=7972.msg116265#msg116265 (http://forum.bitcoin.org/index.php?topic=7972.msg116265#msg116265)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 17, 2011, 11:51:52 PM
I see a lot of good ideas in there. But I think people are letting the perfect be the enemy of the good. Nobody will ever agree on the best possible solution and complex solutions have possible complex failure modes. I think a baby step in the right direction might be a good idea.

You might only need 8 hubs/supernodes to handle the network from a technical standpoint, but I think you want vastly more because someone who controls a significant fraction of the network could potentially grab all of at least one client's connections and keep that client in the dark. If we move to concentrate power in a small number of supernodes, we may create new attack possibilities.

There are many other steps we could take specifically to deal with the 'mushroom' attack. For example, well-known DNS names under separate administration could each publish a recent block hash as a TXT record. If you aren't seeing the blocks published by most of those well-known names, you're not caught up to the real network. But ideally, the network would just be dense and diverse enough that this would be impractical.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Matt Corallo on July 18, 2011, 02:20:54 AM
ideally, the network would just be dense and diverse enough that this would be impractical.
Yep, very much would be.  In fact it used to absolutely be, its only with the recent block download stuff, plus possibly the IRC split and other factors have made the network very unstable and poorly connected.  My hope is that if people would get off their ass and upgrade to 0.3.24, it would get much better, but as it stands we dont even have a mac build and people are being very, very slow to upgrade.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 18, 2011, 03:16:22 AM
previewdiff seems to be working fine for me now... just wanted to report back.

I'm a little confused, is the new 4diff update incorporating previewdiff now?  Should I be recompiling at this point?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: backburn on July 18, 2011, 07:27:35 AM
I had to revert back to the old 3.23 code today. The preview patch + 3.24 locked up the system, used up all available sockets after about 8-10hrs, twice. I had to kill -9 to get the process to end.  Possibly erroring when 2nd block is found.

Any changes v.97 3.24 that might address this?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Caesium on July 18, 2011, 09:46:29 AM
JoelKatz,

I appear to have some sort of memory/map leak with 0.3.24 + 0.96

bitcoind has grown to the following:
Code:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
24646 btc       20   0 9032m 246m 9136 S    2  6.2  13:02.59 bitcoind_0.3.24    

9032M of mapped memory. If I check /proc/24646/smaps, I can see just over  a thousand maps of 8192k but sadly they have no real information, here's a sample of one:

Code:
7f70c4ffb000-7f70c57fb000 rw-p 00000000 00:00 0 
Size:               8192 kB
Rss:                  16 kB
Pss:                  16 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        16 kB
Referenced:           16 kB
Anonymous:            16 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB

They appear to be increasing at approx one every few minutes.
Any idea where they might be coming from?
(edit: it's not leaking fds by the way, only 123 open descriptors and 108 of those are connections as reported by getinfo)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 18, 2011, 04:32:34 PM
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
26217 XXXX      20   0  9.8g 243m 7664 S    2  2.0  55:09.60 bitcoind

I am seeing similar to Caesium


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 18, 2011, 05:22:17 PM
Argh, that's not good. On the bright side, it shouldn't be too terribly hard to find and fix. I'll try to have a fix in the next few hours.

The new 4-diff is the same as the preview diff with a few small fixes and rebased against the latest release version. It's the same as the old 4-diff plus the update patches minus the 'getblock' "precompute skeleton block" patch which was causing problems.


Update: There is no change so simple I can't screw it up somehow!
See if this change fixes it:
--- new/util.h  2011-07-17 09:33:40.826055435 -0700
+++ util.h      2011-07-18 12:49:02.095159533 -0700
@@ -625,9 +625,10 @@ inline pthread_t CreateThread(void(*pfn)
         return (pthread_t)0;
     }
     if (!fWantHandle)
-        return (pthread_t)-1;
-    else
+    {
         pthread_detach(hthread);
+        return (pthread_t)-1;
+    }
     return hthread;
 }


The final CreateThread should look like this:

inline pthread_t CreateThread(void(*pfn)(void*), void* parg, bool fWantHandle=f
{
    pthread_t hthread = 0;
    int ret = pthread_create(&hthread, NULL, (void*(*)(void*))pfn, parg);
    if (ret != 0)
    {
        printf("Error: pthread_create() returned %d\n", ret);
        return (pthread_t)0;
    }
    if (!fWantHandle)
    {
        pthread_detach(hthread);
        return (pthread_t)-1;
    }
    return hthread;
}



Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: backburn on July 18, 2011, 10:39:55 PM
    if (!fWantHandle)
-        return (pthread_t)-1;
-    else
+    {
         pthread_detach(hthread);
+        return (pthread_t)-1;
+    }
     return hthread;
 }

UPDATE: Looks stable so far, running for 6 hrs.

Compiling and will test today, thank you bunches.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 18, 2011, 10:41:15 PM
Looks good so far, after running a couple hours.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Caesium on July 19, 2011, 09:25:10 AM
Yep can confirm fixed here too  :)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 19, 2011, 11:55:07 AM
3151 XXX      20   0  389m 167m 7556 S    8  1.4  20:04.13 bitcoind

That's about 16 hours of running, so looks good to me.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 19, 2011, 04:24:34 PM
Thanks guys.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: phorensic on July 19, 2011, 06:04:51 PM
Is everything stable, yet?  Time to freeze the code?  Release Candidate #1?  No new features planned?  I really want to run this code on my pool and it sounds like you have hit a milestone.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Matt Corallo on July 19, 2011, 06:11:48 PM
Is everything stable, yet?  Time to freeze the code?  Release Candidate #1?  No new features planned?  I really want to run this code on my pool and it sounds like you have hit a milestone.
And then pull request it so it can get a more thorough look-through and get merged upstream.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 19, 2011, 06:13:13 PM
I think there's still the getwork skeleton issue to work out, but other than that it's a big improvement.

I'm not sure if the changes to hub mode have made it into the diff yet, as well?

As far as a pull request, it's unlikely it will be mainstreamed due to some of the features/changes are not good for clients.  Though maybe I misunderstood that part?  I'd be happy to see it in the mainstream code.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Matt Corallo on July 19, 2011, 06:15:27 PM
As far as a pull request, it's unlikely it will be mainstreamed due to some of the features/changes are not good for clients.  Though maybe I misunderstood that part?  I'd be happy to see it in the mainstream code.
Well yes, hub mode probably won't get implemented as its really a hack to get it to work with the current network segmentation, but most, if not all, of the various code optimizations and RPC multithreading/asio/etc absolutely can and will eventually get merged upstream.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 19, 2011, 06:22:57 PM
The less ugly optimizations and straight bug fixes should certainly be merge candidates. The hub mode patches and uglier optimizations are not sensible merge candidates.

Obvious merge candidates would be:

1) strRPCUser/strRPCPass variables to avoid extracting them on each RPC call.

2) Native long polling support.

3) ToHex replacing HexStr.

4) Faster Base64 decoder.

5) Leak fix in pthreads version of CreateThread.

Also, some version of the RPC/keepalive fix needs to be merged but it doesn't have to be my version.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: error on July 19, 2011, 10:25:37 PM
Please send pull requests for these!


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: phorensic on July 20, 2011, 12:50:53 AM
As far as a pull request, it's unlikely it will be mainstreamed due to some of the features/changes are not good for clients.  Though maybe I misunderstood that part?  I'd be happy to see it in the mainstream code.
I agree with you.  This is a pool oriented fork and Jay Garzik will probably take a couple seconds to look at the code and then close the pull request.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 20, 2011, 02:22:48 AM
I agree with you.  This is a pool oriented fork and Jay Garzik will probably take a couple seconds to look at the code and then close the pull request.
I think the five things I specifically mentioned would be accepted. The long polling support is really the only maybe.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: simplecoin on July 20, 2011, 02:13:53 PM
Is there a unified patch ready that is stable, or better yet working repo? I've been running the last stable 3diff against .23 I saw weeks ago.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Furyan on July 20, 2011, 05:16:08 PM
Is there a unified patch ready that is stable, or better yet working repo? I've been running the last stable 3diff against .23 I saw weeks ago.

I have 4diff running stably against 0.3.24.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 20, 2011, 05:43:00 PM
As do I


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: backburn on July 20, 2011, 08:29:45 PM
3.24 4-diff is running stable for over 24 hrs. Solved several blocks @ 175 GHash

Code:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                    
22895 tacos    20   0  466m 260m 6800 S    4  3.3  52:03.41 bitcoind-24prev



Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: TeraPool on July 20, 2011, 09:39:22 PM
Can somebody please spoon feed me the process as to how to patch the source code with JoelKatz' diff .txt file?

I know how to compile bitcoind, but am a noob when it comes to patching things :S


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 20, 2011, 10:01:00 PM
Can somebody please spoon feed me the process as to how to patch the source code with JoelKatz' diff .txt file?

I know how to compile bitcoind, but am a noob when it comes to patching things :S
Go to the 'src' directory, and then:

1) If you don't have the diff already, use this command:
wget http://davids.webmaster.com/~davids/bitcoin-4diff.txt

2) To patch it:
patch -p1 < bitcoin-4diff.txt

If you did it correctly, the output should be this:
patching file init.cpp
patching file main.cpp
patching file net.cpp
patching file net.h
patching file rpc.cpp
patching file util.cpp
patching file util.h


Update: Sorry about the typo in the URL.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: phorensic on July 21, 2011, 06:00:49 AM
Testing bitcoind v0.3.24 + Joel Katz v0.98 4diff on our pool.  Going very well so far.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: TeraPool on July 21, 2011, 11:02:25 PM
Can somebody please spoon feed me the process as to how to patch the source code with JoelKatz' diff .txt file?

I know how to compile bitcoind, but am a noob when it comes to patching things :S
Go to the 'src' directory, and then:

1) If you don't have the diff already, use this command:
wget http://davids.webmaster.com/~davids/bitcoin-4diff.txt

2) To patch it:
patch -p1 < bitcoin-4diff.txt

If you did it correctly, the output should be this:
patching file init.cpp
patching file main.cpp
patching file net.cpp
patching file net.h
patching file rpc.cpp
patching file util.cpp
patching file util.h


Update: Sorry about the typo in the URL.


THANK YOU!

P.S. My hand has been forced to make a donation!

Just sent 1 to your payb.tc address. Thanks again.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: AnnihilaT on July 27, 2011, 02:10:36 PM
Testing bitcoind v0.3.24 + Joel Katz v0.99 4diff on our pool now.... so far so good in the last 15 mins...

Does anyone know the difd from 0.98 and 0.99 ?  couldnt find a copy of the older version anywhere.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 27, 2011, 02:24:57 PM
Why would you want the older version?

Anyway ... I think there might be a small memory leak somewhere still.  My bitcoind is up to almost 700m at this point and I'm getting some connection errors occasionally, though that may be due to some other issues not related to bitcoind.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: AnnihilaT on July 27, 2011, 03:25:41 PM
Why would you want the older version?

Anyway ... I think there might be a small memory leak somewhere still.  My bitcoind is up to almost 700m at this point and I'm getting some connection errors occasionally, though that may be due to some other issues not related to bitcoind.


I wanted it only to diff against 0.99 since you guys said you were running that version without problems so i was curious to see what changes had been made in 0.99. 

Logical, no?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: AnnihilaT on July 28, 2011, 01:37:27 PM
Im seeing the same thing regarding a possible memory leak.  Continual growth of used memory over the last 24 hours or so... up to 453368kB now and growing...


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: phorensic on July 28, 2011, 03:54:54 PM
Been running it for a week strong now.  I don't know if I would consider it a memory leak, but ours is at just under 700m used.  Running in -hub level 1 with built in LP.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: AnnihilaT on July 28, 2011, 04:25:55 PM
Im seeing the same thing regarding a possible memory leak.  Continual growth of used memory over the last 24 hours or so... up to 453368kB now and growing...

spoke too soon.... it just went down to just under 350m :)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Inaba on July 28, 2011, 04:30:04 PM
Mine is right around 700m after a week or so, same as Phorensic.  We'll see how to goes as time goes on.  Seems a little excessive though.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 28, 2011, 06:35:08 PM
Testing bitcoind v0.3.24 + Joel Katz v0.99 4diff on our pool now.... so far so good in the last 15 mins...

Does anyone know the difd from 0.98 and 0.99 ?  couldnt find a copy of the older version anywhere.
There was a change from Giel de Nijs for an issue that maybe could cause stale shares. Neither of us is sure the issue can actually happen, but the patch is obviously safe and might fix a real problem, so I merged it. The change moves the code to send a SIGUSR1 for native long-polling to later in the code path.

Sorry, I meant to put a changelog at the top of the patch and it just didn't happen.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: AnnihilaT on July 28, 2011, 07:50:00 PM
Testing bitcoind v0.3.24 + Joel Katz v0.99 4diff on our pool now.... so far so good in the last 15 mins...

Does anyone know the difd from 0.98 and 0.99 ?  couldnt find a copy of the older version anywhere.
There was a change from Giel de Nijs for an issue that maybe could cause stale shares. Neither of us is sure the issue can actually happen, but the patch is obviously safe and might fix a real problem, so I merged it. The change moves the code to send a SIGUSR1 for native long-polling to later in the code path.

Sorry, I meant to put a changelog at the top of the patch and it just didn't happen.


Ok clear!  THanks for the reply :)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: denis2342 on July 29, 2011, 10:30:11 AM
Hi

I tested the native long polling patch and noticed that it fired the signal so early that pushpoold got the old block and not the new one. So it seems you have it fixed now.

I also moved the patch to the end of the function and I never saw the problem again.

Denis

PS: I couldn't write here because my account was not activated until now


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on July 29, 2011, 10:33:33 AM
I remember specifically thinking, when I placed the code, that the code held a lock that wouldn't let the 'getwork' code run until it was safe anyway. But then I looked back after I got the patch from Giel, and I couldn't figure out what I saw that justified that belief.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Balthazar on October 17, 2011, 01:09:21 PM
...
Hi. Do you have plans for bitcoin v0.4? :) Current patches are incompatible with it because of some changes in RPC. :(


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on October 17, 2011, 06:50:49 PM
...
Hi. Do you have plans for bitcoin v0.4? :) Current patches are incompatible with it because of some changes in RPC. :(
http://davids.webmaster.com/~davids/bitcoin-4diff-beta.txt


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Balthazar on October 18, 2011, 09:18:55 AM
...
Hi. Do you have plans for bitcoin v0.4? :) Current patches are incompatible with it because of some changes in RPC. :(
http://davids.webmaster.com/~davids/bitcoin-4diff-beta.txt
Thanks. http://itzod.ru/images/smilies/drink2_mini.gif I'll try test this today and write the report here.

// /me tried to implement support for 0.4 yesterday, but faced with stability problems.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Balthazar on October 18, 2011, 12:49:15 PM
It works great. 80% CPU usage & disconnects ==> 10-15% & rock stable.  :)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Jine on October 23, 2011, 07:03:01 AM
...
Hi. Do you have plans for bitcoin v0.4? :) Current patches are incompatible with it because of some changes in RPC. :(
http://davids.webmaster.com/~davids/bitcoin-4diff-beta.txt

Hi!

Have anyone tested out the 0.4 beta patch successfully?
Any comparison with the 0.32 patch?

I'm thinking of setting up a new environment with poolserverj + 0.4.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on October 23, 2011, 11:41:12 PM
Have anyone tested out the 0.4 beta patch successfully?
Any comparison with the 0.32 patch?

I'm thinking of setting up a new environment with poolserverj + 0.4.
Several people have been using it for two weeks or more, and I haven't received any issue reports yet.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Justin Bieber on October 25, 2011, 03:47:22 PM
Quote from: JoelKatz
awesomeness
Is there one in the pipeline for 0.5? The RPC changes are a little too much for me to handle.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on October 25, 2011, 08:34:47 PM
Quote from: JoelKatz
awesomeness
Is there one in the pipeline for 0.5? The RPC changes are a little too much for me to handle.
I'll try to find time to work on it in the next few days. It's hard to refuse a request from someone with perfect hair.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: Justin Bieber on October 27, 2011, 02:32:19 PM
I'll try to find time to work on it in the next few days. It's hard to refuse a request from someone with perfect hair.
I'm flattered!

Thanks.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: terrytibbs on October 27, 2011, 07:05:47 PM
watching this


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on October 28, 2011, 02:56:52 AM
http://davids.webmaster.com/~davids/bitcoin-4diff-5beta.txt

This is a "first working version". It patches cleanly, compiles, and passes my basic functional tests.

Quite a few of my optimizations were already made. Yay!

I guess I'll have to start working on some new ones.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: c_k on November 25, 2011, 10:36:48 AM
http://davids.webmaster.com/~davids/bitcoin-4diff-5beta.txt

This is a "first working version". It patches cleanly, compiles, and passes my basic functional tests.

Quite a few of my optimizations were already made. Yay!

I guess I'll have to start working on some new ones.


Hi David,

What are your thoughts on the final 0.5.0 release?


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on November 25, 2011, 09:17:33 PM
Hi David,

What are your thoughts on the final 0.5.0 release?
Unfortunately, I've been completely swamped with work lately, and I haven't had a chance to look at it yet.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: c_k on January 04, 2012, 11:56:59 PM
Results when attempting to apply bitcoin-4diff-5beta.txt to bitcoin v0.5.1:

Quote
patching file orig/bitcoinrpc.cpp
Hunk #3 succeeded at 52 (offset 2 lines).
Hunk #4 succeeded at 1610 (offset -4 lines).
Hunk #5 succeeded at 1640 (offset -4 lines).
Hunk #6 succeeded at 1660 (offset -4 lines).
Hunk #7 succeeded at 1974 (offset -4 lines).
Hunk #8 succeeded at 2003 (offset -4 lines).
Hunk #9 succeeded at 2012 (offset -4 lines).
Hunk #10 succeeded at 2026 (offset -4 lines).
Hunk #11 succeeded at 2064 (offset -4 lines).
Hunk #12 succeeded at 2080 (offset -4 lines).
Hunk #13 FAILED at 2109.
Hunk #14 succeeded at 2142 (offset -9 lines).
Hunk #15 succeeded at 2210 (offset -9 lines).
Hunk #16 succeeded at 2310 (offset -6 lines).
Hunk #17 succeeded at 2375 (offset -6 lines).
Hunk #18 succeeded at 2421 (offset -6 lines).
Hunk #19 succeeded at 2430 (offset -6 lines).
Hunk #20 succeeded at 2481 (offset -6 lines).
Hunk #21 succeeded at 2514 (offset -6 lines).
Hunk #22 succeeded at 2542 (offset -6 lines).
1 out of 22 hunks FAILED -- saving rejects to file orig/bitcoinrpc.cpp.rej
patching file orig/init.cpp
Hunk #1 FAILED at 189.
Hunk #2 succeeded at 222 (offset 11 lines).
Hunk #3 succeeded at 258 (offset 11 lines).
1 out of 3 hunks FAILED -- saving rejects to file orig/init.cpp.rej
patching file orig/main.cpp
Hunk #1 succeeded at 1149 (offset 20 lines).
Hunk #2 succeeded at 1192 (offset 20 lines).
Hunk #3 succeeded at 1206 (offset 20 lines).
patching file orig/net.cpp
Hunk #2 succeeded at 82 (offset 1 line).
Hunk #3 succeeded at 99 (offset 1 line).
Hunk #4 succeeded at 977 (offset 1 line).
Hunk #5 succeeded at 1007 (offset 1 line).
Hunk #6 succeeded at 1046 (offset 1 line).
Hunk #7 succeeded at 1108 (offset 1 line).
Hunk #8 succeeded at 1146 (offset 1 line).
Hunk #9 succeeded at 1415 (offset 42 lines).
Hunk #10 succeeded at 1449 (offset 42 lines).
Hunk #11 succeeded at 1515 (offset 42 lines).
Hunk #12 succeeded at 1522 (offset 42 lines).
Hunk #13 succeeded at 1554 (offset 42 lines).
Hunk #14 succeeded at 1572 (offset 42 lines).
Hunk #15 succeeded at 1630 (offset 42 lines).
Hunk #6 succeeded at 1046 (offset 1 line).
Hunk #7 succeeded at 1108 (offset 1 line).
Hunk #8 succeeded at 1146 (offset 1 line).
Hunk #9 succeeded at 1415 (offset 42 lines).
Hunk #10 succeeded at 1449 (offset 42 lines).
Hunk #11 succeeded at 1515 (offset 42 lines).
Hunk #12 succeeded at 1522 (offset 42 lines).
Hunk #13 succeeded at 1554 (offset 42 lines).
Hunk #14 succeeded at 1572 (offset 42 lines).
Hunk #15 succeeded at 1630 (offset 42 lines).
Hunk #16 succeeded at 1654 (offset 42 lines).
Hunk #17 succeeded at 1706 (offset 42 lines).
Hunk #18 succeeded at 1718 (offset 42 lines).
Hunk #19 succeeded at 1746 (offset 42 lines).
Hunk #20 succeeded at 1953 (offset 48 lines).
Hunk #21 succeeded at 1968 with fuzz 2 (offset 48 lines).
patching file orig/net.h
Hunk #2 succeeded at 79 (offset -1 lines).
Hunk #3 succeeded at 123 (offset -1 lines).
Hunk #4 succeeded at 185 (offset -1 lines).
patching file orig/util.cpp
patching file orig/util.h
Hunk #4 succeeded at 710 (offset 10 lines).


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: backburn on January 12, 2012, 06:35:28 AM
Results when attempting to apply bitcoin-4diff-5beta.txt to bitcoin v0.5.1:

Quote
patching file orig/bitcoinrpc.cpp
Hunk #3 succeeded at 52 (offset 2 lines).
Hunk #4 succeeded at 1610 (offset -4 lines).
Hunk #5 succeeded at 1640 (offset -4 lines).
Hunk #6 succeeded at 1660 (offset -4 lines).
Hunk #7 succeeded at 1974 (offset -4 lines).
Hunk #8 succeeded at 2003 (offset -4 lines).
Hunk #9 succeeded at 2012 (offset -4 lines).
Hunk #10 succeeded at 2026 (offset -4 lines).
Hunk #11 succeeded at 2064 (offset -4 lines).
Hunk #12 succeeded at 2080 (offset -4 lines).
Hunk #13 FAILED at 2109.
Hunk #14 succeeded at 2142 (offset -9 lines).
Hunk #15 succeeded at 2210 (offset -9 lines).
Hunk #16 succeeded at 2310 (offset -6 lines).
Hunk #17 succeeded at 2375 (offset -6 lines).
Hunk #18 succeeded at 2421 (offset -6 lines).
Hunk #19 succeeded at 2430 (offset -6 lines).
Hunk #20 succeeded at 2481 (offset -6 lines).
Hunk #21 succeeded at 2514 (offset -6 lines).
Hunk #22 succeeded at 2542 (offset -6 lines).
1 out of 22 hunks FAILED -- saving rejects to file orig/bitcoinrpc.cpp.rej
patching file orig/init.cpp
Hunk #1 FAILED at 189.
Hunk #2 succeeded at 222 (offset 11 lines).
Hunk #3 succeeded at 258 (offset 11 lines).
1 out of 3 hunks FAILED -- saving rejects to file orig/init.cpp.rej
patching file orig/main.cpp
Hunk #1 succeeded at 1149 (offset 20 lines).
Hunk #2 succeeded at 1192 (offset 20 lines).
Hunk #3 succeeded at 1206 (offset 20 lines).
patching file orig/net.cpp
Hunk #2 succeeded at 82 (offset 1 line).
Hunk #3 succeeded at 99 (offset 1 line).
Hunk #4 succeeded at 977 (offset 1 line).
Hunk #5 succeeded at 1007 (offset 1 line).
Hunk #6 succeeded at 1046 (offset 1 line).
Hunk #7 succeeded at 1108 (offset 1 line).
Hunk #8 succeeded at 1146 (offset 1 line).
Hunk #9 succeeded at 1415 (offset 42 lines).
Hunk #10 succeeded at 1449 (offset 42 lines).
Hunk #11 succeeded at 1515 (offset 42 lines).
Hunk #12 succeeded at 1522 (offset 42 lines).
Hunk #13 succeeded at 1554 (offset 42 lines).
Hunk #14 succeeded at 1572 (offset 42 lines).
Hunk #15 succeeded at 1630 (offset 42 lines).
Hunk #6 succeeded at 1046 (offset 1 line).
Hunk #7 succeeded at 1108 (offset 1 line).
Hunk #8 succeeded at 1146 (offset 1 line).
Hunk #9 succeeded at 1415 (offset 42 lines).
Hunk #10 succeeded at 1449 (offset 42 lines).
Hunk #11 succeeded at 1515 (offset 42 lines).
Hunk #12 succeeded at 1522 (offset 42 lines).
Hunk #13 succeeded at 1554 (offset 42 lines).
Hunk #14 succeeded at 1572 (offset 42 lines).
Hunk #15 succeeded at 1630 (offset 42 lines).
Hunk #16 succeeded at 1654 (offset 42 lines).
Hunk #17 succeeded at 1706 (offset 42 lines).
Hunk #18 succeeded at 1718 (offset 42 lines).
Hunk #19 succeeded at 1746 (offset 42 lines).
Hunk #20 succeeded at 1953 (offset 48 lines).
Hunk #21 succeeded at 1968 with fuzz 2 (offset 48 lines).
patching file orig/net.h
Hunk #2 succeeded at 79 (offset -1 lines).
Hunk #3 succeeded at 123 (offset -1 lines).
Hunk #4 succeeded at 185 (offset -1 lines).
patching file orig/util.cpp
patching file orig/util.h
Hunk #4 succeeded at 710 (offset 10 lines).

You'll need to do the diffs in bitcoinrpc.cpp.rej and init.cpp.rej by hand.
- means remove this line.
+ means add this line.
no sign is a reference point ( leave line alone)

have fun :)


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: backburn on January 13, 2012, 01:51:45 AM
Some observations on the http://davids.webmaster.com/~davids/bitcoin-4diff-beta.txt patch.

bitcoind v0.4.0 + 4diff-beta seems to use about 5% more CPU and 3% less memory when compared to bitcoind v0.3.24 + 4diff.

Other than that, v0.4.0 has been stable for almost a week in a live environment. ( And a week before that on testnet.) We have some private 0.3.24 based pools that have been taking a stable 50+ Ghash for 5 months without a restart. I assume that will be the case with v0.4.0 :D

We though about messing with the .5 branch + Joel's patch. However, with the rapid release schedule of late and massive changes coming with the 0.6 release of bitcoin. We decided to hold off until .6 is stable before upgrading again.


Joel, thanks so much for all of your hard work. I'm glad to see many of your changes have been mainlined :D


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: JoelKatz on January 13, 2012, 02:59:02 AM
Joel, thanks so much for all of your hard work. I'm glad to see many of your changes have been mainlined :D
Thanks. I've been completely swamped with Real Work (TM) lately or I would have merged the remaining changes into the 0.5 branch.


Title: Re: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind
Post by: btcx on April 27, 2012, 06:36:37 AM
For anyone that still wants to use this for the 0.6.0 release, I've made some patches with slight modifications to David's code.  It's split up into two parts.  The first part should still be relevant for pool operators (keep-alive connections, hub mode, USR1 signals on block changes).  The second part is mostly obsolete now with merged mining and better pooling software (faster getwork).

You can find the patches at http://pastebin.com/viJV1NLi (http://pastebin.com/viJV1NLi) and http://pastebin.com/UXep0r3c (http://pastebin.com/UXep0r3c)

Some of the updates from 0.4 to 0.6 are non-trivial so this should help.  This particular patch hasn't been tested much but a different one for 0.6.0 rc4 has been running without any problems.

Edit: Oops, the patches were for the current head instead of 0.6.0.
Edit2: Updated the above links for the right patches.

For future reference, these may give you a head start on patching against 0.6.1: http://pastebin.com/XSBqRAaP (http://pastebin.com/XSBqRAaP) and http://pastebin.com/5R9N29aA (http://pastebin.com/5R9N29aA).