[20 BTC] Multithreaded Keep-alive Implementation in Bitcoind

[JoelKatz]

It's a resource leak, and it's not in my changes! It's in the 'CreateThread' function. I guess I use that more heavily than anything else, so I exposed the resource leak. Here's the fix:

--- orig/util.h 2011-06-28 08:28:03.070006598 -0700
+++ new/util.h  2011-06-28 19:51:51.186449938 -0700
@@ -624,7 +624,10 @@ inline pthread_t CreateThread(void(*pfn)
         return (pthread_t)0;
     }
     if (!fWantHandle)
+    {
+        pthread_detach(hthread);
         return (pthread_t)-1;
+    }
     return hthread;
 }

Sorry about the rocky road. This one was just bad luck though. This change should be merged with the official client -- it's an obviously-correct, obviously-safe change.

[1714043646]

1714043646

[1714043646]

1714043646

[Jine]

Compiled it and pushed it live now.
No segfault or errors in the beginning anyway

We'll see in a few hours. I'll post here if anything happens.

[Jine]

Forgot to mention, thanks a lot for your hard work. 
We'll remember this in the future, that you can be sure of.

[JoelKatz]

I have a new version out at:
http://davids.webmaster.com/~davids/bitcoin-3diff.txt

It has the turbo RPC changes, the new hub mode to reduce stale/lost blocks, and native long polling support. Please read the documentation at the top of the file.

CAUTION: This code was just finished and has not been well-tested. Test results are appreciated. Bug reports and success reports are equally welcome.

Update: Version 0.4 is now up. It fixes a bug that could cause bitcoind to hang when directed to shutdown.

[simplecoin]

I have a new version out at:
http://davids.webmaster.com/~davids/bitcoin-3diff.txt

It has the turbo RPC changes, the new hub mode to reduce stale/lost blocks, and native long polling support. Please read the documentation at the top of the file.

CAUTION: This code was just finished and has not been well-tested. Test results are appreciated. Bug reports and success reports are equally welcome.

Update: Version 0.4 is now up. It fixes a bug that could cause bitcoind to hang when directed to shutdown.

Currently testing v0.5 on live server.... (a whopping 3 users since relaunch). Will let you know how it turns out.

So far so good. LP notifications seem faster than a certain unnamed big pool Also it seems to favor my pool during new getwork causing my pool to get a higher share of hashrates than said big pool when using hashkill.

[eleuthria]

800 GH/sec and counting on the BTC Guild stress test.
UPDATE:  900 and counting.

BTC Guild, where test runs are done on production servers!

[eleuthria]

1 TH/sec is where idles started coming back for my DE server, which is an i7-920 system.  I believe the new bottleneck is the getwork processing.

[JoelKatz]

Thanks for the testing and the report.

I believe your diagnosis is probably right. I've been trying to figure out why that latency is so high. It turns out that when processing a significant change, like when a new block has been found, the code has to do a whole bunch of stuff. Coincidentally, this is exactly the same time every client needs new work. I believe the mechanism is contention on the cs_main mutex, which 'getwork' needs in this case because it calls CreateNewBlock. I'll do some head scratching.

One way to fix this might be to pre-compute the information 'getwork' needs so we don't have to wait for a 'getwork' request to do it. That would allow 'getwork' to run without touching the cs_main mutex. I'll need to instrument the code to confirm that's the issue before putting the effort into a change that might fix something that's not actually a problem.

[Jine]

We're running the latest 0.5 patch since last night - and it seems to work just fine.
Although - a couple of users have complained about higher stale-count.

Is that something you are aware of that could be a sideeffect of the modifications?

LP seems to work just fine tho.

[JoelKatz]

We're running the latest 0.5 patch since last night - and it seems to work just fine.
Although - a couple of users have complained about higher stale-count.

Is that something you are aware of that could be a sideeffect of the modifications?

LP seems to work just fine tho.

I'm not really quite sure. Two questions:

1) Are you using any of the hub modes? If so, which level?

2) Are you using my long polling or are you using blkmond?

[Jine]

Hi!

This should answer both of those:

Code:

BITCOIND_OPTS="-conf=/etc/bitcoin.conf -pid=/var/run/bitcoind.pid -hub=2 -pollpidfile=/var/run/pushpoold.pid -daemon"

In other words: hub level 2, and yes - built in LP.

[JoelKatz]

Awesome. I'm glad all of my code bits are getting "real use". Let me know if people are still seeing a higher percentage of stale blocks. It's possible the daemon just needed some time to 'settle in' as it makes connections and so on. My next set of changes may fix this anyway, it's designed specifically to reduce latency in 'getwork', particularly around reorganizations and transaction list changes.

Make sure your CPU has the juice to run at "-hub=2". Note that with a multi-core CPU, things can be deceptive. Bitcoind isn't very well multi-threaded -- if one core is maxed out, there's a good chance bitcoind is CPU limited. My next changes after the changes I'm working on now are expected to be better multi-threading of the code that handles I/O.

You did remember not to run blkmond, right? It will conflict with native long polling.

[Jine]

Hi!

We're running the "master node" upon a 2*Intel Xeon E5504 (8 cores in total);
http://data.fuskbugg.se/skalman02/4e0deb49dc5b9_tits.png

So far it looks good for me. And no, no blkmond.
But i think we're still experiencing a bit of lag / stales when LP is pushed, due to huge amounts of connections I'd think.

[phorensic]

Great progress!  One question, are we tracking this fork on github?  I didn't see the patches on the network graph there.

[JoelKatz]

Great progress!  One question, are we tracking this fork on github?  I didn't see the patches on the network graph there.

No, I'm not doing that ... yet!

[MrSam]

hmm, where can i get v0.5 ?

I tried 0.4 yesterday but wont work, bitcoind crashes after pushpool sends many
'HTTP request failed: The requested URL returned error: 500' and opened like 20k sockets in seconds

[JoelKatz]

hmm, where can i get v0.5 ?

I tried 0.4 yesterday but wont work, bitcoind crashes after pushpool sends many
'HTTP request failed: The requested URL returned error: 500' and opened like 20k sockets in seconds

http://davids.webmaster.com/~davids/bitcoin-4diff.txt

I'm starting to suspect the newer issues are coming from pushpool, but I'm not 100% sure yet. I'm not going to have much time today, but I hope to investigate in more detail tomorrow.

[MrSam]

.

[MrSam]

Joel,

I wanted to thank you ! The patch works really well now ..

The only issue that remains is that the USR1 signal has a bit of a delay.

pushpool debug:
[2011-07-05 23:36:1.710434] USR1 signal received, flushing LP waiters
if i check blockexplorer, that block was found at 2011-07-05 23:35:31)

[2011-07-05 23:33:36.427621] USR1 signal received, flushing LP waiters
real time 2011-07-05 23:33:24

[2011-07-05 23:30:36.337827] USR1 signal received, flushing LP waiters
real time 2011-07-05 23:30:20

When i test this on my mining rig at home, the rejected blocks are just those few seconds between the found block and the sig.

[JoelKatz]

Thanks for that report. I'll check the code and see if I can get the SIGUSR1 to be sent faster. I think that will solve half the problem though. While pushpool will issue fewer blocks that will turn into stale shares, bitcoind will still be processing the new block when all the new 'getwork' requests start piling in, and you'll get miner idles instead.

I have a fix for this that has other benefits as well, but the first time I tried to implement it, it was a disaster. I think I know the right way to do it now, having run out of wrong ways to do it already. What I'd like to do is pre-compute the skeleton 'getwork' block when a new block or new transaction comes in, rather than waiting to hijack the next 'getwork' request to do it. But to make this work, the locking has to be exactly right. The funny thing is, in my day job ten years ago, this was exactly the type optimization I specialized in. And now I can't seem to quite get it right.