Bitcoin Forum
December 09, 2016, 01:47:34 PM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: « 1 2 3 4 [5] 6 7 8 9 10 11 12 13 »  All
  Print  
Author Topic: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind  (Read 29394 times)
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
June 29, 2011, 02:52:23 AM
 #81

It's a resource leak, and it's not in my changes! It's in the 'CreateThread' function. I guess I use that more heavily than anything else, so I exposed the resource leak. Here's the fix:

--- orig/util.h 2011-06-28 08:28:03.070006598 -0700
+++ new/util.h  2011-06-28 19:51:51.186449938 -0700
@@ -624,7 +624,10 @@ inline pthread_t CreateThread(void(*pfn)
         return (pthread_t)0;
     }
     if (!fWantHandle)
+    {
+        pthread_detach(hthread);
         return (pthread_t)-1;
+    }
     return hthread;
 }

Sorry about the rocky road. This one was just bad luck though. This change should be merged with the official client -- it's an obviously-correct, obviously-safe change.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481291254
Hero Member
*
Offline Offline

Posts: 1481291254

View Profile Personal Message (Offline)

Ignore
1481291254
Reply with quote  #2

1481291254
Report to moderator
1481291254
Hero Member
*
Offline Offline

Posts: 1481291254

View Profile Personal Message (Offline)

Ignore
1481291254
Reply with quote  #2

1481291254
Report to moderator
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
June 29, 2011, 03:10:22 AM
 #82

Compiled it and pushed it live now.
No segfault or errors in the beginning anyway Wink

We'll see in a few hours. I'll post here if anything happens.

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
June 29, 2011, 03:30:13 AM
 #83

Forgot to mention, thanks a lot for your hard work.  Shocked
We'll remember this in the future, that you can be sure of.


Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
June 29, 2011, 08:04:16 AM
 #84

I have a new version out at:
http://davids.webmaster.com/~davids/bitcoin-3diff.txt

It has the turbo RPC changes, the new hub mode to reduce stale/lost blocks, and native long polling support. Please read the documentation at the top of the file.

CAUTION: This code was just finished and has not been well-tested. Test results are appreciated. Bug reports and success reports are equally welcome.

Update: Version 0.4 is now up. It fixes a bug that could cause bitcoind to hang when directed to shutdown.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
simplecoin
Sr. Member
****
Offline Offline

Activity: 406



View Profile WWW
June 29, 2011, 06:10:46 PM
 #85

I have a new version out at:
http://davids.webmaster.com/~davids/bitcoin-3diff.txt

It has the turbo RPC changes, the new hub mode to reduce stale/lost blocks, and native long polling support. Please read the documentation at the top of the file.

CAUTION: This code was just finished and has not been well-tested. Test results are appreciated. Bug reports and success reports are equally welcome.

Update: Version 0.4 is now up. It fixes a bug that could cause bitcoind to hang when directed to shutdown.

Currently testing v0.5 on live server.... (a whopping 3 users since relaunch). Will let you know how it turns out.

So far so good. LP notifications seem faster than a certain unnamed big pool Wink Also it seems to favor my pool during new getwork causing my pool to get a higher share of hashrates than said big pool when using hashkill.

Donations: 1VjGJHPtLodwCFBDWsHJMdEhqRcRKdBQk
eleuthria
Legendary
*
Offline Offline

Activity: 1750


BTC Guild Owner


View Profile WWW
June 29, 2011, 06:22:20 PM
 #86

800 GH/sec and counting on the BTC Guild stress test.
UPDATE:  900 and counting.

BTC Guild, where test runs are done on production servers!

R.I.P. BTC Guild, 2011 - 2015.
BTC Guild Forum Thread
eleuthria
Legendary
*
Offline Offline

Activity: 1750


BTC Guild Owner


View Profile WWW
June 29, 2011, 06:45:48 PM
 #87

1 TH/sec is where idles started coming back for my DE server, which is an i7-920 system.  I believe the new bottleneck is the getwork processing.

R.I.P. BTC Guild, 2011 - 2015.
BTC Guild Forum Thread
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
June 29, 2011, 06:52:49 PM
 #88

Thanks for the testing and the report.

I believe your diagnosis is probably right. I've been trying to figure out why that latency is so high. It turns out that when processing a significant change, like when a new block has been found, the code has to do a whole bunch of stuff. Coincidentally, this is exactly the same time every client needs new work. I believe the mechanism is contention on the cs_main mutex, which 'getwork' needs in this case because it calls CreateNewBlock. I'll do some head scratching.

One way to fix this might be to pre-compute the information 'getwork' needs so we don't have to wait for a 'getwork' request to do it. That would allow 'getwork' to run without touching the cs_main mutex. I'll need to instrument the code to confirm that's the issue before putting the effort into a change that might fix something that's not actually a problem.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
July 01, 2011, 04:41:04 AM
 #89

We're running the latest 0.5 patch since last night - and it seems to work just fine.
Although - a couple of users have complained about higher stale-count.

Is that something you are aware of that could be a sideeffect of the modifications?

LP seems to work just fine tho.

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
July 01, 2011, 05:00:31 AM
 #90

We're running the latest 0.5 patch since last night - and it seems to work just fine.
Although - a couple of users have complained about higher stale-count.

Is that something you are aware of that could be a sideeffect of the modifications?

LP seems to work just fine tho.
I'm not really quite sure. Two questions:

1) Are you using any of the hub modes? If so, which level?

2) Are you using my long polling or are you using blkmond?

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
July 01, 2011, 05:17:55 AM
 #91

Hi!

This should answer both of those:
Code:
BITCOIND_OPTS="-conf=/etc/bitcoin.conf -pid=/var/run/bitcoind.pid -hub=2 -pollpidfile=/var/run/pushpoold.pid -daemon"


In other words: hub level 2, and yes - built in LP.

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
July 01, 2011, 06:25:12 AM
 #92

Awesome. I'm glad all of my code bits are getting "real use". Let me know if people are still seeing a higher percentage of stale blocks. It's possible the daemon just needed some time to 'settle in' as it makes connections and so on. My next set of changes may fix this anyway, it's designed specifically to reduce latency in 'getwork', particularly around reorganizations and transaction list changes.

Make sure your CPU has the juice to run at "-hub=2". Note that with a multi-core CPU, things can be deceptive. Bitcoind isn't very well multi-threaded -- if one core is maxed out, there's a good chance bitcoind is CPU limited. My next changes after the changes I'm working on now are expected to be better multi-threading of the code that handles I/O.

You did remember not to run blkmond, right? It will conflict with native long polling.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
July 01, 2011, 03:42:10 PM
 #93

Hi!

We're running the "master node" upon a 2*Intel Xeon E5504 (8 cores in total);
http://data.fuskbugg.se/skalman02/4e0deb49dc5b9_tits.png

So far it looks good for me. And no, no blkmond.
But i think we're still experiencing a bit of lag / stales when LP is pushed, due to huge amounts of connections I'd think.

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
phorensic
Hero Member
*****
Offline Offline

Activity: 630



View Profile
July 01, 2011, 03:56:24 PM
 #94

Great progress!  One question, are we tracking this fork on github?  I didn't see the patches on the network graph there.
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
July 01, 2011, 06:16:33 PM
 #95

Great progress!  One question, are we tracking this fork on github?  I didn't see the patches on the network graph there.
No, I'm not doing that ... yet!

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
MrSam
Member
**
Offline Offline

Activity: 112



View Profile WWW
July 04, 2011, 10:12:57 PM
 #96

hmm, where can i get v0.5 ?

I tried 0.4 yesterday but wont work, bitcoind crashes after pushpool sends many
'HTTP request failed: The requested URL returned error: 500' and opened like 20k sockets in seconds
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
July 04, 2011, 10:17:16 PM
 #97

hmm, where can i get v0.5 ?

I tried 0.4 yesterday but wont work, bitcoind crashes after pushpool sends many
'HTTP request failed: The requested URL returned error: 500' and opened like 20k sockets in seconds

http://davids.webmaster.com/~davids/bitcoin-4diff.txt

I'm starting to suspect the newer issues are coming from pushpool, but I'm not 100% sure yet. I'm not going to have much time today, but I hope to investigate in more detail tomorrow.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
MrSam
Member
**
Offline Offline

Activity: 112



View Profile WWW
July 05, 2011, 05:39:10 PM
 #98

.
MrSam
Member
**
Offline Offline

Activity: 112



View Profile WWW
July 05, 2011, 11:41:07 PM
 #99

Joel,

I wanted to thank you ! The patch works really well now ..

The only issue that remains is that the USR1 signal has a bit of a delay.

pushpool debug:
[2011-07-05 23:36:1.710434] USR1 signal received, flushing LP waiters
if i check blockexplorer, that block was found at 2011-07-05 23:35:31)

[2011-07-05 23:33:36.427621] USR1 signal received, flushing LP waiters
real time 2011-07-05 23:33:24

[2011-07-05 23:30:36.337827] USR1 signal received, flushing LP waiters
real time 2011-07-05 23:30:20

When i test this on my mining rig at home, the rejected blocks are just those few seconds between the found block and the sig.



JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
July 06, 2011, 03:50:54 AM
 #100

Thanks for that report. I'll check the code and see if I can get the SIGUSR1 to be sent faster. I think that will solve half the problem though. While pushpool will issue fewer blocks that will turn into stale shares, bitcoind will still be processing the new block when all the new 'getwork' requests start piling in, and you'll get miner idles instead.

I have a fix for this that has other benefits as well, but the first time I tried to implement it, it was a disaster. I think I know the right way to do it now, having run out of wrong ways to do it already. What I'd like to do is pre-compute the skeleton 'getwork' block when a new block or new transaction comes in, rather than waiting to hijack the next 'getwork' request to do it. But to make this work, the locking has to be exactly right. The funny thing is, in my day job ten years ago, this was exactly the type optimization I specialized in. And now I can't seem to quite get it right.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
Pages: « 1 2 3 4 [5] 6 7 8 9 10 11 12 13 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!