Bitcoin Forum
March 28, 2024, 03:04:32 PM *
News: Latest Bitcoin Core release: 26.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 2 3 4 5 6 [7] 8 9 10 11 12 13 »  All
  Print  
Author Topic: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind  (Read 31392 times)
MrSam
Member
**
Offline Offline

Activity: 112
Merit: 10



View Profile WWW
July 06, 2011, 02:34:51 PM
 #121


Quote
If you want to make sure your blocks are more likely to get accepted quicker, find out the IPs of other large miner nodes and addnode those instead of just blindly making a ton of connections as this patch does.
Won't those generally be the nodes that are accepting incoming connections?

I tend to use https://en.bitcoin.it/wiki/Fallback_Nodes , but the wikibot doesnt seem to update it.
1711638272
Hero Member
*
Offline Offline

Posts: 1711638272

View Profile Personal Message (Offline)

Ignore
1711638272
Reply with quote  #2

1711638272
Report to moderator
1711638272
Hero Member
*
Offline Offline

Posts: 1711638272

View Profile Personal Message (Offline)

Ignore
1711638272
Reply with quote  #2

1711638272
Report to moderator
1711638272
Hero Member
*
Offline Offline

Posts: 1711638272

View Profile Personal Message (Offline)

Ignore
1711638272
Reply with quote  #2

1711638272
Report to moderator
In order to get the maximum amount of activity points possible, you just need to post once per day on average. Skipping days is OK as long as you maintain the average.
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
Matt Corallo
Hero Member
*****
Offline Offline

Activity: 755
Merit: 515


View Profile
July 06, 2011, 02:39:26 PM
 #122

B. Not even pool operators should be using this, a pool bitcoind should be accepting incoming connections, and if you do that, you will get plenty of connections really fast.
Yes, but from lots of small nodes. That doesn't help you very much.
True, but those nodes are probably only connected to other large nodes who listen, making two hops is really very fast anyway.

If you want to make sure your blocks are more likely to get accepted quicker, find out the IPs of other large miner nodes and addnode those instead of just blindly making a ton of connections as this patch does.
Won't those generally be the nodes that are accepting incoming connections?
Yep, but if you blindly make a ton of connections, you are still fairly unlikely to make connections to other miners, better to just addnode the best ones and make 10 connections instead of making 30 and only getting a couple large miners.

Bitcoin Core, rust-lightning, http://bitcoinfibre.org etc.
PGP ID: 07DF 3E57 A548 CCFB 7530  7091 89BB B866 3E2E65CE
JoelKatz
Legendary
*
Offline Offline

Activity: 1596
Merit: 1012


Democracy is vulnerable to a 51% attack.


View Profile WWW
July 06, 2011, 02:42:42 PM
 #123

I can't find the problem.
I've compiled it many times from scratch and every time the same result.
Can you test it on your end?
Are you saying it works fine with just my patches but if you add the 'getblock' patches, then you have problems? Are you using the 'getblock*' RPC calls that the patch adds? It looks to me like they access mapBlockIndex without holding the appropriate lock. Can you try this change: (And if this fixes it, you may want to share this fix with whoever wrote those patches!)

--- patch/rpc.cpp       2011-07-06 07:39:21.647871060 -0700
+++ rpc.cpp     2011-07-06 07:41:38.033681011 -0700
@@ -277,9 +277,13 @@ Value getblockbycount(const Array& param
 
     string blkname = strprintf("blk%d", height);
 
+    CBlock block;
     CBlockIndex* pindex;
     bool found = false;
 
+    CRITICAL_BLOCK(cs_main)
+    {
+
     for (map<uint256, CBlockIndex*>::iterator mi = mapBlockIndex.begin();
          mi != mapBlockIndex.end(); ++mi)
     {
@@ -295,9 +299,9 @@ Value getblockbycount(const Array& param
             "getblockbycount height\n"
             "Dumps the block existing at specified height");
 
-    CBlock block;
     block.ReadFromDisk(pindex);
     block.BuildMerkleTree();
+    }
 
     return BlockToValue(block);
 }
@@ -312,6 +316,10 @@ Value getblockbyhash(const Array& params
 
     uint256 hash;
     hash.SetHex(params[0].get_str());
+    CBlock block;
+
+    CRITICAL_BLOCK(cs_main)
+    {
 
     map<uint256, CBlockIndex*>::iterator mi = mapBlockIndex.find(hash);
     if (mi == mapBlockIndex.end())
@@ -319,9 +327,9 @@ Value getblockbyhash(const Array& params
 
     CBlockIndex* pindex = (*mi).second;
 
-    CBlock block;
     block.ReadFromDisk(pindex);
     block.BuildMerkleTree();
+    }
 
     return BlockToValue(block);
 }

I am an employee of Ripple. Follow me on Twitter @JoelKatz
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
talpan
Full Member
***
Offline Offline

Activity: 228
Merit: 100


View Profile
July 06, 2011, 05:19:17 PM
 #124

Thanks a lot for your help, so far it is all running perfect.
But I think the problem was somewhere else, after a little bit more testing I got an exception.
It had something to do with libboost, I did reinstall that and the problem was solved.

I'll send you small donation(haven't that much, yet Wink) in the next days.

And the patch is from jgarzik, I'll make a pull with your patch and of course your credits of course.

regards, talpan
somebadger
Member
**
Offline Offline

Activity: 170
Merit: 10



View Profile
July 06, 2011, 11:13:00 PM
 #125

fyi: renamed the repository to bitcoin4pools, new url is:

https://github.com/somebadger/bitcoin4pools

Any addition suggestions are welcome, will start a new thread later today and reference this thread and give credit to JoelKatz Smiley
JoelKatz
Legendary
*
Offline Offline

Activity: 1596
Merit: 1012


Democracy is vulnerable to a 51% attack.


View Profile WWW
July 06, 2011, 11:57:26 PM
 #126


I have some updates available at:
http://davids.webmaster.com/~davids/updates.diff.txt

This includes a performance improvement for ToHex, some style cleanups, some tuning changes to hub mode to reduce outbound connections, and a huge change to 'getwork'.

The change to 'getwork' is one I attempted before with not so good results, but I've rethought the way it's implemented, and this is much simpler. The idea is to update the 'skeleton block' used to build 'getwork' requests when a new transaction or block is received rather than waiting for the next 'getwork' request to come along to do it. The reason this makes a difference is that it means the 'getwork' request won't have to wait for the code handling the new block/transaction to release the 'cs_main' lock. Testing shows this really smooths out the 'spikes' in the 'getwork' response time.

However, this is a risky change. It is invasive to not just the code to get a work unit but also the code to confirm a valid block. Some people may wish to apply just the ToHex change and the cleanups.


I am an employee of Ripple. Follow me on Twitter @JoelKatz
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
MrSam
Member
**
Offline Offline

Activity: 112
Merit: 10



View Profile WWW
July 07, 2011, 12:36:52 AM
 #127

I'll run it on testnet to see what it gives
TeraPool
Newbie
*
Offline Offline

Activity: 42
Merit: 0


View Profile
July 07, 2011, 04:15:20 AM
 #128

fyi: renamed the repository to bitcoin4pools, new url is:

https://github.com/somebadger/bitcoin4pools

Any addition suggestions are welcome, will start a new thread later today and reference this thread and give credit to JoelKatz Smiley

Thanks for that!

Please make a post of said new through in this post, as I am watching this post for some time now as I assume many other pool creators are.
Caesium
Hero Member
*****
Offline Offline

Activity: 546
Merit: 500


View Profile
July 07, 2011, 10:13:50 AM
 #129

JoelKatz,

Managing to get segfaults when I send getwork to bitcoind with this set of patches now, backtrace below:

Code:
(gdb) bt
#0  0x00007f710a6e8394 in pthread_mutex_lock ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x0000000000447103 in boost::interprocess::interprocess_recursive_mutex::lock (this=<value optimized out>)
    at /usr/include/boost/interprocess/sync/posix/interprocess_recursive_mutex.hpp:53
#2  0x0000000000477d98 in Enter (this=0x0, nIndex=@0x9c1968, keypool=...)
    at util.h:245
#3  CCriticalBlock (this=0x0, nIndex=@0x9c1968, keypool=...) at util.h:260
#4  CWallet::ReserveKeyFromKeyPool (this=0x0, nIndex=@0x9c1968, keypool=...)
    at wallet.cpp:1041
#5  0x000000000047889c in CReserveKey::GetReservedKey (this=0x9c1960)
    at wallet.cpp:1117
#6  0x0000000000458562 in CreateNewBlock (reservekey=...) at main.cpp:2713
#7  0x000000000048f630 in GetWorkBlock (
    pmidstate=0x7f70feff9ef0 "C\377\376p\177", pdata=0x7f70feff9e30 "#",
    phash1=0x7f70feff9eb0 "\250\223\f\003", hashTarget=...) at rpc.cpp:1359
#8  0x000000000048fca2 in FastGetWork (id=...) at rpc.cpp:1461
#9  0x000000000049d03b in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:2031
#10 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#11 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
---Type <return> to continue, or q <return> to quit---
    at rpc.cpp:1981
#12 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#13 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#14 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#15 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#16 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#17 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#18 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#19 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#20 0x00007f710a6e5d8c in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#21 0x00007f7109c9004d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#22 0x0000000000000000 in ?? ()
(gdb)

This happening both with a fresh blockchain downloaded and a new wallet.

Let me know what I can give you to help debug Smiley

Tired of annoying signature ads? Ad block for signatures
JoelKatz
Legendary
*
Offline Offline

Activity: 1596
Merit: 1012


Democracy is vulnerable to a 51% attack.


View Profile WWW
July 07, 2011, 03:56:12 PM
 #130


What build is this exactly?

I am an employee of Ripple. Follow me on Twitter @JoelKatz
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
Jine (OP)
Sr. Member
****
Offline Offline

Activity: 403
Merit: 250


View Profile
July 07, 2011, 03:59:25 PM
 #131

wallet.cpp is in the github version - no the stable one.

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
Jine (OP)
Sr. Member
****
Offline Offline

Activity: 403
Merit: 250


View Profile
July 07, 2011, 04:21:28 PM
 #132


I have some updates available at:
http://davids.webmaster.com/~davids/updates.diff.txt

This includes a performance improvement for ToHex, some style cleanups, some tuning changes to hub mode to reduce outbound connections, and a huge change to 'getwork'.

The change to 'getwork' is one I attempted before with not so good results, but I've rethought the way it's implemented, and this is much simpler. The idea is to update the 'skeleton block' used to build 'getwork' requests when a new transaction or block is received rather than waiting for the next 'getwork' request to come along to do it. The reason this makes a difference is that it means the 'getwork' request won't have to wait for the code handling the new block/transaction to release the 'cs_main' lock. Testing shows this really smooths out the 'spikes' in the 'getwork' response time.

However, this is a risky change. It is invasive to not just the code to get a work unit but also the code to confirm a valid block. Some people may wish to apply just the ToHex change and the cleanups.



Quote
jine@bitcoins:~$ uptime
 18:20:27 up 9 days,  6:45,  3 users,  load average: 0.00, 0.19, 0.22

I say no more.

We had some serious issues a while ago with a lot of disconnects, so i took a shot and applied this....
0.00 load.. seriously? Tongue Cheesy

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
Jine (OP)
Sr. Member
****
Offline Offline

Activity: 403
Merit: 250


View Profile
July 07, 2011, 04:23:11 PM
 #133

... and now it froze.

Reverted back to old bitcoind... Sad

EDIT:
Quote
received: addr (31 bytes)
received: inv (37 bytes)
  got inventory: tx f9e2f6b050644cc94341  have
sending: addr (31 bytes)
received: addr (31 bytes)
sending: inv (37 bytes)
sending: addr (31 bytes)
sending: inv (37 bytes)
sending: inv (37 bytes)
received: addr (31 bytes)
received: addr (61 bytes)
sending: addr (31 bytes)
sending: inv (37 bytes)
sending: inv (37 bytes)
BitcoinMiner:
proof-of-work found
  hash: 00000000000004f00d620b3a2fd5e4ba19043e2053e9f4060a3795d2d00c5155
target: 0000000000000abbcf0000000000000000000000000000000000000000000000
CBlock(hash=00000000000004f00d62, ver=1, hashPrevBlock=0000000000000a6772c7, hashMerkleRoot=7e05d00288, nTime=1310055675, nBits=1a0abbcf, nNonce=568783248, vtx=37)
  CTransaction(hash=14b69a7cab, ver=1, vin.size=1, vout.size=1, nLockTime=0)
    CTxIn(COutPoint(0000000000, -1), coinbase 04cfbb0a1a035fc601)
    CTxOut(nValue=50.07110000, scriptPubKey=044dcd4ee79a23daccd12df3450a23)
  CTransaction(hash=da5496d248, ver=1, vin.size=16, vout.size=2, nLockTime=0)
    CTxIn(COutPoint(9c55619ba7, 1), scriptSig=3045022100f6c19dcd06577d)
    CTxIn(COutPoint(8d7f3d9c1d, 1), scriptSig=304602210095f7f99816769e)
    CTxIn(COutPoint(3255f7943d, 0), scriptSig=304602210089bbaa23e5be3c)
    CTxIn(COutPoint(a8be29ad98, 1), scriptSig=3045022100bab4ddaaf7da9c)
    CTxIn(COutPoint(abdc5e1a70, 1), scriptSig=30450220442f40e8b65ef407)

... new block?

    CTxOut(nValue=0.01000000, scriptPubKey=OP_DUP OP_HASH160 91d45c1f79fa)
  CTransaction(hash=a51043e11b, ver=1, vin.size=2, vout.size=2, nLockTime=0)
    CTxIn(COutPoint(6701c1b7a5, 0), scriptSig=30450220224680e1db9d648d)
    CTxIn(COutPoint(60cc6580e0, 0), scriptSig=304402207cb98fb1870604cd)
    CTxOut(nValue=0.00775252, scriptPubKey=OP_DUP OP_HASH160 c1e8b98c8c3d)
    CTxOut(nValue=0.02000000, scriptPubKey=OP_DUP OP_HASH160 401ce258491c)
  vMerkleTree: 14b69a7cab da5496d248 035078318f ea17127588 c0c344418b eef1e852f6 649507f180 ac905c95d4 0842b073a4 305372759d 9a6e764565 6794e2e09c d76296138f fb2fbfb31b 827e418d7f b142831de3 cd499e996d 5b43802900 68971974ef 29a1b4054b 1ca1ee01f1 f7abc8ab40 9f5768fc54 47394$
07/07/11 16:21 generated 50.0711
keypool keep 10145
AddToWallet 14b69a7cab  new
AddToWallet 68971974ef  update
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
accepted connection 178.47.108.93:60078
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo

* was unresponsive, so i runned ./bitcoind stop*


ThreadRPCServer method=stop
DBFlush(false)
addr.dat refcount=0
addr.dat flush
ThreadSocketHandler exiting
ThreadOpenConnections exiting
ThreadRPCServer exiting
blkindex.dat refcount=1
wallet.dat refcount=25
StopNode()







* restarted *





Bitcoin version 0.3.23-beta
Default data directory /root/.bitcoin

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
JoelKatz
Legendary
*
Offline Offline

Activity: 1596
Merit: 1012


Democracy is vulnerable to a 51% attack.


View Profile WWW
July 07, 2011, 05:13:31 PM
 #134

Getting the locking right on this patch (the one to precompute the skeleton block) is proving to be a pain.

I am an employee of Ripple. Follow me on Twitter @JoelKatz
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
Jine (OP)
Sr. Member
****
Offline Offline

Activity: 403
Merit: 250


View Profile
July 07, 2011, 05:22:47 PM
 #135

Is it any other way to do it? Possibly by not requesting work at all? Generating it in pushpoold perhaps?
I don't have any other ideas.

Both me and Eleuthria is talking about suicide on IRC (not serious tho... yet ;P) because of all the issues.. (Not only bitcoind-related, but server/ddos/stability/stupid users etc)

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
JoelKatz
Legendary
*
Offline Offline

Activity: 1596
Merit: 1012


Democracy is vulnerable to a 51% attack.


View Profile WWW
July 07, 2011, 05:27:21 PM
 #136

I've been there with the DDoS attacks, I know how awful that is.

Giel de Nijs suggested a binary protocol between bitcoind and pushpool. This would eliminate all the HTTP/JSON/hex/base64 nonsense entirely. It would also permit pushpool to request work units in blocks of, say, 10, reducing the handoffs between the two programs. This would mean long polling could be implemented in a saner way than a signal as well.

On the bright side, my current secret project will make all mining pools obsolete anyway and drastically reduce the feasibility of the 51% attack so there's some hope. Wink

I am an employee of Ripple. Follow me on Twitter @JoelKatz
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
Caesium
Hero Member
*****
Offline Offline

Activity: 546
Merit: 500


View Profile
July 07, 2011, 06:47:37 PM
 #137


What build is this exactly?


It's a git clone of bitcoin4pools as posted a while back:
https://github.com/somebadger/bitcoin4pools

As of commit https://github.com/somebadger/bitcoin4pools/commit/fbfd3ecbd322edd2c478135fb43e63b8ad4edeab

Which I notice happens to mention CReserveKey, which is also in the backtrace..

Tired of annoying signature ads? Ad block for signatures
Zagitta
Full Member
***
Offline Offline

Activity: 302
Merit: 100


Presale is live!


View Profile
July 07, 2011, 11:21:06 PM
 #138

Is it any other way to do it? Possibly by not requesting work at all? Generating it in pushpoold perhaps?
I don't have any other ideas.

Both me and Eleuthria is talking about suicide on IRC (not serious tho... yet ;P) because of all the issues.. (Not only bitcoind-related, but server/ddos/stability/stupid users etc)

That's an idea i've had myself and i'm trying to implement it in my own backend however i still haven't quit understood what exactly the data is calculated from (besides that the previous block is involved somehow) but once i get it, it should be pretty trival to implement considering the code for handling the block chain is already ported to C#: http://code.google.com/p/bitcoinsharp/

So if anyone got a simple explenation or links that explains the getwork process please share Smiley

somebadger
Member
**
Offline Offline

Activity: 170
Merit: 10



View Profile
July 11, 2011, 01:29:54 AM
Last edit: July 11, 2011, 10:57:50 PM by somebadger
 #139


What build is this exactly?


It's a git clone of bitcoin4pools as posted a while back:
https://github.com/somebadger/bitcoin4pools

As of commit https://github.com/somebadger/bitcoin4pools/commit/fbfd3ecbd322edd2c478135fb43e63b8ad4edeab

Which I notice happens to mention CReserveKey, which is also in the backtrace..

I have updated the repo, reverted back to .23 with patches and updates, plus --help additions.
.24 had some issues will get back on those once i get some time.
https://github.com/somebadger/bitcoin4pools/commits/v0.3.23-pool

ps. running now for 22hrs without a glitch Smiley
Caesium
Hero Member
*****
Offline Offline

Activity: 546
Merit: 500


View Profile
July 12, 2011, 04:47:19 PM
 #140

Looks good here too, cheers Smiley

Tired of annoying signature ads? Ad block for signatures
Pages: « 1 2 3 4 5 6 [7] 8 9 10 11 12 13 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!