Bitcoin Forum
December 06, 2016, 09:57:07 PM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: « 1 2 3 4 5 [6] 7 8 9 10 11 12 13 »  All
  Print  
Author Topic: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind  (Read 29381 times)
somebadger
Jr. Member
*
Offline Offline

Activity: 56



View Profile WWW
July 06, 2011, 04:22:40 AM
 #101

i just merged your diff with the latest from the official bitcoin git,

https://github.com/somebadger/bitcoin4pools

I also added the following to --help

Code:
 -pollpidfile=<file>      Pushpoold Pid file for long polling support
  -hub=<n>      Hub mode for pool operators

other than some manual intervention and fixing CReserveKey::CReserveKey(CWallet*) syntax for the latest build it wasnt too much of a headache. enough though that i'm keeping it in git now Tongue

ps. i hate manually doing .rej's lol

http://pool.betcoin.co/banners/1.sig.png <-- Click here to join me Smiley
--
Have I been helpful ? Kick some coins over to 1BLkDhijWne8dAhK4TA9dceUC6C6K6Fite Smiley
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
July 06, 2011, 04:50:22 AM
 #102

If you get the chance, a pull request for the two resource leak fixes would be nice.

That would be this change to net.cpp:

     // Send and receive from sockets, accept connections
-    pthread_t hThreadSocketHandler = CreateThread(ThreadSocketHandler, NULL, true);
+    CreateThread(ThreadSocketHandler, NULL);

And this change to util.h:

         return (pthread_t)0;
     }
     if (!fWantHandle)
+    {
+        pthread_detach(hthread);
         return (pthread_t)-1;
+    }

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
somebadger
Jr. Member
*
Offline Offline

Activity: 56



View Profile WWW
July 06, 2011, 05:08:34 AM
 #103

If you get the chance, a pull request for the two resource leak fixes would be nice.

That would be this change to net.cpp:

     // Send and receive from sockets, accept connections
-    pthread_t hThreadSocketHandler = CreateThread(ThreadSocketHandler, NULL, true);
+    CreateThread(ThreadSocketHandler, NULL);

And this change to util.h:

         return (pthread_t)0;
     }
     if (!fWantHandle)
+    {
+        pthread_detach(hthread);
         return (pthread_t)-1;
+    }


Patched and submitted pull request
https://github.com/bitcoin/bitcoin/pull/385

http://pool.betcoin.co/banners/1.sig.png <-- Click here to join me Smiley
--
Have I been helpful ? Kick some coins over to 1BLkDhijWne8dAhK4TA9dceUC6C6K6Fite Smiley
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
July 06, 2011, 05:33:42 AM
 #104

Thanks.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
somebadger
Jr. Member
*
Offline Offline

Activity: 56



View Profile WWW
July 06, 2011, 06:50:37 AM
 #105

No probs Wink

http://pool.betcoin.co/banners/1.sig.png <-- Click here to join me Smiley
--
Have I been helpful ? Kick some coins over to 1BLkDhijWne8dAhK4TA9dceUC6C6K6Fite Smiley
TeraPool
Jr. Member
*
Offline Offline

Activity: 42


View Profile
July 06, 2011, 06:55:34 AM
 #106

i just merged your diff with the latest from the official bitcoin git,

https://github.com/somebadger/bitcoin

I also added the following to --help

Code:
  -pollpidfile=<file>      Pushpoold Pid file for long polling support
  -hub=<n>      Hub mode for pool operators

other than some manual intervention and fixing CReserveKey::CReserveKey(CWallet*) syntax for the latest build it wasnt too much of a headache. enough though that i'm keeping it in git now Tongue

ps. i hate manually doing .rej's lol

QUESTION

JoelKatz's optimizations here will probably never make it into the official bitcoin github account, will they? Because they typically only make sense for pool operators or am I wrong?

The patches I believe will definitely make it in because that can benefit everybody. (Great work!)

So if anyone downloads the latest "v0.3.24rc3" or later from somebadger's github account, then it will contain JoelKatz's latest "diff" as seen here http://davids.webmaster.com/~davids/bitcoin-3diff.txt Huh

Will somebadger's fork always contain the new pool related code? Just wondering if I should just make my own github fork of bitcoin just for my own pool so I can keep track of this and any future similar changes.

Great work JoelKatz. Thank you for that contribution to the community!

somebadger
Jr. Member
*
Offline Offline

Activity: 56



View Profile WWW
July 06, 2011, 07:13:19 AM
 #107

QUESTION

JoelKatz's optimizations here will probably never make it into the official bitcoin github account, will they? Because they typically only make sense for pool operators or am I wrong?

The patches I believe will definitely make it in because that can benefit everybody. (Great work!)

So if anyone downloads the latest "v0.3.24rc3" or later from somebadger's github account, then it will contain JoelKatz's latest "diff" as seen here http://davids.webmaster.com/~davids/bitcoin-3diff.txt Huh

Will somebadger's fork always contain the new pool related code? Just wondering if I should just make my own github fork of bitcoin just for my own pool so I can keep track of this and any future similar changes.

Great work JoelKatz. Thank you for that contribution to the community!

My plan is to keep the fork upto date with any improvements for pool operators, i'll also be adding bitcoin pulls that help pool performance

ps. its patched based on http://davids.webmaster.com/~davids/bitcoin-4diff.txt and modifications to make it apply to the latest build

http://pool.betcoin.co/banners/1.sig.png <-- Click here to join me Smiley
--
Have I been helpful ? Kick some coins over to 1BLkDhijWne8dAhK4TA9dceUC6C6K6Fite Smiley
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
July 06, 2011, 07:16:52 AM
 #108

JoelKatz's optimizations here will probably never make it into the official bitcoin github account, will they? Because they typically only make sense for pool operators or am I wrong?
Some of them may not, some of them might. Once I'm convinced this code is stable, I can work on getting some of them into the main distribution. I think the hub mode changes and native long polling could get in. The RPC turbocharge and keepalive fixes could.

Some of the stuff in the latest diff might get in too. Stashing the RPC user/pass, avoiding re-processing partial blocks, faster base64 decode, and faster bin2hex all might get in. (Change 3diff to 4diff to see them.)

Some code should never get in. The JSON bypass code is awful. Perhaps a change to using Jansson is possible. But that's a pretty big change.

Quote
The patches I believe will definitely make it in because that can benefit everybody. (Great work!)
...
Great work JoelKatz. Thank you for that contribution to the community!
Thanks.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
MrSam
Member
**
Offline Offline

Activity: 112



View Profile WWW
July 06, 2011, 10:09:06 AM
 #109

Thanks for that report. I'll check the code and see if I can get the SIGUSR1 to be sent faster. I think that will solve half the problem though. While pushpool will issue fewer blocks that will turn into stale shares, bitcoind will still be processing the new block when all the new 'getwork' requests start piling in, and you'll get miner idles instead.

I have a fix for this that has other benefits as well, but the first time I tried to implement it, it was a disaster. I think I know the right way to do it now, having run out of wrong ways to do it already. What I'd like to do is pre-compute the skeleton 'getwork' block when a new block or new transaction comes in, rather than waiting to hijack the next 'getwork' request to do it. But to make this work, the locking has to be exactly right. The funny thing is, in my day job ten years ago, this was exactly the type optimization I specialized in. And now I can't seem to quite get it right.


I solved it with what i call the "pushpool feelgood patch". This patch accepts solutions that he knows are from the prev block, and marks them that way in the database rather then rejecting them to the miner.

This way the users have 0 rejected shares, and they feel good!

Thanks again for all your work, and i'll be following all of your updates Smiley
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
July 06, 2011, 10:15:09 AM
 #110

So you're actually lying to your users, about their amount of shares?  Shocked
Or are you paying for those invalid shares as well?

/ Jim

EDIT: I feel rather bad about that, to be honest.

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
MrSam
Member
**
Offline Offline

Activity: 112



View Profile WWW
July 06, 2011, 10:19:53 AM
 #111

So you're actually lying to your users, about their amount of shares?  Shocked
Or are you paying for those invalid shares as well?


Well the idea of pushpool shares at difficulty 1 is that it serves as proof of work, and i think that it is a proof of work, it's not their fault that they submit work that has become invalid.

So yes, i accept this type 'recentstale' for payout.

Why feel bad about this ? proof of work is proof of work ?
talpan
Full Member
***
Offline Offline

Activity: 228


View Profile
July 06, 2011, 10:22:24 AM
 #112

Hello,

I've patched the official .23 with patch 4.
so far so good, no errors during compiling.
But when i start bitcoind the cpu spikes after 40-50s through the roof, without using it.

regards, talpan
Jine
Sr. Member
****
Offline Offline

Activity: 405


View Profile
July 06, 2011, 10:28:12 AM
 #113

So yes, i accept this type 'recentstale' for payout.

So i would be able to resend the same stale-share, over and over again? Or just keep sending stales - it doesn't help the pool at ALL, but i get paid more? Cheesy
No seriously, I'm disappointed.

Previous founder of Bit LC Inc. | I've always loved the idea of bitcoin.
MrSam
Member
**
Offline Offline

Activity: 112



View Profile WWW
July 06, 2011, 10:33:45 AM
 #114

So yes, i accept this type 'recentstale' for payout.

So i would be able to resend the same stale-share, over and over again? Or just keep sending stales - it doesn't help the pool at ALL, but i get paid more? Cheesy
No seriously, I'm disappointed.

I never said that i accept stales .. only those specific recentstales ..

Happy disappointment day !

https://www.triplemining.com - Probably the best pool in the world
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
July 06, 2011, 10:36:29 AM
 #115

Hello,

I've patched the official .23 with patch 4.
so far so good, no errors during compiling.
But when i start bitcoind the cpu spikes after 40-50s through the roof, without using it.

regards, talpan
Hmm, that's pretty strange. Can you confirm you're using version 0.95?

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
talpan
Full Member
***
Offline Offline

Activity: 228


View Profile
July 06, 2011, 11:47:06 AM
 #116

Hi,

Sorry I made a mistake, I'm using a custom .23 with a little patch.
Without this patch and with the patch from everythings fine.
But I can't figure out whats wrong with this patch:

Code:
diff -up orig/rpc.cpp new/rpc.cpp
--- orig/rpc.cpp
+++ new/rpc.cpp
@@ -184,6 +184,149 @@
 }
 
 
+Value BlockToValue(CBlock &block)
+{
+    Object obj;
+    obj.push_back(Pair("hash", block.GetHash().ToString().c_str()));
+    obj.push_back(Pair("version", block.nVersion));
+    obj.push_back(Pair("prev_block", block.hashPrevBlock.ToString().c_str()));
+    obj.push_back(Pair("mrkl_root", block.hashMerkleRoot.ToString().c_str()));
+    obj.push_back(Pair("time", (uint64_t)block.nTime));
+    obj.push_back(Pair("bits", (uint64_t)block.nBits));
+    obj.push_back(Pair("nonce", (uint64_t)block.nNonce));
+    obj.push_back(Pair("n_tx", (int)block.vtx.size()));
+    obj.push_back(Pair("size", (int)::GetSerializeSize(block, SER_NETWORK)));
+
+    Array tx;
+    for (int i = 0; i < block.vtx.size(); i++) {
+    Object txobj;
+
+ txobj.push_back(Pair("hash", block.vtx[i].GetHash().ToString().c_str()));
+ txobj.push_back(Pair("version", block.vtx[i].nVersion));
+ txobj.push_back(Pair("lock_time", (uint64_t)block.vtx[i].nLockTime));
+ txobj.push_back(Pair("size",
+ (int)::GetSerializeSize(block.vtx[i], SER_NETWORK)));
+
+ Array tx_vin;
+ for (int j = 0; j < block.vtx[i].vin.size(); j++) {
+     Object vino;
+
+     Object vino_outpt;
+
+     vino_outpt.push_back(Pair("hash",
+     block.vtx[i].vin[j].prevout.hash.ToString().c_str()));
+     vino_outpt.push_back(Pair("n", (uint64_t)block.vtx[i].vin[j].prevout.n));
+
+     vino.push_back(Pair("prev_out", vino_outpt));
+
+     if (block.vtx[i].vin[j].prevout.IsNull())
+     vino.push_back(Pair("coinbase", HexStr(
+ block.vtx[i].vin[j].scriptSig.begin(),
+ block.vtx[i].vin[j].scriptSig.end(), false).c_str()));
+     else
+     vino.push_back(Pair("scriptSig",
+ block.vtx[i].vin[j].scriptSig.ToString().c_str()));
+     if (block.vtx[i].vin[j].nSequence != UINT_MAX)
+     vino.push_back(Pair("sequence", (uint64_t)block.vtx[i].vin[j].nSequence));
+
+     tx_vin.push_back(vino);
+ }
+
+ Array tx_vout;
+ for (int j = 0; j < block.vtx[i].vout.size(); j++) {
+     Object vouto;
+
+     vouto.push_back(Pair("value",
+     (double)block.vtx[i].vout[j].nValue / (double)COIN));
+     vouto.push_back(Pair("scriptPubKey",
+ block.vtx[i].vout[j].scriptPubKey.ToString().c_str()));
+
+     tx_vout.push_back(vouto);
+ }
+
+ txobj.push_back(Pair("in", tx_vin));
+ txobj.push_back(Pair("out", tx_vout));
+
+ tx.push_back(txobj);
+    }
+
+    obj.push_back(Pair("tx", tx));
+
+    Array mrkl;
+    for (int i = 0; i < block.vMerkleTree.size(); i++)
+    mrkl.push_back(block.vMerkleTree[i].ToString().c_str());
+
+    obj.push_back(Pair("mrkl_tree", mrkl));
+
+    return obj;
+}
+
+
+Value getblockbycount(const Array& params, bool fHelp)
+{
+    if (fHelp || params.size() != 1)
+        throw runtime_error(
+            "getblockbycount height\n"
+            "Dumps the block existing at specified height");
+
+    int64 height = params[0].get_int64();
+    if (height > nBestHeight)
+        throw runtime_error(
+            "getblockbycount height\n"
+            "Dumps the block existing at specified height");
+
+    string blkname = strprintf("blk%d", height);
+
+    CBlockIndex* pindex;
+    bool found = false;
+
+    for (map<uint256, CBlockIndex*>::iterator mi = mapBlockIndex.begin();
+         mi != mapBlockIndex.end(); ++mi)
+    {
+    pindex = (*mi).second;
+ if ((pindex->nHeight == height) && (pindex->IsInMainChain())) {
+ found = true;
+ break;
+ }
+    }
+
+    if (!found)
+        throw runtime_error(
+            "getblockbycount height\n"
+            "Dumps the block existing at specified height");
+
+    CBlock block;
+    block.ReadFromDisk(pindex);
+    block.BuildMerkleTree();
+
+    return BlockToValue(block);
+}
+
+
+Value getblockbyhash(const Array& params, bool fHelp)
+{
+    if (fHelp || params.size() != 1)
+        throw runtime_error(
+            "getblockbyhash hash\n"
+            "Dumps the block with specified hash");
+
+    uint256 hash;
+    hash.SetHex(params[0].get_str());
+
+    map<uint256, CBlockIndex*>::iterator mi = mapBlockIndex.find(hash);
+    if (mi == mapBlockIndex.end())
+        throw JSONRPCError(-18, "hash not found");
+
+    CBlockIndex* pindex = (*mi).second;
+
+    CBlock block;
+    block.ReadFromDisk(pindex);
+    block.BuildMerkleTree();
+
+    return BlockToValue(block);
+}
+
+
 Value getconnectioncount(const Array& params, bool fHelp)
 {
     if (fHelp || params.size() != 0)
@@ -1423,6 +1566,8 @@
 {
     make_pair("help",                  &help),
     make_pair("stop",                  &stop),
+    make_pair("getblockbycount",       &getblockbycount),
+    make_pair("getblockbyhash",        &getblockbyhash),
     make_pair("getblockcount",         &getblockcount),
     make_pair("getblocknumber",        &getblocknumber),
     make_pair("getconnectioncount",    &getconnectioncount),
@@ -2113,6 +2258,7 @@
         if (strMethod == "listtransactions"       && n > 1) ConvertTo<boost::int64_t>(params[1]);
         if (strMethod == "listtransactions"       && n > 2) ConvertTo<boost::int64_t>(params[2]);
         if (strMethod == "listaccounts"           && n > 0) ConvertTo<boost::int64_t>(params[0]);
+        if (strMethod == "getblockbycount"        && n > 0) ConvertTo<boost::int64_t>(params[0]);
         if (strMethod == "sendmany"               && n > 1)
         {
             string s = params[1].get_str();

regards, talpan
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
July 06, 2011, 11:58:02 AM
 #117

That patch looks safe to me. I don't think that explains your issue.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
talpan
Full Member
***
Offline Offline

Activity: 228


View Profile
July 06, 2011, 01:40:26 PM
 #118

I can't find the problem.
I've compiled it many times from scratch and every time the same result.
Can you test it on your end?
Matt Corallo
Hero Member
*****
Offline Offline

Activity: 751


View Profile
July 06, 2011, 02:09:54 PM
 #119

Some of them may not, some of them might. Once I'm convinced this code is stable, I can work on getting some of them into the main distribution. I think the hub mode changes and native long polling could get in. The RPC turbocharge and keepalive fixes could.

Some of the stuff in the latest diff might get in too. Stashing the RPC user/pass, avoiding re-processing partial blocks, faster base64 decode, and faster bin2hex all might get in. (Change 3diff to 4diff to see them.)

Some code should never get in. The JSON bypass code is awful. Perhaps a change to using Jansson is possible. But that's a pretty big change.
1) Micro-optimizations
Havent had much of a chance to really look through these, but some of them look pull-able others maybe not, but I haven't looked too closely.

2) Native Long Polling Support
This one I'm not sure about, its a useful feature to be sure, but I'm not sure about pulling yet.  When the client codebase gets cleaned up a ton, features like this could very well be pulled, but IMHO not so much yet.

3) A fix from Luke Dash Jr.
Almost certainly can be pulled.

4) Hub Mode
No, this is really not a good change for several reasons.  A. the percent of nodes accepting incoming connections is already low enough, encouraging people to start making a ridiculous number of outgoing connections and filling up the few nodes who do accept incoming connection's connections slots is not a good idea at all.  B. Not even pool operators should be using this, a pool bitcoind should be accepting incoming connections, and if you do that, you will get plenty of connections really fast.  If you want to make sure your blocks are more likely to get accepted quicker, find out the IPs of other large miner nodes and addnode those instead of just blindly making a ton of connections as this patch does.

5) RPC Turbocharge
Absolutely, there is already a asio patch for this waiting as a pull request, so some version of asio/keepalive support/threaded RPC will absolutely be merged at some point in the not-too-distant future.

6) Resource Leak Fix
Absolutely, thanks somebadger for the pull.

Bitcoin Ubuntu PPA maintainer - donate to me personally: 1JBMattRztKDF2KRS3vhjJXA7h47NEsn2c
http://bitcoinrelaynetwork.org maintainer
PGP ID: 07DF 3E57 A548 CCFB 7530  7091 89BB B866 3E2E65CE
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
July 06, 2011, 02:32:36 PM
 #120

No, this is really not a good change for several reasons.  A. the percent of nodes accepting incoming connections is already low enough, encouraging people to start making a ridiculous number of outgoing connections and filling up the few nodes who do accept incoming connection's connections slots is not a good idea at all.
That's not a point I had considered. I think I may make some adjustments.

Quote
B. Not even pool operators should be using this, a pool bitcoind should be accepting incoming connections, and if you do that, you will get plenty of connections really fast.
Yes, but from lots of small nodes. That doesn't help you very much.

Quote
If you want to make sure your blocks are more likely to get accepted quicker, find out the IPs of other large miner nodes and addnode those instead of just blindly making a ton of connections as this patch does.
Won't those generally be the nodes that are accepting incoming connections?

Thanks for the feedback. You've given me a few ideas for ways to make this work better.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
Pages: « 1 2 3 4 5 [6] 7 8 9 10 11 12 13 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!