Decentralized mining protocol standard: getblocktemplate (ASIC ready!)

[jgarzik]

Sigh. In that case, with p2pool being 400 GH of a 20TH network, then GBT is only going to benefit 2% of the miners. It's pretty obvious by now you can't force people to do what you think is best. If anything, GBT with non-p2pool pooled mining will make it more likely that pools will not want to pass on lots of transactions and will be worse than getwork.

How about adding an extension to GBT that has some kind of mode that sends a "suggested merkle root" for pooled mining?

Definitely interested in extensions that broaden the userbase as much as possible.

Even for bitcoind (rather than a pool server), extensions like this make sense.  The old "getwork" protocol did as much work as possible on the server side, thereby minimizing the miner's work -- notably the miner did not have to care about anything besides the block header itself.  No transactions or other parsing to worry about.

getblocktemplate is intended to replace getwork.  If there are improvements to be made, I'm quite open to that.

Ping me on IRC, if you want to rapidly iterate some changes to make bitcoind's GBT work better for a wider selection of mining softwares.

I did the final getblocktemplate polish, simplifying and cleaning things up from its original BIP 22/getmemorypool state.  If BIP 22 / BIP 23 are currently missing something you want...  I want to put it in there.

[1714036735]

1714036735

[1714036735]

1714036735

[Luke-Jr]

I do intend to improve BFGMiner's GBT implementation such that it transparently combines pooled and solo mining, at least.

That is planned in release BFGMiner's 3.0?

Probably that will have to wait for 3.1... 3.0 is the ASIC release, and that's coming up too soon.

I do intend to improve BFGMiner's GBT implementation such that it transparently combines pooled and solo mining, at least.

That's cool, but solo mining should not be an option - it should be mandatory. Else the whole project is just another house of cards.

Unfortunately, simply making it mandatory isn't practical. The closest we could come is giving strong incentives for doing so. For example, I intend to set up Eligius in such a way that those who are solo mining in combination can profit more by claiming the fees from transactions they accept over the base set Eligius provides.

[-ck]

Sigh. In that case, with p2pool being 400 GH of a 20TH network, then GBT is only going to benefit 2% of the miners. It's pretty obvious by now you can't force people to do what you think is best. If anything, GBT with non-p2pool pooled mining will make it more likely that pools will not want to pass on lots of transactions and will be worse than getwork.

How about adding an extension to GBT that has some kind of mode that sends a "suggested merkle root" for pooled mining?

Definitely interested in extensions that broaden the userbase as much as possible.

Even for bitcoind (rather than a pool server), extensions like this make sense.  The old "getwork" protocol did as much work as possible on the server side, thereby minimizing the miner's work -- notably the miner did not have to care about anything besides the block header itself.  No transactions or other parsing to worry about.

getblocktemplate is intended to replace getwork.  If there are improvements to be made, I'm quite open to that.

Ping me on IRC, if you want to rapidly iterate some changes to make bitcoind's GBT work better for a wider selection of mining softwares.

I did the final getblocktemplate polish, simplifying and cleaning things up from its original BIP 22/getmemorypool state.  If BIP 22 / BIP 23 are currently missing something you want...  I want to put it in there.

Thanks. When I get some extended block of time on IRC in the very near future, and you're online, I will most certainly.

[firefop]

I do intend to improve BFGMiner's GBT implementation such that it transparently combines pooled and solo mining, at least.

That's cool, but solo mining should not be an option - it should be mandatory. Else the whole project is just another house of cards.

Unfortunately, simply making it mandatory isn't practical. The closest we could come is giving strong incentives for doing so. For example, I intend to set up Eligius in such a way that those who are solo mining in combination can profit more by claiming the fees from transactions they accept over the base set Eligius provides.

Your efforts are noble, but it might be better for you to start working on alt currency, the one with really strong foundation. Let retards like
those 2 above, who despise you, me and anyone else not fitting their utopian views, finish off Bitcoin. Don't waste your time here, seriously.

I really like your prev-post with the graph substrata - you basically said everything I was thinking but couldn't articulate. Thank you for that!

However, I respectfully disagree. We need to encourage people like Luke to stick around, add even more sanity to the development of the network and slowly marginalize those who don't think in the right direction. If you look at the network as a whole, that's already happening. 2 years ago I wasn't here and you'll see more actual computer scientists become involved in development as bitcoin grows.

@Luke - incentives are nice - but I'd settle for having it be the default option - If you don't have time/inclination to build a custom install script to set that all up (when/if it's supported) I'd be happy to create and submit it to you for approval / publication.

[kano]

...
We need to encourage people like Luke to stick around, add even more sanity to the development of the network and slowly marginalize those who don't think in the right direction.
...

Sigh - another nut-case.

[firefop]

Sigh - another nut-case.

Hate all you want, but it doesn't invalidate anything I've said. The fact remains that certain posters are more interested in trash talking rather than contributing, while others are overworking themselves to try and get this whole bitcoin thing working to the point where we can make it grow into it's potential.

[kano]

Seriously - what are you looking for?

Decentralisation or control?

Pick one.

Saying that you want decentralisation ... except it has to be how you or Luke-jr or whoever else wants it ... isn't decentralisation.
As soon as you start deciding how things like mining MUST be done, you've lost the plot.

[Inaba]

Requiring a full bitcoind instance on a mining node is ludicrous, at least in the current implementation of bitcoind.  It's far, far, far too resource intensive to put on most serious mining setups.  I don't want to run heavy weight computing back end, just to satisfy the ridiculously complex blockchain processing for a machine that is intended to solely mine, which requires few resources.

Now, if you want to redesign a bitcoind that doesn't require heavy computing resources, then I'm listening, but until then, it's a non-starter.  A mining backend is a netbook, DD-WRT router, Raspberry Pi, Sheeva Plug or Android tablet.  Trying to run a full blockchain on any of those would be an exercise in futility.

[-ck]

Requiring a full bitcoind instance on a mining node is ludicrous, at least in the current implementation of bitcoind.  It's far, far, far too resource intensive to put on most serious mining setups.  I don't want to run heavy weight computing back end, just to satisfy the ridiculously complex blockchain processing for a machine that is intended to solely mine, which requires few resources.

Now, if you want to redesign a bitcoind that doesn't require heavy computing resources, then I'm listening, but until then, it's a non-starter.  A mining backend is a netbook, DD-WRT router, Raspberry Pi, Sheeva Plug or Android tablet.  Trying to run a full blockchain on any of those would be an exercise in futility.

Which means you pretty much unintentionally agree with my position: GBT in its current form is not advantageous to miners. Luckily jgarzik is open to modifying it based on what I have recommended. We will be talking about it in the near future.

[P_Shep]

Requiring a full bitcoind instance on a mining node is ludicrous, at least in the current implementation of bitcoind.  It's far, far, far too resource intensive to put on most serious mining setups.  I don't want to run heavy weight computing back end, just to satisfy the ridiculously complex blockchain processing for a machine that is intended to solely mine, which requires few resources.

Now, if you want to redesign a bitcoind that doesn't require heavy computing resources, then I'm listening, but until then, it's a non-starter.  A mining backend is a netbook, DD-WRT router, Raspberry Pi, Sheeva Plug or Android tablet.  Trying to run a full blockchain on any of those would be an exercise in futility.

I've been trying to get bitcoind on the router... not having much luck so far. Not entirely given up yet.

[Inaba]

Well, I've been mining with GBT for about a month now on a lower power netbook with no problems (and no bitcoind), so I'm not sure what you're referring to exactly?

[-ck]

Well, I've been mining with GBT for about a month now on a lower power netbook with no problems (and no bitcoind), so I'm not sure what you're referring to exactly?

https://bitcointalk.org/index.php?topic=108854.msg1284623#msg1284623

[Inaba]

Interesting... I would like to know more about that at some point.  I've not run into any problems and my server load is dramatically decreased.  Regardless, though, I'm not sure that pools ignoring some transactions in favor of others is necessarily a negative thing.  Right now, I limit the amount of transactions I put into any one block and I don't see any major issue with favoring higher transaction fee transactions over lower ones. 

If there are no higher fee transactions waiting, then the lower fee ones will get processed and that's as designed as far as I know.  I'm not saying it's a bad thing to "fix" GBT, but I've not run into any problems so far and I'd like to know at what point GBT becomes more cumbersome than getwork from a transaction standpoint.

[-ck]

Interesting... I would like to know more about that at some point.  I've not run into any problems and my server load is dramatically decreased.  Regardless, though, I'm not sure that pools ignoring some transactions in favor of others is necessarily a negative thing.  Right now, I limit the amount of transactions I put into any one block and I don't see any major issue with favoring higher transaction fee transactions over lower ones. 

If there are no higher fee transactions waiting, then the lower fee ones will get processed and that's as designed as far as I know.  I'm not saying it's a bad thing to "fix" GBT, but I've not run into any problems so far and I'd like to know at what point GBT becomes more cumbersome than getwork from a transaction standpoint.

The issue is the payload the longer a block takes to solve as the number of transactions rises. It would not be surprising if the pool was sending 100k sized payloads with all the transactions after 10 minutes on the block. While the load on the pool is definitely less, the data being spewed out to miners will eventually rise, and the more transactions and more popular bitcoin becomes, the larger the payload grows in a linear fashion with transactions. Eventually sending out 100kb payloads  say every 30 or 60 seconds from the pool to all your miners will become a burden. Prioritising transactions is your prerogative but with this disincentive to sending transactions we will see a deepbit like situation where pool ops will put a static upper limit on the amount of transactions in total they care about including. This is much worse for bitcoin in general. But with an extension to the gbt protocol that more closely resembles the merkle root base of stratum it can easily be fixed.

[kano]

Real world considerations should be part of GBT - not ideal world ...

The GBT data transfer should currently ALWAYS be high at the moment ...

Checking the bitcoind debug.log for the last 3 days:
The smallest poolsz was:
10/20/12 06:58:09 CTxMemPool::accept() : accepted a661e71a68 (poolsz 579)

Yes the minimum outstanding transactions for the last 3 days (which was after block 204102) was 579 waiting txns
(also note that I restarted my bitcoind 4 days ago - so that figure may be low ...)

So unless pools are ignoring lots of transactions, there should never have been a GBT workload sent from any GBT pool with fewer than a few hundred transactions in it for the past 3 days.

... but of course I've very little doubt that isn't the case ... since we know what pool software supports GBT.

[Inaba]

You forgot to specify what CPU you are using, also how much RAM total you have.  Also your HD usage.. 4.5GB. That is larger than the entire usage of my 500 GH/s farm.  Combined.  That 0 - 2% CPU usage you quote, I'd wager it's a modern Intel or AMD based CPU.  You just go ahead and try to run that on an ARM based CPU or a C6 or something, see how far you get (you'll never finish loading the block chain, new blocks will come in faster than you can process old ones). 

The blockchain is not suitable for lower power environments.  It simply isn't.  The blockchain would have to be reduced to under a few megabytes and RAM usage would have to be a few hundred kilobytes before it becomes feasible to run a full node on a mining rig. 

[Luke-Jr]

The blockchain would have to be reduced to under a few megabytes

Getting off-topic here, but this is already possible.

and RAM usage would have to be a few hundred kilobytes

This is completely unreasonable. Merely being a C++ program requires more than a few hundred KB.

[Inaba]

You forgot to specify what CPU you are using, also how much RAM total you have.  Also your HD usage.. 4.5GB. That is larger than the entire usage of my 500 GH/s farm.  Combined.  That 0 - 2% CPU usage you quote, I'd wager it's a modern Intel or AMD based CPU.  You just go ahead and try to run that on an ARM based CPU or a C6 or something, see how far you get (you'll never finish loading the block chain, new blocks will come in faster than you can process old ones).

Athlon 64 x2 4800+ and 2GB RAM. Bitcoin client mem usage can be as low as few MB and as high as 100+ MB, depending on something,
no idea what exactly. HDD usage value stands for client installation and stuff in datadir. I have no clue about ARM or C6, but here's the
nasty part = you invested huge money into system that can't help Bitcoin network if all pools go down. That's quite a gamble, you know.

The blockchain is not suitable for lower power environments.  It simply isn't.  The blockchain would have to be reduced to under a few megabytes and RAM usage would have to be a few hundred kilobytes before it becomes feasible to run a full node on a mining rig. 

Nowhere I said my suggestion must be implemented right now. Think long-term. Step by step - in right direction - and it can be done.

Are you using the "royal" "I" in this case?  Because I did no such thing - I can stand up a new pool on my local network in about 30 minutes if all the pools suddenly went down.  But anyway, comparing an A64 X2 to an ARM processor is not exactly a fair comparison.  Go ahead and try to load the blockchain on an old netbook and see how far you get before everything grinds to a halt.

[DavinciJ15]

Go ahead and try to load the blockchain on an old netbook and see how far you get before everything grinds to a halt.

I did that on a Asus EEE and it took 2 weeks!
When I turn it on after a day it's about 1 hour to catch up.

Crazy!

Anyhow: Luke-jr!

Personally I think stratum is better, however, if you want to win create a pool back end that's easy to use.   Better doesn't matter when it comes to wanting to accomplish a task.  I want a pool, and so does many other people!  So create a sold backend that's easy to config and setup and you win.

I know you have created Eloipool but I'm willing to bet it's not so easy to setup.

Another option is to create vm for people to download with everything working only thing that the end user needs to do is add a username and password to the worker table and they can mine and shares can be obtained from the shares table.

Just my 2 Satoshis.

[Luke-Jr]

Personally I think stratum is better, however, if you want to win

Win is when Stratum undergoes the BIP process and gets feature parity with GBP. As I understand it, slush is taking it down that road now.

create a pool back end that's easy to use.

Yes, I did that about a year ago...

I know you have created Eloipool but I'm willing to bet it's not so easy to setup.

Easier than the alternatives

Another option is to create vm for people to download with everything working only thing that the end user needs to do is add a username and password to the worker table and they can mine and shares can be obtained from the shares table.

Eh, why do we need passwords?