Decentralized mining protocol standard: getblocktemplate (ASIC ready!)

[Luke-Jr]

Since Eleuthria and slush have both recently announced their own developed-behind-closed-doors mining protocols, I felt it appropriate to make a thread here for end-user discussion of the "getblocktemplate" standard which they intend to compete with. As a poolserver author, a pool operator, a miner author, and a contributor to the reference client, my life is not improved by having yet another protocol to support, and what's bad for the authors of the infrastructure tools is also bad for all their users.

Getblocktemplate Features

• Decentralized: the miners decide what they will mine, and make their own blocks - no more blindly giving up mining control to pool operators
• ASIC-ready: miners can produce as much work as they need with a single request
• Usable yesterday: Eligius has supported it since February, and more pools real soon
• Solo mining: just point your miner to bitcoind 0.7 and mine by yourself (no bitcoind load issues)
• Simple: the basic implementation requires only minimal block building
• Extensible: extensions are available today for miners who wish to take more control over their miners, and future improvements can easily be added later

Documentation

See the new Bitcoin Wiki page for getblocktemplate, with lots of useful resources including end user and developer documentation.

Technical specifications:

• BIP 22: getblocktemplate - Fundamentals
• BIP 23: getblocktemplate - Pooled Mining

Supported by... (post or PM me to get yours added)

Poolservers (providing getblocktemplate to miners):

• Eloipool
• bitcoind 0.7+
• ecoinpool (planned)

Mining software:

• BFGMiner 2.8+
• cgminer 2.9+
• Butterfly Labs's EasyMiner (planned)
• poclbm (planned)
• Ufasoft Bitcoin Miner (planned)

Libraries for mining software:

• libblkmaker (C library for easy integration into any miner)
• python-blkmaker (Python port of libblkmaker)

Mining pools:

• BitArena.net
• EclipseMC
• Eligius
• Bitparking merged mining pool (planned)
• TripleMining (planned)

History/Background

Getblocktemplate's origins trace back to forrestv's getmemorypool JSON-RPC method for bitcoind. He created it so that his pool (P2Pool) could piggy-back on bitcoind so as to avoid harming the Bitcoin network (up to this point, it mined only empty blocks that never confirmed transactions). Having been fighting with scalability problems in pushpool/bitcoind for months on my pool (Eligius), I set out to implement a fast pool server using getmemorypool to do its own block production (this became Eloipool, this first open source makes-its-own-blocks poolserver). Other poolservers also implemented block creation using getmemorypool over the months following.

At about the same time, interest in decentralizing pooled mining became a hot topic. While BitPenny had initially released its own decentralized mining proxy months prior, P2Pool's implementation became rapidly popular. Anyone involved in Bitcoin mining protocols could see the need to move control of block creation back into the hands of the miners. Unfortunately, both BitPenny and P2Pool had used very pool-specific proprietary protocols to implement their decentralization. On the other hand, getmemorypool was almost a perfect fit for the task - it just lacked support for the now-ubiquitous pooled mining extensions that had developed around the getwork protocol over time.

In February, I implemented and deployed a first draft of getmemorypool mining support in Eloipool (and on Eligius) along with a proof-of-concept getwork proxy (now known as gmp-proxy), adding revisions as needed to function as a general-purpose decentralized mining protocol. After I had confirmed it was working, I documented and proposed it on the Bitcoin development mailing list for review on February 28th, where discussion began on what was missing and what needed to be changed or clarified. During the following few months, a number of others, both developers and testers, provided constructive criticism and suggestions, which were integrated into the standard. I also actively encouraged participation in the development of the standard among pool operators and poolserver authors, especially as it became necessary to move forward into the ASIC "mining generation". Eventually, it was decided it would be best to rename it to the more appropriate "getblocktemplate" name and drop backward compatibility with getmemorypool for simplicity. The standard was split into two pieces and the technical specification can be found in BIP 22 and BIP 23.

Support for getblocktemplate mining was announced for BFGMiner a week ago, and is currently functional (implementing the basics is quite easy) though not entirely optimized yet. A number of other software developers and pools have told me they intend to add getblocktemplate support soon.

(side note: all of this occurred before Eleuthria or slush made their protocols public, and neither of them chose to participate in the open development of the getblocktemplate standard)

Contribute

All that being said, there is always room for improvement. Reception of the proprietary proposals strongly suggests getblocktemplate is in desperate need of good miner-side example documentation. If you have the time to help bring getblocktemplate to everyone sooner in any way, please get in touch

Thanks to... (in no particular order)

• Forrest Voight
• Pieter Wuille
• Jeff Garzik
• Gregory Maxwell
• Geir Harald Hansen
• Gavin Andresen
• Stefan Thomas
• Michael Grønager
• ...anyone else I forgot to mention

[1713556154]

1713556154

[1713556154]

1713556154

[gmaxwell]

I haven't yet read about about these other protocols, but I'm really disappointed to see more protocol proliferation. Getblocktemplate is well established, peer reviewed, and is good for bitcoin's decentralized security. I hope GBT gains whatever features its missing so it can subsume these things.  Having yet another protocol to cope with is a bummer.

[Graet]

I haven't yet read about about these other protocols, but I'm really disappointed to see more protocol proliferation. Getblocktemplate is well established, peer reviewed, and is good for bitcoin's decentralized security. I hope GBT gains whatever features its missing so it can subsume these things.  Having yet another protocol to cope with is a bummer.

It shows me interest from varied developers is strong - is this not good for Bitcoin?
what is wrong with proposing a new way for miners to communicate with pools, it isn't like it is changing a Bitcoin protocol it is a mining change.
there is no single pool software everyone uses, there is no single mining software everyone uses - I for one am interested at looking at options.

Thanks to all the developers taking interest in this topic and offering some options to consider

https://bitcointalk.org/index.php?topic=108533.0

[eleuthria]

I started my development privately because I wasn't sure it was ever going to be completed.  I did know one thing.  The problem that slush and I were both addressing separately (though we wound up doing almost the exact same thing) was that POOLS will not be able to cope with ever increasing speeds, because pooled mining is using the standard getwork protocol that is used in bitcoin.

Our protocols were designed to be specific for pooled mining.  The primary Bitcoin project does not need to be polluted with new protocols and RPC commands for mining related purposes.  Bitcoin is not about mining, and developer time should be spent on more important features that matter.

[slush]

Well, I don't want to start flamewar and personal fights, really. I did the Stratum just because of my requirements, not because I wanted to compete with anybody. Let me explain it:

Actually I see the difference between Stratum and GMP as Electrum client and Satoshi client. You can achieve similar results with both of them, but every of such solution has pros and cons.

Two days ago I summarized my reasons for Stratum here: http://mining.bitcoin.cz/stratum-mining

I was comparing it mostly with the getwork, because it's the only used protocol by real miners now (voting by the hashrate). Maybe I should write up longer comparsion between Stratum and GMP. Back to the topic.

THere's really no need to have one super-protocol which fits all needs. By the way, I'm using getblocktemplate in the Stratum implementation myself and I consider it as a great way how to relocate pool logic outside the bitcoind. GBT is surely very complete protocol and you can do anything what you want, *if* you know how. I see GBT as very complex and very hard to implement full specs without breaking anything. Luke will say the oposite, but GBT has simply unnecesary compexity for the environment of pooled mining. You know, many optional features etc. But pooled miners don't need that. Period.

In the oposite we proposed complete stack for pooled mining, which *is* very lightweight and straightforward to implement. Simply said, there's not a *single* optional feature, which is possibly the biggest benefit of whole concept. Miners don't want tons of options, they need simple and clear algorithm how to talk with the server. It includes also exact specification of used transports and mechanisms for real-time broadcasting new jobs to the client. Current BIPs don't care about this anyhow, because they don't need to. Both are just for different use cases.

Luke told me yesterday, that GBT is not bounded to HTTP, that it can be implement with any transport. By mistake he expressed the exact reason why it isn't suitable for pooled mining. It gives miners/pool ops degree of freedom which don't help anything, just mess the source code of miners and pools, because interpretation of the current protocol "as is" is ambiguous and cannot be used by some additonal specification about transports etc.

By the way, pool servers must be super-optimalized for handle high loads. Current Stratum protocol implementing coinbase templates is definitely the most optimized solution for given purpose and I doubt that any getblocktemplate pool would be more optimized that this.

[Luke-Jr]

what is wrong with proposing a new way for miners to communicate with pools, it isn't like it is changing a Bitcoin protocol it is a mining change.
there is no single pool software everyone uses, there is no single mining software everyone uses - I for one am interested at looking at options.

The problem is more about when people don't work together with others. Everyone was (and is) welcome to contribute to the development of GBT, so why quietly reinvent an incompatible protocol on the side? GBT vs StratumMP is like W3c HTML/CSS vs Microsoft Internet Explorer "quirks" HTML/CSS.

The problem that slush and I were both addressing separately (though we wound up doing almost the exact same thing) was that POOLS will not be able to cope with ever increasing speeds, because pooled mining is using the standard getwork protocol that is used in bitcoin.

Our protocols were designed to be specific for pooled mining.  The primary Bitcoin project does not need to be polluted with new protocols and RPC commands for mining related purposes.  Bitcoin is not about mining, and developer time should be spent on more important features that matter.

GBT is primarily a pooled mining protocol, and does address the problem you mention perfectly, while also addressing other (arguably) more serious problems (centralized pool control).

I did the Stratum just because of my requirements, not because I wanted to compete with anybody. Let me explain it:

Actually I see the difference between Stratum and GMP as Electrum client and Satoshi client. You can achieve similar results with both of them, but every of such solution has pros and cons.

This analogy doesn't really make sense. Electrum and Satoshi implement things very differently. The only thing StratumMP really does differently from GBT is the protocol/communication syntax.

THere's really no need to have one super-protocol which fits all needs. ... I see GBT as very complex and very hard to implement full specs without breaking anything. Luke will say the oposite, but GBT has simply unnecesary compexity for the environment of pooled mining. You know, many optional features etc. But pooled miners don't need that. Period.

Pooled miners already need those features enough that they've been hacked on top of getwork for a long time. Probably no miner or pool implements full specs for all the getwork extensions, and that's fine for GBT as well. The important thing is that in cases where those features are needed, there is a consistent protocol for it rather than every pool/miner doing the same thing 5 different ways.

[slush]

The problem is more about when people don't work together with others. Everyone was (and is) welcome to contribute to the development of GBT

It's the same like designing the universal car. We should have just one model which will fit small city car, family car, a van and the crane. We definitely can design one machine which will fit all these needs just by mounting some special equipment over it. But we just don't want to, because it will lead to overcomplicated monster with many exceptions in the *real* implementation.

Look around you at this time. Miner software cannot follow even such dumb protocol as current getwork is. I don't want to show you my pool source, because I'm shy for tons of hacks involved because of crappy miner support. This is real-world lesson and we should learn from it. Please don't repeat same design mistakes with the full featured protocol which is ten-times complicated than getwork.

The only thing StratumMP really does differently from GBT is the protocol/communication syntax.

Then please show me exact payload done by GBT and providing exactly the same behaviour like Stratum does in the documentation example. And please don't use submitting whole coinbase. It's impossible to do full validation of coinbase for every share.

Pooled miners already need those features

Which features they *really* need from GBT except?

Probably no miner or pool implements full specs for all the getwork extensions, and that's fine for GBT as well. The important thing is that in cases where those features are needed, there is a consistent protocol for it rather than every pool/miner doing the same thing 5 different ways.

For the god sake, how you can manage miner implementation if every pool operator will have full freedom in implement any of BIP option?

[Graet]

what is wrong with proposing a new way for miners to communicate with pools, it isn't like it is changing a Bitcoin protocol it is a mining change.
there is no single pool software everyone uses, there is no single mining software everyone uses - I for one am interested at looking at options.

The problem is more about when people don't work together with others.

Indeed a lot of people have had issues working with you before this, the history is spread through this forum :/ the alternative proposals were so close together the 2 developers involved decided to work together - not sure of your point here

Everyone was (and is) welcome to contribute to the development of GBT

but no-one is required to? right?,

so why quietly reinvent an incompatible protocol on the side? GBT vs StratumMP is like W3c HTML/CSS vs Microsoft Internet Explorer "quirks" HTML/CSS.

Both of these guys announced in their pool threads that they were working on ideas to cope with upcoming changes to mining hardware and most pools have said they are looking at options, either quietly or loudly....

I'm probably wrong again but looking around Bitcoin there is no monopoly on anything, there are even alternatives to the satoshi client
I think a lot of Bitcoin's growth has come from the different ideas that have been developed by a lot of people both on the main dev team and outside by other interested people, both foss and closed source

I welcome choices

[P_Shep]

I welcome choices

And may the strongest survive

[zvs]

For the god sake

ahh, reminds me of Adam's Apples subtitles, 'for the sake of fuck', one of my favorites

ok, carry on

[optimator]

miners decide what they will mine, and make their own blocks - no more blindly giving up mining control to pool operators

I thought one of the benefits of a pool was to determine the efficiency of including transactions in a block. It is the pool that could decide whether to include or exclude a transaction based on the transaction fees.

Doesn't having the miner take responsibility for the transactions undermine some of the benefit of the pool?

If the miner is building the transaction why doesn't the miner submit a block when his hash is less than the difficulty?

[Luke-Jr]

miners decide what they will mine, and make their own blocks - no more blindly giving up mining control to pool operators

I thought one of the benefits of a pool was to determine the efficiency of including transactions in a block. It is the pool that could decide whether to include or exclude a transaction based on the transaction fees.

Doesn't having the miner take responsibility for the transactions undermine some of the benefit of the pool?

No, the benefit of pooled mining is mostly cooperation/economics: you're saying "if you guys split the block with me if you find it, I'll split it with you if I do". Centralized control over transaction acceptance is one of the flaws of the original pool design, as it compromises Bitcoin's security if any one person has too much influence. On the other hand, one benefit of pools is that there is an entity to arrange a bulk fee contract with for transactions; GBT preserves this ability by allowing pools to flag those transactions as "required", while not weakening the control of the individual miner who can still make an informed decision not to mine for the pool (with centralized mining, the miner is in the dark about what exactly he is working on).

If the miner is building the transaction why doesn't the miner submit a block when his hash is less than the difficulty?

I don't understand this question.

[optimator]

If the miner is building the transaction why doesn't the miner submit a block when his hash is less than the difficulty?

I don't understand this question.

Thanks - I missed the part about flagging transactions. Why can't a miner just submit the block header and the transactions when he finds it? Doesn't the miner have everything he needs to make a block? or am i missing something?

[Luke-Jr]

If the miner is building the transaction why doesn't the miner submit a block when his hash is less than the difficulty?

I don't understand this question.

Thanks - I missed the part about flagging transactions. Why can't a miner just submit the block header and the transactions when he finds it? Doesn't the miner have everything he needs to make a block? or am i missing something?

He can, that's an added optimization. However, that only works for full blocks, not shares.

[optimator]

But then how do the other miners get credit for the completed block? Doesn't this allow dishonest miners to usurp the block from the pool?

[Luke-Jr]

But then how do the other miners get credit for the completed block? Doesn't this allow dishonest miners to usurp the block from the pool?

No, the block rewards can't be changed after mining the block.

[Luke-Jr]

BFGMiner 2.8.0 released with integrated getblocktemplate support.

[Luke-Jr]

Question off-topic in another thread, pertaining to bitcoind GBT support:

luke-jr: Can you give me a reason why it was felt necessary to remove getmemorypool, instead of having it exist alongside GBT?
The existing solution breaks many of the pool packages out there. Backwards compatibility is normally a concern for developers.

The GBT patch for bitcoind was originally fully backward compatible, but was changed to replaced GMP instead per the request of the rest of the bitcoind development team.

[makomk]

I haven't yet read about about these other protocols, but I'm really disappointed to see more protocol proliferation. Getblocktemplate is well established, peer reviewed, and is good for bitcoin's decentralized security. I hope GBT gains whatever features its missing so it can subsume these things.  Having yet another protocol to cope with is a bummer.

The problem with GBT is not so much what features it's missing, from what I can tell. Everything that can be done with Stratum can be done with GBT, it'd just far require more bandwidth and every piece of mining software would need code to parse and serialize Bitcoin transactions and blocks and to implement a whole bunch of different cases. Plus, I suspect most mining software would end up requiring a different subset of the GBT protocol than the one bitcoind supports so it's not like using the same protocol for both would even help solo miners.

[Luke-Jr]

I've mostly finished a more friendly writeup on getblocktemplate on the Bitcoin Wiki, covering (almost?) everything StratumMP's page covered. Take a look and let me know what you think, or how it might be improved! Feel free to edit it directly too, of course