Decentralized mining protocol standard: getblocktemplate (ASIC ready!)

[jgarzik]

The problem with not receiving the transactions is that the miner program must have a full bitcoin connection to the network.

No, that is not correct at all.

The pool server may supply sufficient information to modify extranonce in the coinbase txn, and update the merkle root.  Nothing more.

And even that's going over the absolute minimum requirement of simply a block header, with ntime rolling supported.

[1715615019]

1715615019

[1715615019]

1715615019

[1715615019]

1715615019

[1715615019]

1715615019

[-ck]

What a mess. With payloads increasing linearly with transaction count in GBT that makes each getwork equivalent much bigger than plain getwork already is the longer a block takes to solve. As bitcoin becomes more popular and transaction counts increase, gbt becomes less and less efficient. I'd hazard a guess that it may even be less efficient than plain getwork at some stage unless pools start ignoring transactions... there's a disincentive to include them.

Unless something has changed, getblocktemplate should include a mode that skips sending the transactions themselves.  See https://en.bitcoin.it/wiki/BIP_0023

I agree it is unlikely that non-p2pool miners will use the full bore GBT + all transactions, even if it is better for network security.

But don't dismiss GBT based on faulty "full mode only" assumptions.

So what are you anticipating the pools vs miners will be doing with transactions in pooled mining (ignoring p2pool)?

[kano]

The problem with not receiving the transactions is that the miner program must have a full bitcoin connection to the network.

No, that is not correct at all.

The pool server may supply sufficient information to modify extranonce in the coinbase txn, and update the merkle root.  Nothing more.

...

Stratum ...

[jgarzik]

So what are you anticipating the pools vs miners will be doing with transactions in pooled mining (ignoring p2pool)?

Transactions don't need to make it to the miners, just the pool server.  The pool server can handle most of the merkle building bits, and verify the returned work from the miner.

The miner runs through nonce.
If miner needs more work, it runs through ntime range.
If miner needs more work, it increments extranonce.

That will keep your ASIC miner mining through the next century ;p

[-ck]

So what are you anticipating the pools vs miners will be doing with transactions in pooled mining (ignoring p2pool)?

Transactions don't need to make it to the miners, just the pool server.  The pool server can handle most of the merkle building bits, and verify the returned work from the miner.

The miner runs through nonce.
If miner needs more work, it runs through ntime range.
If miner needs more work, it increments extranonce.

That will keep your ASIC miner mining through the next century ;p

The miner can't hash anything without that merkle root though. I didn't see the merkle root being passed regularly to the miners in the sample on the wiki, just how to hash transactions as they come in and generate a merkle root. Did I miss where the merkle root is passed to the miner?

In which case, we're back to my original comment... the miners need all the transactions every so often or they're doing bitcoin a disservice.

[firefop]

The problem with not receiving the transactions is that the miner program must have a full bitcoin connection to the network.

Suddenly mining includes bitcoin network transaction and block processing.

... and anyone with half an inkling of understanding of what that entails can see the issues of having to deal with all the complications that entails and then also the importance of the bitcoin network performance of the miner ... which before this point was zero.

Complication like "staying online" is complicated?

This is so much of a non-issue, it's just silly.

Miners SHOULD be running a node even if they aren't mining on it. Depending on the pool for information that can be pulled from bitcoind isn't smart. Having to trust the pool for transaction and block processing isn't smart either. Those add unneeded complexity to an already overcomplicated system.

In fact I think pools in general come at mining form the wrong direction. You don't need to develop a way to hand out work where you send it to the miner and they report back with the completed work - what you ought to be doing is making work assignment system where each miner generates, validates and submits the work for the pool - while transmitting just the nonce (or even better a nonce range completed) back to the pool.

Every miner would include a msg from them to the pool stating they mined the block. And now you've got decentralized and much lower bandwidth. And you aren't trusting the pool for what it may or may not be doing with your hash rate.

And this is just off the top of my head --- start thinking about 'how it should be' instead of 'how it is right now' --- and start developing in that direction also.

[-ck]

Sorry but a lot of miners mine without their mining rigs being connected to a bitcoind instance. Yes there may be one somewhere in their network, or possibly not at all and they only fire up their wallet when they want to on a separate network. What you think is ideal is asking people to do more than they currently do. They currently use pools and they trust them already. The community is very vigilant with respect to pools misbehaving. The importance of the miner doing it all themselves is grossly exaggerated by proponents of GBT. Pools will continue to do as much as they already do. It's unrealistic to expect otherwise.

[kano]

The problem with not receiving the transactions is that the miner program must have a full bitcoin connection to the network.

Suddenly mining includes bitcoin network transaction and block processing.

... and anyone with half an inkling of understanding of what that entails can see the issues of having to deal with all the complications that entails and then also the importance of the bitcoin network performance of the miner ... which before this point was zero.

Complication like "staying online" is complicated?

This is so much of a non-issue, it's just silly.

Miners SHOULD be running a node even if they aren't mining on it. Depending on the pool for information that can be pulled from bitcoind isn't smart. Having to trust the pool for transaction and block processing isn't smart either. Those add unneeded complexity to an already overcomplicated system.

In fact I think pools in general come at mining form the wrong direction. You don't need to develop a way to hand out work where you send it to the miner and they report back with the completed work - what you ought to be doing is making work assignment system where each miner generates, validates and submits the work for the pool - while transmitting just the nonce (or even better a nonce range completed) back to the pool.

Every miner would include a msg from them to the pool stating they mined the block. And now you've got decentralized and much lower bandwidth. And you aren't trusting the pool for what it may or may not be doing with your hash rate.

And this is just off the top of my head --- start thinking about 'how it should be' instead of 'how it is right now' --- and start developing in that direction also.

A few points here:

The most important is yet again someone telling everyone how they should be mining.
"how it should be"
I'm glad you feel that is how it should be.

"Miners SHOULD be running a node"
Again, glad you feel that's how it should be.

Lucky for everyone else, they have freedom of choice and are not under your control.

Even luckier for you - you can do this already for yourself today - without trying to force everyone else to do it.

--

Meanwhile, your first statement is the only one in this conversation that is "just silly"

Ignoring the blatantly obvious issues and pretending to simplify to a stupid statement of "staying online"

bitcoind cannot be used except for solo mining (or p2pool) - as I mentioned, there needs work done to support a combination of pool GBT mining and bitcoind to supply the transactions.

i.e. miners implementing bitcoind yet again is just a problem waiting to happen.
... and that adds complexity, it doesn't reduce it at all.
Do you actually understand what txn and block processing entails?

Then of course there's the fact that you need to run a high performance bitcoind to ensure maximum work success - yeah everyone has one of those ... you think that may be what most pools spend a lot of effort doing?

As for bandwidth, if you really want to reduce it, how is adding a bitcoind into the equation going to do that?

Go stratum - problem solved

[jgarzik]

1) In general, everybody should be running a full node, just to support the network.  It's not a requirement, obviously, but we need as many full nodes as possible to keep the network resistant to Sybil attacks and such.

2) p2pool is the preferred solution for the community, with centralized pools being second best.  Centralized pools make it much easier to be a miner, at the cost of centralizing network power in the hands of the pool operator.

[-ck]

Sigh. In that case, with p2pool being 400 GH of a 20TH network, then GBT is only going to benefit 2% of the miners. It's pretty obvious by now you can't force people to do what you think is best. If anything, GBT with non-p2pool pooled mining will make it more likely that pools will not want to pass on lots of transactions and will be worse than getwork.

How about adding an extension to GBT that has some kind of mode that sends a "suggested merkle root" for pooled mining?

[firefop]

How about adding an extension to GBT that has some kind of mode that sends a "suggested merkle root" for pooled mining?

That's an idea... have to consult with Luke on that.

I also wanted to point out that if we (or someone anyway) doesn't build a scalable solution per pooled mining with an integrated node... then nobody will be able to include it in the POS machines that will eventually be developed, and missing the chance for millions of additional nodes seems sad to me.

[slush]

How about adding an extension to GBT that has some kind of mode that sends a "suggested merkle root" for pooled mining?

How about adding an extension to Stratum that has some kind of mode that sends "source transactions" for given job?

In that way, paranoid users will have a chance to introspect jobs generated by Stratum pools and common users which don't care will have optimized mining protocol.

And yes, I'm working on it already. Important thing in this extension is that list of transactions can be broadcasted "lazily" (=few miliseconds after job updates), because that list is not important for mining itself. So it won't hurt efficiency of mining, but still give a tool for online block template monitoring.

[-ck]

slush, not a bad idea but I'm actually not questioning stratum here since I obviously think it's the better protocol for pooled mining as I've already implemented it in cgminer. It's just that this is the GBT thread, I have been asked to include GBT support as well and I'm not really happy with the protocol as is.

[Luke-Jr]

How about adding an extension to GBT that has some kind of mode that sends a "suggested merkle root" for pooled mining?

That's an idea... have to consult with Luke on that.

Not really, Con or anyone else is welcome to write a draft BIP to enable centralized mining over GBT, and open the topic to discussion. But I think it's possible to adopt Stratum's benefits without losing the decentralization benefits - though I don't think it is a problem that needs to be addressed immediately, nor do I have time to do it myself right now. BIP 22 and 23 were developed by collaboration of the community, and while I may be the primary organizer, there is no reason someone else couldn't take it forward more on their own initiative.

I also wanted to point out that if we (or someone anyway) doesn't build a scalable solution per pooled mining with an integrated node... then nobody will be able to include it in the POS machines that will eventually be developed, and missing the chance for millions of additional nodes seems sad to me.

GBT is pretty scalable for now.

[-ck]

No, it's quite alright. I have enough to do myself as well and I'm not in the business of helping pool designs. I'll implement it as is and let the mining community determine which pools and with what protocols they prefer.

[kano]

How about adding an extension to GBT that has some kind of mode that sends a "suggested merkle root" for pooled mining?

That's an idea... have to consult with Luke on that.

Not really, Con or anyone else is welcome to write a draft BIP to enable centralized mining over GBT, and open the topic to discussion. But I think it's possible to adopt Stratum's benefits without losing the decentralization benefits - though I don't think it is a problem that needs to be addressed immediately, nor do I have time to do it myself right now. BIP 22 and 23 were developed by collaboration of the community, and while I may be the primary organizer, there is no reason someone else couldn't take it forward more on their own initiative.

Um - yes you keep making this 'community' comment - but the reality is the so called 'community' is pretty much you and not much else.

I also wanted to point out that if we (or someone anyway) doesn't build a scalable solution per pooled mining with an integrated node... then nobody will be able to include it in the POS machines that will eventually be developed, and missing the chance for millions of additional nodes seems sad to me.

GBT is pretty scalable for now.

Well, unfortunately 'pretty scalable' seems to be 'worse than getwork' and that should already be a MAJOR flag for anyone to see that work NEEDS to be done, if anyone wants to consider 'choice'
Between using Stratum and GBT at the moment that clearly says to use Stratum

[Luke-Jr]

I do intend to improve BFGMiner's GBT implementation such that it transparently combines pooled and solo mining, at least.

[Sitarow]

I do intend to improve BFGMiner's GBT implementation such that it transparently combines pooled and solo mining, at least.

That is planned in release BFGMiner's 3.0?

[kano]

I do intend to improve BFGMiner's GBT implementation such that it transparently combines pooled and solo mining, at least.

That's cool, but solo mining should not be an option - it should be mandatory. Else the whole project is just another house of cards.

See now you know what you should do.
Go create an alt-coin that tells everyone what they must be doing and see how that fares.
If you are right, then that alt-coin will outlast BTC and you'll be famous ...................

[slush]

...I think not... ...I think we should not gamble... ...Bitcoin is extreme fragile... ...demand from users... ...I know it can be done... ...it should be mandatory...

Honestly, as I'm reading your posts, I think that you should create separate blockchain with those rules hardcoded. Feel free to name it NaziCoin.