How does pool mining and mining work under the hood?

[riekinho]

I read almost every answer on Bitcoin Stack Exchange and bitcoin wiki but there are still things that trouble.
I would really love to understand it.

I have no IT background and therefore I'm unable to read code and come up with answers to my questions.

Questions are:
1. One has no way of predicting what a hash function will spit out, that's why you add nonce to transactions and voila you are the winner if you get a hash below certain number.
Now the question is: Is it possible for every nonce coupled with transactions to produce a number greater then required? What happens then?

2. It's often cited that if you don't include transaction fee the miners won't include the transaction for a long time and if you do include the fee it will get 1 confirmation every 10 minutes. How do miners "know" that transaction fee is included if they work on hashes of transactions?

3. How does pool mining work?
I understand that each individual miner is handed a piece of the puzzle to solve.
Each pool miner isn't a node in the network.
Pool software is a node in the network

So how does pool software "hand out" tasks?
Does it say to individual miner in the pool: here, you try these transactions with nonces 1-1000000 and to other miner you try nonces 1000001-2000000?

Also if miner finds a nonce that solves a block what is to prevent that pool miner saying: "Screw the pool I'm having 25 BTC for myself and I'll broadcast the nonce and the hash?

How does pool handle that problem, after all the pool cannot simply make miners do some useless proof-of-work. The pool being a node has to come up with solution to be rewarded?

4. When you are mining you are combining transactions + nonce to create the winning hash, but transactions are constantly flowing in. How is that handled?

Thanks!
I also apologize if the answers are readily available, I couldn't find satisfactory answers.

[wheatstone]

1. One has no way of predicting what a hash function will spit out, that's why you add nonce to transactions and voila you are the winner if you get a hash below certain number.
Now the question is: Is it possible for every nonce coupled with transactions to produce a number greater then required? What happens then?

The nonce isn't the only thing that can be varied (e.g. extraNonce), so the practical answer is probably "not possible", but I'm not aware of any theoretical proof.

2. It's often cited that if you don't include transaction fee the miners won't include the transaction for a long time and if you do include the fee it will get 1 confirmation every 10 minutes. How do miners "know" that transaction fee is included if they work on hashes of transactions?

Solominers and pools can see the transactions and select the ones they wish to include. They are completely free to chose which and how many to include.

3. How does pool mining work?
I understand that each individual miner is handed a piece of the puzzle to solve.

That's not an accurate way to look at it. There is no "part of the puzzle". Only "the puzzle". The name of the game is to get a hash that starts with a some number of zeros (higher difficulty -> more zeros). In a pool, each participant sends in answers to the puzzle that are of some much easier difficulty. This allows the pool to gauge how much work they are doing and give out rewards in accordance with that.

[GoldenWings91]

I don't have the knowledge to answer 1 but I can answer the others.

2. It's often cited that if you don't include transaction fee the miners won't include the transaction for a long time and if you do include the fee it will get 1 confirmation every 10 minutes. How do miners "know" that transaction fee is included if they work on hashes of transactions?

A miner will get the entire transaction then check that its script is valid and follows all protocol rules. If its priority is high enough the hash of this transaction will be added to the list of transactions to be mined aka build merkle root.

3. How does pool mining work?
I understand that each individual miner is handed a piece of the puzzle to solve.
Each pool miner isn't a node in the network.
Pool software is a node in the network

There are many ways to do pool mining I'll give an intro to the stratum protocol as it's the most used.
With the stratum protocol each worker is given an ExtraNonce1, which is essentially the workers id each worker will have a different ExtraNonce1, and the size of ExtraNonce2, the number of bytes the worker can iterate. Example ExtraNonce1 "08000002" and ExtraNonce2 4. Each worker is also given enough information to build a unique block header: hash of previous block, coinbase1 script, coinbase2 script, list of merkle branches, block version, bits, and time.
Coinbase1 + ExtraNonce1 + ExtraNonce2 + coinbase2 = the coinbase transaction, this is the transaction where the miners reward goes to. The workers receive coinbase1 and coinbase2 they can not change it.
Once the the coinbase transaction is created it is combined with the merkle branches to create a merkle root. At this point the worker has all the components to create a header which can be worked on by incrementing the standard nonce.

02000000         Version
5655640000.... previous block
354685498b.... merkle root
50ff00ea           time
f2b9441a          bits
00000001         nonce ( this gets incremented )

Once the nonce runs out the worker increments ExtraNonce2 and creates a new coinbase transaction hash which leads to a new merkle root which means the nonce can be started at 0 again.

4. When you are mining you are combining transactions + nonce to create the winning hash, but transactions are constantly flowing in. How is that handled?

Most pools will add new transactions to be worked on every 30 seconds.

[riekinho]

That's not an accurate way to look at it. There is no "part of the puzzle". Only "the puzzle". The name of the game is to get a hash that starts with a some number of zeros (higher difficulty -> more zeros). In a pool, each participant sends in answers to the puzzle that are of some much easier difficulty. This allows the pool to gauge how much work they are doing and give out rewards in accordance with that.

Thanks wheatstone!

This is exactly what I understood about mining so far.
There is no part of the puzzle, only THE puzzle. To pick up the reward you have to come up with low enough hash.

I was wondering how does a pool split this computationally difficult task into smaller tasks while at the same time preventing any individual miner to claim the reward if he happens to be the one hitting the correct solution?

There are many ways to do pool mining I'll give an intro to the stratum protocol as it's the most used.
With the stratum protocol each worker is given an ExtraNonce1, which is essentially the workers id each worker will have a different ExtraNonce1, and the size of ExtraNonce2, the number of bytes the worker can iterate. Example ExtraNonce1 "08000002" and ExtraNonce2 4. Each worker is also given enough information to build a unique block header: hash of previous block, coinbase1 script, coinbase2 script, list of merkle branches, block version, bits, and time.
Coinbase1 + ExtraNonce1 + ExtraNonce2 + coinbase2 = the coinbase transaction, this is the transaction where the miners reward goes to. The workers receive coinbase1 and coinbase2 they can not change it.
Once the the coinbase transaction is created it is combined with the merkle branches to create a merkle root. At this point the worker has all the components to create a header which can be worked on by incrementing the standard nonce.

Thanks GoldenWings91

I need some time to digest this, some of the terms are unfamiliar to me so I'll get back to reading some more.

Solominers and pools can see the transactions and select the ones they wish to include. They are completely free to chose which and how many to include.

Then what incentive do miners have to include zero-fee transactions?
Can it happen that such transactions float forever without being included? Are they dropped after some time?

[dbbit]

I was wondering how does a pool split this computationally difficult task into smaller tasks while at the same time preventing any individual miner to claim the reward if he happens to be the one hitting the correct solution?

The 'puzzle' that is solved contains, among other things, the Coinbase transaction that will assign the reward to the pool's address.

It the miner happens to solve the block himself, and instead of submitting it to the pool, he submits it somewhere else, it wouldn't matter since the pool is going to get the reward regardless.

And if the miner goes and modify the Coinbase transaction to assign the reward to himself, he will have to computed a new block hash... which, if you could do that, you should just always do that in the first place and skip the pool .

And if he ALWAYS changes the Coinbase transaction to assign the reward to himself "just incase", the pool will reject all his shares, and he wouldn't get any reward until he solves the block. Then might as well solo mine.


It's pretty ingenious if you think about it. Was it Slush who invented it? Either way, hats of to whoever did.

[wheatstone]

I was wondering how does a pool split this computationally difficult task into smaller tasks while at the same time preventing any individual miner to claim the reward if he happens to be the one hitting the correct solution?

dbbit covered the latter part and we both alluded to the former. However, let me try to be a bit clearer:

The task of calculating the hash is not difficult (it is so easy miners do hundreds of millions, even billions, of these every second).

What the pool needs is proof that a participating miner is trying and how hard. It's not enough to simply look at hashes that actually produce a valid block, since these will be too few and far between. Instead, the pool sets a much lower difficulty and miners submit all hashes they produce that match this lower difficulty (i.e. the hash value can have fewer zeros).

From the number of contributed low-difficulty hashes, the pool has proof positive that the miner is trying to find a valid hash and a good estimate of how much work the miner is doing.

Depending on how one looks at it, the argument could be made that miners who don't find a hash that is valid for the high difficulty, don't actually contribute anything. None of the work they perform is used in any way.

However, that's the nature of mining pools. Instead of playing the lottery (solo-mining), miners share the rewards equally (well, according to workload) regardless of who finds the "solution".

To take an analogy, imagine a group of people searching a field for a missing diamond. You can either reward the person who finds the diamond, or you can reward everyone who participates in the search with a share of the diamond according to how hard they are looking (easier to quantify when it's a miner).

[primer10]

I don't have the knowledge to answer 1 but I can answer the others.

2. It's often cited that if you don't include transaction fee the miners won't include the transaction for a long time and if you do include the fee it will get 1 confirmation every 10 minutes. How do miners "know" that transaction fee is included if they work on hashes of transactions?

A miner will get the entire transaction then check that its script is valid and follows all protocol rules. If its priority is high enough the hash of this transaction will be added to the list of transactions to be mined aka build merkle root.

3. How does pool mining work?
I understand that each individual miner is handed a piece of the puzzle to solve.
Each pool miner isn't a node in the network.
Pool software is a node in the network

There are many ways to do pool mining I'll give an intro to the stratum protocol as it's the most used.
With the stratum protocol each worker is given an ExtraNonce1, which is essentially the workers id each worker will have a different ExtraNonce1, and the size of ExtraNonce2, the number of bytes the worker can iterate. Example ExtraNonce1 "08000002" and ExtraNonce2 4. Each worker is also given enough information to build a unique block header: hash of previous block, coinbase1 script, coinbase2 script, list of merkle branches, block version, bits, and time.
Coinbase1 + ExtraNonce1 + ExtraNonce2 + coinbase2 = the coinbase transaction, this is the transaction where the miners reward goes to. The workers receive coinbase1 and coinbase2 they can not change it.
Once the the coinbase transaction is created it is combined with the merkle branches to create a merkle root. At this point the worker has all the components to create a header which can be worked on by incrementing the standard nonce.

02000000         Version
5655640000.... previous block
354685498b.... merkle root
50ff00ea           time
f2b9441a          bits
00000001         nonce ( this gets incremented )

Once the nonce runs out the worker increments ExtraNonce2 and creates a new coinbase transaction hash which leads to a new merkle root which means the nonce can be started at 0 again.

4. When you are mining you are combining transactions + nonce to create the winning hash, but transactions are constantly flowing in. How is that handled?

Most pools will add new transactions to be worked on every 30 seconds.

How about within the miner program running on machine with multiple mining devices (ASIC, GPU, etc), does the work gets divided down? I would suppose so right?

[GoldenWings91]

How about within the miner program running on machine with multiple mining devices (ASIC, GPU, etc), does the work gets divided down? I would suppose so right?

Yes. Most pools use an extraNonce2 size of 4 bytes which gives each miner 2^32 different block headers that can be created from a single workload. The mining program can than split all this work up to all the different devices connected to it.

[Polemos]

Quote
2. It's often cited that if you don't include transaction fee the miners won't include the transaction for a long time and if you do include the fee it will get 1 confirmation every 10 minutes. How do miners "know" that transaction fee is included if they work on hashes of transactions?

Solominers and pools can see the transactions and select the ones they wish to include. They are completely free to chose which and how many to include.

How about if 2 or more miners choose the same transactions?

[primer10]

How about within the miner program running on machine with multiple mining devices (ASIC, GPU, etc), does the work gets divided down? I would suppose so right?

Yes. Most pools use an extraNonce2 size of 4 bytes which gives each miner 2^32 different block headers that can be created from a single workload. The mining program can than split all this work up to all the different devices connected to it.

Thanks GoldenWings91!

I was posting this related scenario on other forum:

1 remote wallet connected by 1 machine with 2 GPUs doing solo mining

vs

1 remote wallet connected by 2 machines with 1 GPU each doing solo ming.

So the first case I would have higher chance of finding block compared to the 2nd case since wallet does not function like a pool where work is dished out ... based on your reply, looks like I am right?

[gmaxwell]

That isn't right. Mining is memoryless, there is no progress made— analogous to throwing fair dice, if you go 100 rolls without rolling a 1 your next roll is still no more or less likely than your first roll.

With respect to OP's questions on fees and selecting transactions. The important thing is that as far as the protocol is concerned someone who is just working on hashes isn't a miner any more than AMD is— someone who is just working on hashes is just selling CPU time to the real miner elsewhere (the pool).  Only P2Pool users, solo miners, and mining pools themselves are actual miners from the perspective of the protocol.