Are pools more efficient?

[Dan The Man]

I'm not super technically into how bitcoin works so maybe I am being dense here. But the way I see it, if there is a brute force problem that everyone is trying to solve. It will be faster if you don't waste time redoing the solutions you have already tried once. So if one computer is working on a brute force problem, they are just randomly trying solutions. But if a group of computers are working on a problem, do they somehow distribute solutions to check or tell each-other which solutions they have tried so nobody is wasting time? Is this how pools work? Do they get more out of their combined computing power than the sum of individual miners?

[sturle]

Finding Bitcoin blocks is a lottery.  You get a certain number of tickets every second, depending on your hardware speed.  Chances of winning this lottery is slim, but the winner gets 50 coins + fees of transactions in the block.

A pool is where many miners join forces and put all tickets they buy into a common pool.  When someone wins, the 50 coins and possibly the fees (some pool owners keep the fees) are shared among all participants based on how much each participant contributed.

A pool will typically find blocks more often than a solo miner, because the combined hardware power of the pool is so much larger than a single miner.  For each miner the result is the same in the long run, given that the pool owner takes no fees.  In practice solo mining is more profitable in the long run because a solo miner is only dependant on his own miner and network, and can even tolerate short network outages.  Fewer dependencies means fewer interruptions in mining.  Pools are down now and then, and even DDOSed quite often.

The Bitcoin network would be more robust if more people mined solo.  When pools are DDOSed, block rate slows down and transactions confirms slower as a result.

[Dan The Man]

That completely did not answer my question at all.

[Meni Rosenfeld]

Pools aren't "more efficient" in the way you mean. The search space of solutions is virtually infinite, so there is no chance of people duplicating work even without any coordination. Basically miners are trying random numbers until they get a hit, whether they're mining pool or solo. The average number of blocks found by a miner is the same in both cases.

What pools do is reduce the variance in a miner's payout. I discuss the basics of this in chapter 1 of Analysis of Bitcoin pooled mining reward systems.

The Bitcoin network would be more robust if more people mined solo.  When pools are DDOSed, block rate slows down and transactions confirms slower as a result.

Not if people properly set up failover pools. And other current problems of pools with respect to network robustness have solutions which will be implemented in the future.

[Dan The Man]

But to find a single solution from an "essentially infinite" space of solutions, you will have to sample a significant proportion of those "essentially infinite" solutions. That means it's not infinite at all. Probabilistically, you would expect to sample as many solutions as there are total possible solutions if you just randomly guess at it. If you keep randomly guessing a number between 1 and 10, you will expect to guess 10 times before you get it. You could possibly guess wrong twenty times in a row.

But if you split up the solutions and systematically eliminate them, then you would only expect to sample  half of them, actually (n+1)/2. If you systematically try to guess numbers between 1 and 10 without repeating the same guess, you will expect to guess 5.5 times before you get it. So if you are eliminating known wrong solutions you cut your work time in half. You will never guess more than 10 times. I was wondering if pools are doing this, or if the Bitcoin protocol somehow incorporated it automatically.  

[Gabi]

I'm not super technically into how bitcoin works so maybe I am being dense here. But the way I see it, if there is a brute force problem that everyone is trying to solve. It will be faster if you don't waste time redoing the solutions you have already tried once. So if one computer is working on a brute force problem, they are just randomly trying solutions. But if a group of computers are working on a problem, do they somehow distribute solutions to check or tell each-other which solutions they have tried so nobody is wasting time? Is this how pools work? Do they get more out of their combined computing power than the sum of individual miners?

No.

As far as i know no one ever redo the solutions already tried by someone else. Pool mining or solo mining you always try different solutions so there is no overlap and wasted time.

Only advantage of pool mining is that maybe they receive faster new blocks/when they find a new block they immediately start crunching for the next one while in solo you would have to first download it...

[Dan The Man]

How do they assign who will try which solutions so that there is no overlap?

[eleuthria]

How do they assign who will try which solutions so that there is no overlap?

Your solutions/shares are specific to a set of headers for the block.  When you're solo mining, the headers include YOUR payout address, so any random nonce you try will produce a different hash, even though that same nonce/extraNonce has been tried by someone else.

In pool mining, the work is distributed so everybody is trying difference nonces without overlap.  That overhead is handled by the pool and its bitcoind.

[Dan The Man]

So wouldn't that imply that 100 computers working in a pool will solve a block faster than 100 computers working for themselves? Because the computers in the pool can all use the pool address and split up the possible solutions. 100 computers working by themselves would essentially be all independently trying to solve a much larger problem because of their unique addresses.

[DeathAndTaxes]

But if you split up the solutions and systematically eliminate them, then you would only expect to sample  half of them, actually (n+1)/2. If you systematically try to guess numbers between 1 and 10 without repeating the same guess, you will expect to guess 5.5 times before you get it. So if you are eliminating known wrong solutions you cut your work time in half. You will never guess more than 10 times. I was wondering if pools are doing this, or if the Bitcoin protocol somehow incorporated it automatically.  

No because there is no defined space or more technically the defined space is constantly changing.

The input for a hash is:
merkle root hash (a hash of all transactions in block including the pool reward address so you can't cheat)
hash of the last block (prevents you from working on block "ahead of time" and also creates the "chain")
timestamp (current time)
nonce (a 32 bit number 2^23)

So a miner will try every nonce value.  That is defined space.  There are roughly 4 billion possibilities however at current difficulty (1.2 million) there is only a 1 in 1.2 million chance that any value in the nonce range will produce a value hash.  1,199,999 out of 1,200,000 complete nonce ranges (4 billion hashes) will produce no solution (on average).

So once that defined space is exausted your miner gets new work.  The simplest way to get new work is simply increment the timestamp and now you are working on an entire new solution space.  Each complete nonce range hashed gives you a 1 / (difficulty) chance of finding the solution.

Periodically the pool will update the other elements in block header.   When a block is found the header changes to reflect new "last block hash".  As transactions are added they change the merkle root hash.  To avoid 2 miners in same pool duplicating work the pool uses a value called "Extra nonce" in the coinbase transaction to ensure each miner is given a unique blockheader to work on at all times.

[Dan The Man]

Thanks, that makes more sense to me. The key thing was that there was no certainty of finding a solution within a certain space.

What is the importance of the timestamp? Would there be any benefit to incrementing the reward address rather than incrementing the timestamp?

[DeathAndTaxes]

Thanks, that makes more sense to me. The key thing was that there was no certainty of finding a solution within a certain space.

Exactly.  The space is 2^256 so it can't be exhaustively searched.

What is the importance of the timestamp? Would there be any benefit to incrementing the reward address rather than incrementing the timestamp?

Timestamp is just used to provide a record of the approximate time when block was solved.  You could change reward address but that would be complicated on the backend.  The coinbase field can hold an "extra nonce" which is just a pseudo-random value with no meaning.  Changing that will result in a different block header so the pool can ensure each miner is working on a different header.

More efficient pools use NTimeRolling which instructs the miner to locally increment the timestamp reusing all the same values.
So a miner will start with a timestamp of x.
try all nonces in nonce range (roughly 4 billion).
update timestamp
try all nonces in nonce range (roughly 4 billion).
...
until the NTimeRolling value expires.  This improves communication efficiency between pool and miner by cutting down on predictable block header changes. 

[Meni Rosenfeld]

But to find a single solution from an "essentially infinite" space of solutions, you will have to sample a significant proportion of those "essentially infinite" solutions. That means it's not infinite at all. Probabilistically, you would expect to sample as many solutions as there are total possible solutions if you just randomly guess at it. If you keep randomly guessing a number between 1 and 10, you will expect to guess 10 times before you get it. You could possibly guess wrong twenty times in a row.

What you're missing is that there are many correct solutions. The block header is 640 bits (some of them can't be easily controlled, but never mind that) so there are 2^640 potential solutions, which is indeed virtually infinite. Of those, there are 2^608/Difficulty correct solutions. So people still have a chance to find a correct solution (1/(2^32*Difficulty) per hash), but no chance to test the same solution someone else did (if they're randomizing properly).

[teukon]

What you're missing is that there are many correct solutions. The block header is 640 bits (some of them can't be easily controlled, but never mind that) so there are 2^640 potential solutions, which is indeed virtually infinite. Of those, there are 2^608/Difficulty correct solutions. So people still have a chance to find a correct solution (1/(2^32*Difficulty) per hash), but no chance to test the same solution someone else did (if they're randomizing properly).

Agreed.  This establishes why the following thought experiment doesn't apply to Bitcoin.

But if you split up the solutions and systematically eliminate them, then you would only expect to sample  half of them, actually (n+1)/2. If you systematically try to guess numbers between 1 and 10 without repeating the same guess, you will expect to guess 5.5 times before you get it. So if you are eliminating known wrong solutions you cut your work time in half.

In this example, keeping track of failed guesses is very helpful because you will expect the number of failed guesses for a round to be much more than the number of different solutions (one in this case).  But with Bitcoin, for every block the number of failed hashes (worldwide) is utterly insignificant compared to the number of potential solutions.  Even if everyone were using the same pool and using random nonces there would be practically no efficiency loss due to repeated work.

[sd]

I'm not super technically into how bitcoin works so maybe I am being dense here. But the way I see it, if there is a brute force problem that everyone is trying to solve. It will be faster if you don't waste time redoing the solutions you have already tried once. So if one computer is working on a brute force problem, they are just randomly trying solutions. But if a group of computers are working on a problem, do they somehow distribute solutions to check or tell each-other which solutions they have tried so nobody is wasting time? Is this how pools work? Do they get more out of their combined computing power than the sum of individual miners?

Pools do not get more out of their combined computing power than the sum of their individual miners. Due to the way BitCoin is designed there is no need for miners to tell each-other which solutions they have tried.

The only reason pools exist is to reduce the variance of payouts.

[Transisto]

This has to be the most illuminating post on how block are generated.

Scenario : Two similar PC, same hardware, same date, same cloned hard-drive, same cloned client, being started at the exact same time.

Would this result in the same work being done twice ?

[DeathAndTaxes]

No.

Three scenarios:

Two computers have different wallets.
The coinbase transaction contains the reward address so with different wallets they will always have different merkle tree roots


Two computers share the same wallet (one copy).
The coinbase also contains an "extra nonce" value which is different for each work request so no duplication.

Two computers have identical copies of the same wallet (very stupid to do)
If the two computers had a duplicated the wallet running simultaneously on both machines at the same time there could be duplicated work in theory.  Even here the amount would be insignificant.  The only duplicated work would be when one of the wallets picked randomly an extra-nonce already picked by the other copy (which it isn't aware of). GIven the small number of extra-nonce changes per block (600 seconds) the amount of work would be a rounding error.

[Meni Rosenfeld]

This has to be the most illuminating post on how block are generated.

Scenario : Two similar PC, same hardware, same date, same cloned hard-drive, same cloned client, being started at the exact same time.

Would this result in the same work being done twice ?

AFAIK the PRNG uses serial numbers on the hardware components, so even if the two computers have the same configuration, they will generate different wallet addresses and extraNonce.

[Transisto]

AFAIK the PRNG uses serial numbers on the hardware components, so even if the two computers have the same configuration, they will generate different wallet addresses and extraNonce.

The addresse is already generated since the HDD is cloned

I ask because It would be rather common (yet stupid at this time) that someone would build 4 similar computer and clone the HDD sector by sector and start them all SOLO mining at once.

[DeathAndTaxes]

AFAIK the PRNG uses serial numbers on the hardware components, so even if the two computers have the same configuration, they will generate different wallet addresses and extraNonce.

The addresse is already generated since the HDD is cloned

I ask because It would be rather common (yet stupid at this time) that someone would build 4 similar computer and clone the HDD sector by sector and start them all SOLO mining at once.

Even IF you used the same wallet on each machine (stupid) rather than point each machine to a single wallet or install a unique wallet on each machine they won't have same RNG seed.  Things like HDD serial #, processor serial #, etc are used in the RNG thus they will generate different extra-nonces.  There may be a tiny amount of duplicated work (probably insignificant) as each instance may use try an extra-nonce already tried by another instance however it would be very small.