How to eliminate Large mining pools

[Sherlock.Holmes]

Mining pools collapsed the key value proposition of Bitcoin, namely, its decentralization. In turn, there has been considerable criticism and backlash. Many people in the community, including us, have called for technical measures to elemate large mining pools.

In this post, we present a specific technical fix, called Two-Phase-Proof-of-Work (2PoW), to elemate large mining pools.

In the Bitcoin network, as the computing power of the mining pool continues to increase, individual nodes can no longer independently participate in mining. Individual miners can only participate in mining by joining the mining pool, which completely deviates from "one-CPU-one-vote" vision. In essence, a mining pool is only equivalent to a node, monopoly mining is very terrible. In addition to unfair incentive, monopoly mining is possible for mining pools with greater computing power to be actively or passively carryed out attacks on the network, resulting in the collapse of the entire system.

We believe that in the Bitcoin network, the formation of the mining pool is inevitable, because in the proof-of-work mechanism, the miner can outsource part of his work to other miners. The cooperative mining between miners is obviously superior to non-cooperative mining. The traditional view is that the formation of the mining pool is due to the ASIC. We believe that this is not the key reason. Many existing projects eventually formed the mining pool after adopting the anti-ASIC strategy. The strategy only limits the growth speed of the computing power of a single node, but the advantages of cooperative mining are still obvious.

In the traditional proof-of-work mechanism, we believe that cooperative mining is inevitable. This is because there are no puzzles can’t be outsourced.

We propose a two-phase-proof-of-work mechanism to solve the problem of cooperative mining. We introduce a concept of "luck" in each round of mining, that is, miners rely on luck to mine. We split each round of mining tasks into two phases of proof of work, the first phase is a lightweight task to complete the calculation of the luck , and then calculate the difficulty of the second phase of the mining task according to the luck, The higher the luck, the smaller the difficulty, the smaller the luck, the greater the difficulty. The second phase is to complete the corresponding work according to the corresponding difficulty. We can prove that under this two-phase-proof-of-work mechanism, the expected output of miners' cooperative mining is lower than that of their non-cooperative mining. This is the key to achieving "one-CPU-one-vote". Whether each node belongs to the same miner, it will choose to mine independently.


For more details about 2POW, refer to https://bitcointalk.org/index.php?topic=5254068.0

Welcome to the Luck:
     https://lucknet.club
     Whitepaper: https://lucknet.club/doc/luck.pdf

Some other media links:
     Discord: https://discord.gg/TN6XhcD
     Telegram: https://t.me/luck_cryptocurrency

[aliashraf]

OP,
I checked your paper, good job  

Basically, what you are proposing is about using a two-phase problem instead of the conventional hashcash problem used in bitcoin and its clones which you prove mathematically as not being pool-friendly, and this way you are looking for a pool-resistant PoW algorithm.

Although it is very inspiring to see people working on such a crucial problem, centralization of mining  because of pools, I afraid it is a bit more complicated than it seems and you need to reconsider the whole issue of pools and centralization of mining from a very different perspective:

First of all, we need to understand that pools are not a first-degree, algorithm dependent problem for PoW. No matter what your algorithm of choice might be, as long as you are using the same idea of conventional/traditional PoW you are doomed. It is because of the Bernoulli distribution of luck for a binary, everything/nothing rewarding function.
This phenomenon in bitcoin is called mining variance and pushes out miners with less than like 10^-4 share out of the business because such miners (nodes in your terminology) won't get rewarded frequently enough to survive and compensate for their costs. I've thoroughly described this problem here.

Obviously, with bitcoin's winner-take-all approach to PoW, which is copied by ALL other PoW coins, there is a natural pressure towards pools and given your approach to this problem is effective, it will not be considered helpful with centralization and on the contrary, it worsens the situation by reducing the total number of independent miners to less than 100 players or so.

Alternatively, an effective approach to this problem has been proposed by me previously and I'm working on it right now. The point is clear:
Instead of finding a way to make it harder for pools to operate find a way to have pools, decentralized, as a part of the protocol.

[NotATether]

This proposal requires a hardfork because it is changing the definition of mining difficulty. There are a number of problems with this proposal that stem from the absence of damage control if some chaotic event happened within the network, such as the forking probability you mentioned in the paper, no matter how low it is, the determination of the luck function that governs the second PoW. The network shouldn't have to do a USAF or MASF to fix problems in hardforks that could have been avoided if the proposal was revised.

7 Unpredictability of the number of nodes

For the proof-of-work mechanism, the computing power of the entire network is determined by the difficulty, which is a publicly determined variable. For the two-phase-proof-of-work mechanism, the change in the difficulty is affected by two factors which are the number of participating nodes and the distribution of computing power. Since the distribution of computing power is completely unpredictable, the number of participating nodes is a parameter that cannot be computed.

This is one of the flaws I'm talking about. If you can't determine the computing power, then how will any miner know how difficult it will be to mine, without using the difficulty value that you are proposing to replace? Computing power tells us what is the total hashrate in the bitcoin network, which secures it from miners trying to subvert it and creating a longer chain by forcing them to mine at a collective hashrate greater than that. In your method, which uses a function of luck (or think of it as a function of a "lottery number"), the function itself generating its own minima and maxima at arbitrary luck values, it is impossible to select a function to determine the second PoW difficulty that makes high enough difficulty for all luck values, because as you mentioned, an attacker can carry out a sybil attack against the first PoW to select the luck with the least difficulty, if he knows what the function of luck is. And everyone will know what that function is and determine its mathematical properties because it would have to be put in the open-source bitcoin core codebase.

9 Ideal incentive scheme

Both the Bitcoin-like proof-of-work mechanism and the two-phase-proof-of-work mechanism proposed in this paper are competitive consensus. The feature of the competitive consensus is that there is only one winner per round. The higher fluctuations in probability make small miners dare not participate in mining, the rest are large mining teams.

An ideal incentive plan should be to reward as many honest participants as possible in small amounts. This is similar to the lottery. A lottery is not only a grand prize for each round, but there are many other prizes such as the first prize, second prize,third prize..

We can't implement this. The block reward has already been fixed in the bitcoin network and it's impossible to introduce other rewards into it. Even if this were to be implemented, it runs the risk of at worst generating more than the total supply of 21 million coins, and at best sharply dividing the current mining reward into "first prize", "second prize" etc. which would be unacceptable to the miners currently mining.


I don't think pools are a threat to bitcoin's viability as a currency because miners connect to these pools using mining software, usually the open source cgminer, and if any pool wanted to get an unfair advantage in the network then they would need to modify cgminer code to do it which won't be allowed by the maintainers.

Also a clarification: this equation



is equivalent to Sum_{l in L} f(l)[(1/(b-a)^m]. It's still much less than E₁ but I thought I would point out this simplification.

P.S. Instead of One-CPU-one-vote vision, I think a better name for that would be One-ASIC-one-vote vision for obvious reasons.

[Sherlock.Holmes]

This is one of the flaws I'm talking about. If you can't determine the computing power, then how will any miner know how difficult it will be to mine, without using the difficulty value that you are proposing to replace? Computing power tells us what is the total hashrate in the bitcoin network, which secures it from miners trying to subvert it and creating a longer chain by forcing them to mine at a collective hashrate greater than that. In your method, which uses a function of luck (or think of it as a function of a "lottery number"), the function itself generating its own minima and maxima at arbitrary luck values, it is impossible to select a function to determine the second PoW difficulty that makes high enough difficulty for all luck values, because as you mentioned, an attacker can carry out a sybil attack against the first PoW to select the luck with the least difficulty, if he knows what the function of luck is. And everyone will know what that function is and determine its mathematical properties because it would have to be put in the open-source bitcoin core codebase.

Because the calculation of luck requires a lot of work, when the number of nodes in the entire network is large, the work of calculating luck will be much larger than that of the node with the best luck in the second phase of calculation. This also effectively prevents sybil attack in the first phase.

[Sherlock.Holmes]

While i don't understand much technical details and game theory of the paper, it doesn't solve the fact there are some wealthy individuals or company who have big mining farms (who usually have their own mining pool) and obviously have big contribution of network hashrate.

In this project, we obviously broke the monopoly of the mining pool's. There will be more independent miners in the network that have no connection with each other, rather than only a few scattered mining pools.

[pooya87]

Many existing projects eventually formed the mining pool after adopting the anti-ASIC strategy.

there is no such thing as "anti-ASIC", ASIC-resistant or ... it was all a sham, a vaporware that was used to advertise copycat altcoins trying to sell themselves as "better than bitcoin" while in fact they were adding more vulnerability to the protocol they were changing.

lets look at the problem you want to solve. the problem is not about "pooling the work" or even about "mining pools being big". it is all about the fact that an individual miner that connects to these pools are no longer participating in decision making. decisions such as which transactions to be included in the block they mine, which chain to follow, which proposal to vote for, or even which coin they are mining!...
pooling the "work" in proof of work is a necessity. it helps keep the small miners, and lets them continue contributing to the total hashrate while making money for whatever amount of work they do.

to solve the real problem without adding unnecessary complications to the protocol that could end up becoming the same vaporware as ASIC-resistance or worse add new vulnerabilities, we can simply change the non-protocol stuff that is happening in the pools. for that, the miners have to change also. they have to start running their own node, and demand the block template that they are about to mine from the pool and verify it using their own node to see if the block is actually what they want, instead of just getting the header and hashing it blindly not even knowing if it is a bitcoin block or some other altcoin with similar PoW or even random bytes!
then they can take their "one CPU one vote" power back without even changing the bitcoin protocol.

one of the related proposals: https://medium.com/hackernoon/betterhash-decentralizing-bitcoin-mining-with-new-hashing-protocols-291de178e3e0

[DaveF]

The other thing that has been discussed is that we do.....nothing.
A few smaller pools closed / closing now after the halving. In 4 years it's going to happen again. But, there will probably be a few of the larger ones that go away too.
3.125BTC + fees might will just not leave enough money in it unless we are at $50K+USD per BTC.
In 2028 unless something else changes when we hit 1.6BTC per block + fees if BTC is not worth a ton you can bet difficulty is going to plummet, and the small PPS pools where people come to play are going to pop up all over the place.

Just my view.

If BTC goes wayyyy up in price then you can ignore all of the above.
If it falls to sub $3K and stays there then it's going to happen sooner.

-Dave

[aliashraf]

Many existing projects eventually formed the mining pool after adopting the anti-ASIC strategy.

there is no such thing as "anti-ASIC", ASIC-resistant or ... it was all a sham, a vaporware that was used to advertise copycat altcoins trying to sell themselves as "better than bitcoin" while in fact they were adding more vulnerability to the protocol they were changing.

lets look at the problem you want to solve. the problem is not about "pooling the work" or even about "mining pools being big". it is all about the fact that an individual miner that connects to these pools are no longer participating in decision making. decisions such as which transactions to be included in the block they mine, which chain to follow, which proposal to vote for, or even which coin they are mining!...
pooling the "work" in proof of work is a necessity. it helps keep the small miners, and lets them continue contributing to the total hashrate while making money for whatever amount of work they do.

to solve the real problem without adding unnecessary complications to the protocol that could end up becoming the same vaporware as ASIC-resistance or worse add new vulnerabilities, we can simply change the non-protocol stuff that is happening in the pools. for that, the miners have to change also. they have to start running their own node, and demand the block template that they are about to mine from the pool and verify it using their own node to see if the block is actually what they want, instead of just getting the header and hashing it blindly not even knowing if it is a bitcoin block or some other altcoin with similar PoW or even random bytes!
then they can take their "one CPU one vote" power back without even changing the bitcoin protocol.

one of the related proposals: https://medium.com/hackernoon/betterhash-decentralizing-bitcoin-mining-with-new-hashing-protocols-291de178e3e0

Honestly, didn't expect it from a decent, independent user like you, Pooya

Firstly, ASIC-resistance is an open problem in cryptocurrency and it should be considered highly inappropriate to accuse people who work on it as being a bunch of scammers. I'm already aware of your anti-Altcoin tendencies but I didn't expect you to make false claims like that.

Secondly, your block-template based idea which Matt's Betterhash is a realization of it won't succeed ever because pool operators are not able to verify every single share submitted to them without reducing the number of shares asking for blocks with higher merit and ending up with a high variance that is an existential contradiction.
It is just oversimplifying a critical and inherent flaw built to bitcoin which requires both deep knowledge and true courage to address.

[pooya87]

Honestly, didn't expect it from a decent, independent user like you, Pooya

To Err Is Human!

Firstly, ASIC-resistance is an open problem in cryptocurrency and it should be considered highly inappropriate to accuse people who work on it as being a bunch of scammers. I'm already aware of your anti-Altcoin tendencies but I didn't expect you to make false claims like that.

i go with what i see. and what i have seen so far is that dozens of altcoins have been created, many with a lot of hype starting from LTC and Monero down to ETH, ZCash and finally Grin. there are 3 groups that i see:
1) 100% fake like LTC and ETH where all they did was making their algorithm "bitcoin-ASIC" resistant not "ASIC" resistant. they use a simple but different mining algorithm (scrypt, keccak respectively) that a different ASIC can be built for if the incentive is there which has happened already.
2) the changing algorithms which i think is what Monero did where they change the algo to prevent ASICs from being manufactured, and they have to keep changing it every now and then! that means a hard fork every time.
3) finally the expensive algorithms where they try to increase the expense, add bottlenecks, ... to make it hard for ASICs to be efficient. but eventually someone finds a workaround. which is where Grin is.

i agree that shouldn't have talk about all of them with such negativity but looking at the whole picture that is what i see.

Secondly, your block-template based idea which Matt's Betterhash is a realization of it won't succeed ever because pool operators are not able to verify every single share submitted to them without reducing the number of shares asking for blocks with higher merit and ending up with a high variance that is an existential contradiction.
It is just oversimplifying a critical and inherent flaw built to bitcoin which requires both deep knowledge and true courage to address.

my point is that if we want to address the problem i outlined, we should take the route that is similar to the betterhash proposal instead of a route that is similar to what OP proposes.

[Sherlock.Holmes]

my point is that if we want to address the problem i outlined, we should take the route that is similar to the betterhash proposal instead of a route that is similar to what OP proposes.

The formation of the mining pool is a result of interest-driven, and the pool protocol such as stratum just designed a allocation mechanism to establish a stable relationship between the pool manager and the miners. If the POW protocol itself is not changed, any new pool protocol only changes the rules for allocation, and there are no substantial changes to the security of the network.

[aliashraf]

Firstly, ASIC-resistance is an open problem in cryptocurrency and it should be considered highly inappropriate to accuse people who work on it as being a bunch of scammers. I'm already aware of your anti-Altcoin tendencies but I didn't expect you to make false claims like that.

i go with what i see. and what i have seen so far is that dozens of altcoins have been created, many with a lot of hype starting from LTC and Monero down to ETH, ZCash and finally Grin.

LTC was meant to decrease block time in the first place, ETH is about Turing completeness/smart-contracts, Monero and Zcash are focused on privacy and finally, Grin belongs to a very different world: Mimblewible none of these projects were built around ASIC-resistance as their main design principle.  
I do agree that some altcoin projects are poisoned by greed but ASIC-resistance is not serving as the main instrument for such projects.

Secondly, your block-template based idea which Matt's Betterhash is a realization of it won't succeed ever because pool operators are not able to verify every single share submitted to them without reducing the number of shares asking for blocks with higher merit and ending up with a high variance that is an existential contradiction.
It is just oversimplifying a critical and inherent flaw built to bitcoin which requires both deep knowledge and true courage to address.

my point is that if we want to address the problem i outlined, we should take the route that is similar to the betterhash proposal instead of a route that is similar to what OP proposes.

Unfortunately, playing around with Stratum is not a promising route either. OP is following the worst path, I see, but it doesn't justify wrong paths neither makes them look even better. Op wants to make it impossible for miners to 'co-operate' (his own term) and Matt Corallo tries to convince miners to choose an inefficient alternative to conventional pools, risking high stale rates. He is too conservative to go with a radical project but it doesn't change anything about his 'solution', Betterhash: it is not gonna happen.

[aliashraf]

If the POW protocol itself is not changed, any new pool protocol only changes the rules for allocation, and there are no substantial changes to the security of the network.

Right but you need to apply a reasonable "change" not a destructive one, your proposal is not helpful as long as it pushes small even medium range mining farms out of the business, just take a look at the link which I provided above-thread.

[Sherlock.Holmes]

Right but you need to apply a reasonable "change" not a destructive one, your proposal is not helpful as long as it pushes small even medium range mining farms out of the business, just take a look at the link which I provided above-thread.

First of all, I agree with you. The most ideal strategy is the collaborator-takes-share strategy.

However, for projects with an open consensus mechanism such as Bitcoin, participant competition is the strongest guarantee of network security, and the result of competition must be a winner-take-all. The disadvantage of the winner-take-all is that it brings large probability fluctuations to the participants. Excessive probability fluctuations will cause the participants to leave the network. In order to effectively reduce the probability fluctuation, we can use some methods such as:

1 Reduce the block time, as in the "luck" project, the block time is 10 seconds, which can reduce the risk of probability fluctuations to a certain level.

2 Many second prizes and third prizes can be set for each round of competition. It must be noted that any collaborator-take-share mechanism must be built on a second layer protocol. The basic protocol must be simple, so we will not design any complicated collaborative-take-share mechanism on the basis protocol.

[hextobig]

Have to say, this is the best solution I have seen to eliminate the threat of mining pools.

Pools take pow problems and break them down into smaller problems, individuals have a much better shot at solving the easier problems and get reimbursed for their CPU/GPU/ASIC cycles. Naturally, the power of these pools has exploded over time with the collective hashrate worrying some of the members within the Bitcoin community of a potential 51% attack.

The 2POW proposed in this article makes cooperation between miners impossible, which aim at one-cpu-one-vote.

Although this looks much better than Bitcoin. In the end, are there still only a few large mining teams, the number of teams will be greater than the number of mining pools?

[Sherlock.Holmes]

Testnet will be opened on June 25, 2020. Welcome to join.
Refer to https://lucknet.club for details.

[elvisage]

In the traditional proof-of-work mechanism, we believe that cooperative mining is inevitable. This is because there are no puzzles can’t be outsourced.

I doubt this claim. These protocols achieve non-outsourceable pow mining.

https://github.com/DEX-ware/vrf-mining/blob/master/paper/main.pdf
https://hackingdistributed.com/2014/06/18/how-to-disincentivize-large-bitcoin-mining-pools/
http://soc1024.ece.illinois.edu/nonoutsourceable_full.pdf

[Wind_FURY]

What's wrong with BetterHash? What miners are really required to be doing is run their own full nodes, and remove the mining pool's political leverage on the network, right?

[Sherlock.Holmes]

In the traditional proof-of-work mechanism, we believe that cooperative mining is inevitable. This is because there are no puzzles can’t be outsourced.

I doubt this claim. These protocols achieve non-outsourceable pow mining.

https://github.com/DEX-ware/vrf-mining/blob/master/paper/main.pdf
https://hackingdistributed.com/2014/06/18/how-to-disincentivize-large-bitcoin-mining-pools/
http://soc1024.ece.illinois.edu/nonoutsourceable_full.pdf

In such solutions, non-outsourceable puzzles require miners to give private keys.

If the private key signature is introduced into the puzzle, in addition to the risk of losing the funds, the problem of cooperative mining still exists, this will bring greater probability fluctuations to miners.

If the private key signature is not introduced in the puzzle, We believe that there are no puzzles can't be outsourceable. because the puzzles required in the proof of work mechanism needs to have three conditions: Difficult to compute, easy to verify, and measurable difficulty. The puzzles that satisfies these three conditions must be outsourcable.

[Sherlock.Holmes]

What's wrong with BetterHash? What miners are really required to be doing is run their own full nodes, and remove the mining pool's political leverage on the network, right?

In fact, BetterHash does not increase the security of Bitcoin through increasing decentralization, it merely shifts the point of centralization.

Because the pool operators still have a large amount of control, they may force miners to review certain rules under the threat of not being rewarded.

2POW proposed in this paper prevent cooperation between miners from the basis protocol, which make pools impossible.

[aliashraf]

What's wrong with BetterHash? What miners are really required to be doing is run their own full nodes, and remove the mining pool's political leverage on the network, right?

It has been 2 years and Matt's proposal is not showing enough stem nor any promising signs on the horizon, so, I don't want to denounce it as a failed project but obviously there are serious shortcomings.

What is Betterhash?
Matt Corallo's Betterhash is designed to 'help' with the centralization of mining scene in bitcoin (and most PoW coins) which is considered as an inevitable consequence of pools dominating it.

The main idea is improving Stratum (the protocol behind pooling) such that individual miners could have a chance to build their custom blocks by including/excluding transactions deliberately while they are using pool operator's address in the coinbase transaction as the wallet that collects the rewards. This is done by a pretty sophisticated handshake phase in which the miner receives a block template from the pool in each round instead of receiving a simple block header (which commits to a full block of pool's choice and the miner has no clue about its contents) and trying to find a nonce for it which is how things work now.

For the miner to be able to fill in the block template, building his own block, it would be necessary to run a full node and it is the first not the last thing that this protocol does for improving decentralization in bitcoin. Other strengths include:
- Resisting covert censorship (my term) threats:
Officially, Matt and proponents of Betterhash believe that it is strong mitigation for censorship as a whole but it is not exactly the case. Pools are efficient when they are large enough and large pools are subject to political and jurisdiction intervention. Betterhash leaves centralized architecture of pool/miner relationship untouched and it is more than obvious that pool operators could be forced to reject miner generated blocks/shares that do include transactions that are hypothetically supposed to be censored and Betterhash can do absolutely nothing about it.

What Betterhash can do for censorship resistance tho is making it very hard and almost impossible for pool operators to do it covertly and in accordance with their private agendas.

- Resisting hash rate hijacking
It has been observed in some occasions pool operators use the miners' hash power to mine alternative chains for political purposes, Betterhash can mitigate this attack because miners are fully engaged with what they are mining.

- Collusion resistance
More block-generators, less collusion feasibility. Given Betterhash was possible to dominate the mining scene of bitcoin it would make it bulletproof against all forms of 51% attack.

So, why not Betterhash?
Unfortunately, Matt's protocol isn't employed and won't be ever because of a basic flaw: cost/benefit justifiability problem.

Looking again at the pros listed above (and other strengths not listed) it is obvious that they all share the same property of being strategically beneficial for bitcoin and lack one simple property: bringing any tangible advantage in terms of income for miners and pools at the same time that it comes with meaningful costs for miners and huge costs for pools.

Miners costs for implementing Betterhash
Besides the obvious cost of setting up a full node, miners need a substantially better internet connection and a local proxy and will experience a more 'stale rate' than what they get with legacy Stratum because it takes more time for the pool server to validate and accept their shares.

Pools costs
Here is the bottleneck. Better hash requires pool server to do an exhaustive job each time a miner issues a share and if they want to compete with the current Stratum utility level in terms of covering the variance it is not possible to use higher target difficulties for lowering the number of shares miners are used to submit. Hence a pool server should meet a very high level of hardware and Internet bandwidth requirement for handling the same number of miners that are connected now. The situation worsens considering the migration phase in which pool servers need to have parallel capacities for both versions of Stratum.

Conclusion
Betterhash is a good but impractical idea because it is trying to compete with the optimized (still fragile) current mining protocol without offering any tangible advantage. In a decentralized, wild environment like bitcoin, actors do not pay for and commit to what is considered good for the community without getting reasonable rewards almost instantly, it is called The Tragedy Of Commons.