Getting rid of pools: Proof of Collaborative Work

[aliashraf]

Proof of Collaborative Work

A proposal for eliminating the necessity of pool mining in bitcoin and other PoW blockchains

Motivation
For bitcoin and the altcoins which are based on common PoW principles,  centralization of mining through using pools, is both inevitable and unfortunate and puts all the reasonings that support the security of PoW in a paradoxical, fragile situation.

A same problem does exist with PoS networks. Things can get even worse  there because of the fact that most PoS based systems enforce long run deposit strategies for miners that is highly discouraging for them to migrate from one pool to another because of the costs involved.

Satoshi Nakamoto's implementation of PoW that is the core of bitcoin client software is based on a winner-takes-all strategy which is the fundamental factor behind two critical flaws: mining variance and proximity premium which are the most important forces that participate in forming pooling pressure.

Until now, both mining variance and proximity premium are known to be unavoidable and hence pooling pressure is considered an inherent flaw for bitcoin and other PoW based currencies.

In this proposal, we are suggesting an alternative variant of PoW in which the traditional winner-takes-all is replaced with a collaborator-takes-share strategy.

The problem of solo mining becoming too risky and impractical for small mining facilities appeared in 2010, less than 2 years after bitcoin had been launched. It was the worst timing ever, although Satoshi Nakamoto made a comment on bitcointalk about first pool proposals,  it was among the latest posts Satoshi made and he just disappeared few days later from this forum, forever, without making a serious contribution to the subject.

This way, the confused community came out with an unbelievable solution for such a critical problem, a second layer centralized protocol, named pooling, boosted by greed and ignorance, supported by junior hackers who as usual missed the forest.

Bitcoin was just 2 years old when pooling age began and eventually dominated almost all the hashpower of the network.

A quick review of Slush thread in which Satoshi has made the above referenced reply, could reveal how immature and naive this solution was and has been discussed and how it has been adopted: In a rush with an obvious greed.
Nobody ever mentioned the possibility of an algorithm tweak to keep PoW decentralized. Instead everybody was talking about how practical was such a centralized service while the answer was more than obvious:
Yes! you can always do everything with a centralized service, don't bother investigating.  

Anyway, in the thread, one couldn't find any arguments about the centralization consequences or the possibility of alternative approaches including the core algorithm improvements

I think it is not fair. PoW is great and can easily be improved to eliminate such a paradoxically centralized second layer solution. This proposal, Proof of Collaborative Work (PoCW) is an example of inherent possibilities and capacities of PoW. I didn't find any similar proposal and it looks to be original but If there is a history, I'll be glad to be informed about.

The Idea is accepting and propagating works with hundreds of thousands times lower difficulties and accumulating them as a proof of work for a given transaction set, letting miners with a very low shares of hash power ( say of orders like 10^-6) to participate directly in the network and yet experience and monitor their performance on an hourly basis.



Imo, now, after almost a decade being passed, Moore law has done enough to make it feasible utilizing more bandwidth and storage resources and it seems to me kinda hypocritic to make arguments about 'poor miners' and pretending to be concerned about centralization threats and making excuses so for rejecting this very specific proposal that although increases the demand for such resources, can radically disrupt current situation with pools and centralized mining.

This proposal is mainly designed for bitcoin. For the sake of convenience and letting the readers to have a more specific perception of the idea, I have deliberately used constants instead of adjustable parameters.

Outlines

• An immediate but not practically feasible approach can be reducing blocktime (along with proportional reduction in block reward). Although this approach, as mentioned, can not be applied because of network propagation problems involved, but a very excellent consequence would be its immediate impact on the scalability problem if employed, we will use it partially (reducing blocktime to 1 minute compared to current 10 minutes period).
  
• As  mentioned earlier (and with all due respects to Core team), I don't take objections about the storage and network requirements implications and consequences of reducing blocktime as a serious criticism. We should not leave mining in hands of 5 mining pools to support a hypothetical poor miner/full node owner who can not afford installing a 1 terabyte HD in next 2 years!.
• Also note, blocktime reduction is not a necessary part of PoCW, the proposed algorithm, I'm just including it as one of my old ideas (adopted from another forum member who suggested it as an alternative to infamous block size debate and later has been developed a bit more by me) which I think deserves more investigation and discussion.
  
• PoCW uses a series of mining relevant data structures to be preserved on the blockchain or transmitted as network messages
  
  • Net Merkle Tree: It is an ordinary Merkle hash tree of transactions with the exception that its coinbase transaction shows no block reward (newly published coins) instead the miner charges all transaction fees to his account (supports SegWit)
  • Collaboration Share: it is  a completely new data structure composed of following fields:
    
    • 1- The root of a Net Merkle Tree
    • 2- Collaborating miner's wallet address
    • 3- A nonce
    • calculated difficulty using previous block hash padded with all previous fields, it is always assumed to be at least as hard as 0.0001 compared to current block difficulty
  • Coinbase Share: it is new too and is composed of
    
    • 1- A Collaborating miner's wallet address
    • 2- A nonce
    • 3- A computed difficulty score using the hash of
      • previous block's hash padded with
        
      • current block's merkle root, padded with
      • Collaborating miner's address padded with the nonce field
    • 4-  A reward amount field
  • Shared Coinbase Transaction: It is a list of Coinbase Shares  
    • First share's difficulty score field is fixed to be  2%
    • For each share difficulty score is at least as good as 0.0001
    • Sum of reward amount fields is equal to block reward and for each share is calculated proportional to its difficulty score
  • Prepared Block: It is an ordinary bitcoin block with some exceptions
    
    • 1- Its merkle root points to a  Net Merkle Tree
    • 2- It is fixed to yield a hash that is as difficult as target difficulty * 0.05
  • Finalization Block: It is an ordinary bitcoin block with some exceptions
    
    • 1- Its merkle root points to a  Net Merkle Tree
    • 2- It is fixed to yield a hash that is as difficult as target difficulty * 0.02
    • 3- It has a new field which is a pointer to (the hash of) a non empty Shared Coinbase Transaction
    • 4- The Shared CoinBase Transaction's sum of difficulty scores is greater than or equal to 0.95

• Mining process goes through 3 phases for each block:
  • Preparation Phase: It takes just few seconds for the miners to produce one or (barely) 2 or 3 Prepared Blocks typically. Note that the transaction fees are already transferred to miner's wallet through coinbase transaction committed to the Net Merkle Tree's root for each block.
  • Contribution Phase: Miners start picking one valid Prepared Block's Merkle root, according to their speculations (which become more accurate as new shares are submitted to the network) about it to get enough shares eventually, and producing/relaying valid Contribution Shares for it.
    As the sum of the difficulty scores for a given Prepared Block's Merkle root grows we expect an exponential convergence rate for the most popular Merkle root to be included in Contribution Shares.  
  • Finalization Phase: After the total scores approaches the 0.93 limit, rational Miners would begin to produce a Finalized block
• Verification process involves:
  • Checking both the hash of the finalized block and all of its Shared Coinbase Transaction items to satisfy network difficulty target cumulatively
  • Checking reward distribution in the shared coinbase transaction
  • Checking Merkle tree to be Net
• UTXO calculation is extended to include Shared Coinbase Transactions committed to finalized blocks on the blockchain as well
• Attacks/forks brief analysis:
  • Short range attacks/unintentional forks that try to change the Merkle root are as hard as they are in traditional PoW networks
  • Short range attacks/unintentional forks that preserve the Merkle root but try to change the Shared CoinBase Transaction has  zero side effects on the users (not the miners) and as of redistributing the shares in favor of the forking miner, they are poorly incentivized as gains won't go anything further than like %2-%10  redistribution ever.
    
  • Long Range attacks with a total rewrite agenda will fail just like Traditional PoW  
  • Long Range attacks with partial coinbase rewrite are again poorly incentivized and the costs won't be justified

Implementation

This is a radical improvement to classical PoW, I admit, but the costs involved are fair for the huge impacts and benefits. I have reviewed the bitcoin Core's code and found it totally feasible and practical form the sole programming perspective. Wallets could easily be upgraded to support the new algorithm as well,  but a series of more complicated issues, mostly political are extremely discouraging but it is just too soon to give up and go for a fresh start with a new coin, or just manage for an immature fork with little support, imo.

Before any further decisions, it would be of high value to have enough feedback from the community. Meanwhile I'll be busy coding canonical parts as a BIP for bitcoin blockchain, I think it takes like 2-3 weeks or even a bit more because I'm not part of the team and have to absorb a lot before producing anything useful, plus, I'm not full time, yet

I have examined the proposed algorithm's feasibility as much as I could, yet I can imagine there might be some flaws overlooked, and the readers are welcome to improve it. Philosophical comments questioning the whole idea of eliminating pools don't look to be constructive tho. Thank you.


Major Edits and Protocol Improvements:

• June 10, 2018 09:30 pm Inspired by a discussion with @ir.hn
  
  
  • A Prepared Block should be saved in the fullnodes for a long period of time enough to mitigate any cheating attempt to avoid Preparation Phase and using non-prepared, trivially generated Net Merkle Roots.  
    
    • Full nodes MAY respond to a query by peers asking for a block's respected Prepared Block if they have decided to save the required data long enough
      
    • For the latest 1000 blocks preserving such a data is mandatory.
    • For blocks with an accumulated difficulty harder than or equal to the respected network difficulty, it would be unnecessary to fulfil the above requirement.^*
    • Prepared Block and Preparation phase terms replaced the original Initiation Block and Initiation Phase terms respectively to avoid ambiguity
    Notes:
    ^* This is added to let miners with large enough hash powers choose not to participate in collaborative work.

• reserved for future upgrades
• July 3, 2018 02:20 pm inspired by discussions with @anunimint
  
  
  • A special  transaction was added to Shared CoinBase Transaction to guarantee/adjust proper reward for the finder of Prepared Block and some enhancements were made to include both block reward and transaction fees (partially) in the final calculations.
    
    • In Finalization Phase, miners should calculate a total reward for the miner of its respected Prepared Block as follows:  
      preparedBlockReward = blockReward *0.04 + totalTransactionFees *0.3
    • Shared Coin Base Transaction should include one input/output address-amount pair that adjusts the amount of reward assigned to the Prepared Block Miner in the ordinary coinbase transaction
    • All the calculations needed for calculating rewards assigned to the miners of finalized block and shares will be carried out on the sum of network block reward and 70% of transaction fees collected in transactions that are committed to Net Merkle Tree
    Note:
     This change is made to both guarantee a minimum reward for miners of Prepared Blocks and incentivize them for including more transactions with higher fees  

• reserved for future upgrades

[goddog]

I'm sorry, I hope I can not fully understand your proposal, but if you are against centralized pools, why  do not simply try to improve p2pool protocol? Pow have to be simple so it don't need tuning costants finding some magic to have things working. Simple mean less errors and problems.

I hope pools are not a problem if miners can point their hashrate where they want, at any time at no cost.
however p2pool is a nice try to have  decentralized pools. You can try to develop a decentralized pool, if it is better than centralized pools solutions, it will be adopted by miners.

[aliashraf]

@goddog
P2Pool has been around for a long time but as of this writing it has not done good enough (like 0.00005 of the total network hash power, last block found according to their official stats page dates back to years ago! ) it is because of it being a lot more resource consuming than my proposal, and less efficient, I guess. I'm not happy to say this, but it looks like a failed project to me.

Nodes in the P2Pool protocol should maintain a separate blockchain plus performing a lot of validation checks on the submitted shares and there is no convergence mechanism unlike PoCW proposed here.

In PoCW, miners just converge on a Prepared Block's merkle root and focus on hashing instead of going through verifying and accounting submitted shares. It is just in Finalization Phase that they need to collect a set of valid shares with the sum of their difficulties being sufficient to satisfy the target difficulty of the network.

Another problem with P2Pool is that the level of convenience provided  is not enough (120 times  lower difficulty). PoCW, proposed here, can accept shares up to 100,000 times easier, like what centralized pools are capable of.

[goddog]

I mentioned p2pool only as a starting point, I know it has a lot of problems and flaws,
From the little I can understand your proposal can be applied to the current pow system as a new kind of decentralized pool software(like an improved f2pool), you can call it C2pool :-D

However I can not, really, understand how your proposal should work. From what I can understand yuor finalization block , is the only thing that matters, so miners will simply fake all others data, or selfish mining all of them at no cost, to maximize their profit.

Also I think it is very susceptible to network latency and/or split, making it vulnerable very vulnerable to a wide range of sybil attack.
But as I was saying for sure I'm missing some foundamental point.
So can you explain me why a miner should be cooperative? I can not see where are the real incentives.

[aliashraf]

@goddog

No it is not a pool of any kind, instead a new type of PoW. I see you are too lazy to read my opening post  

In classical PoW, although miners dedicate resources to generate blocks with average value, in terms of difficulty, they are completely ignored, (this is why people join pools). My Simple question can be formulated this way: Why just don't let this blocks to be accumulated as a proof of collaborative work?

Classical PoW, the way it is implemented in bitcoin, Ethereum, ..., fails to do so because typically the block is hashed as a whole. It is because the block header contains the merkle root that in turn has committed every single information effective on the blockchain state, both conventional transactions and the coinbase.

This prevents miners to collaborate because each miner generates a different coinbase (reflecting his reward address and amount) that produces a different merkle root in turn.

In PoCW, we separate coinbase transaction from the Merkle Tree. This way, miners can share their 'good hashes' for a Merkle root that is getting most popular over time and when enough 'good hashes' are accumulated, they try their chance to generate the final block that now can use the found shares to prove its quality and as an evidence for the way it suggests for distribution of the block rewards, at the same time, by means of the used shares that are unforgeable and can not be separated from the miner's wallet address who generated them.

In Collaboration phase, it is of miners best interest to converge asap on a popular Merkle root (transaction set) and submit their found shares that satisfy the trivially set difficulty.
When producing Finalized Blocks, miners have to include the previously found  share (as they are) and this way they have to admit the founders share, fairly.

In other words, in PoCW miners mainly focus on the actual ordinary transactions made by users and share their efforts to take care of them. Instead of artificially ruining the transaction set by injecting their reward expectation and delaying the consensus mechanism for a fortunate accident of a miner to 'find' a block with high enough difficulty.

PoCW looks to me ways more decent and fair and at the same time very solid, compared to PoW.

I hope this brief explanation could help and encourage you to read the opening post in its entire and more carefully.  

[goddog]

I'm only trying to understand I'm not trying to judge your work, but I need some clarification.

• Initiation Phase: It takes just few seconds for the miners to produce one or (barely) 2 or 3 Initiation Blocks typically. Note that the transaction fees are already transferred to miner's wallet through coinbase transaction committed to the Net Merkle Tree's root for each block.

As I can understand a net merkle is a merkle tree where the coinbase transaction have no block rewards(only comulated fee) so each miner will try to publish his net merkle tree with a coinbase tree pointing to his address.
why are you saying there will be only one or (barely) 2 o 3  initialization blocks? why a miner should use a net merkle tree with a coinbase pointing to an address owned by other miner? where is the incentive to start contribution phase loosing transaction fees? or I'm missing something.

• Contribution Phase: Miners start picking one valid Initiation Block's Merkle root, according to their speculations (which become more accurate as new shares are submitted to the network) about it to get enough shares eventually, and producing/relaying valid Contribution Shares for it.
  As the sum of the difficulty scores for a given Initiation Block's Merkle root grows we expect an exponential convergence rate for the most popular Merkle root to be included in Contribution Shares.  

They will share an inizialization block, adding their address, and their, partial, proof of work to earn score.

• Finalization Phase: After the total scores approaches the 0.93 limit, rational Miners would begin to produce a Finalized block

[/li][/list]

as soon as a miner get enought shares(with partial proof of work) it can begin trying to solve a more difficult proof of work to get only a partial block reward. But producing a final block is very hard as any contrib share can point to a different net merkle tree(because every miner will add a coinbase transaction using his address to earn fees), so he have to validate the same transactions more and more times to be sure final block wil be considered valid. So I hope it is better for a miner to simply  autoproduce his net merkle tree and his contribution shares so he doesn't have to share block reward with others contributors and to get the full fee booty.

So I can not understand where is the point in using contribution shares from others miners?

[aliashraf]

I'm only trying to understand I'm not trying to judge your work, but I need some clarification.

• Initiation Phase: It takes just few seconds for the miners to produce one or (barely) 2 or 3 Initiation Blocks typically. Note that the transaction fees are already transferred to miner's wallet through coinbase transaction committed to the Net Merkle Tree's root for each block.

As I can understand a net merkle is a merkle tree where the coinbase transaction have no block rewards(only comulated fee) so each miner will try to publish his net merkle tree with a coinbase tree pointing to his address.
why are you saying there will be only one or (barely) 2 o 3  initialization blocks? why a miner should use a net merkle tree with a coinbase pointing to an address owned by other miner? where is the incentive to start contribution phase loosing transaction fees? or I'm missing something.

5% difficulty requirement for Prepared Block causes one block to be generated in every 3 seconds, during the preparation phase, network latency and the required time for miners to decide ending preparation attempts and starting contribution phase may take up to 10 seconds, I guess.

When the first few Prepared Blocks have been already propagated in the network, other miners have tried their chance to win the fees and failed to do so. It is not a big deal tho, in the worst case it is about wasting like 5% of the resources and the transaction fees at the same time. Just like when a new block arrives and you have missed your chance to hit. Typically total transaction fees for a fully saturated block is about 0.5 btc i.e. 0.04 with respect to the current 12.5 btc block reward.
From this point on(having few published Prepared Blocks in hand) , the race is about winning the block reward (main meal) and it is of all miners best interest to forget about the previous race (winning tr fees) and focus on the live one(winning block reward).

• Contribution Phase: Miners start picking one valid Initiation Block's Merkle root, according to their speculations (which become more accurate as new shares are submitted to the network) about it to get enough shares eventually, and producing/relaying valid Contribution Shares for it.
  As the sum of the difficulty scores for a given Initiation Block's Merkle root grows we expect an exponential convergence rate for the most popular Merkle root to be included in Contribution Shares.  

They will share an inizialization block, adding their address, and their, partial, proof of work to earn score.

• Finalization Phase: After the total scores approaches the 0.93 limit, rational Miners would begin to produce a Finalized block

[/li][/list]

as soon as a miner get enought shares(with partial proof of work) it can begin trying to solve a more difficult proof of work to get only a partial block reward. But producing a final block is very hard as any contrib share can point to a different net merkle tree(because every miner will add a coinbase transaction using his address to earn fees), so he have to validate the same transactions more and more times to be sure final block wil be considered valid. So I hope it is better for a miner to simply  autoproduce his net merkle tree and his contribution shares so he doesn't have to share block reward with others contributors and to get the full fee booty.

So I can not understand where is the point in using contribution shares from others miners?

No you are missing the most important point here. Contribution shares don't point to different merkle trees, it is the magic of what I have defined as Net Merkle Tree, it is shared among most of the miners because it contains no block reward related information, just the real world transactions and their fees. Miners do not change this tree and produce contribution shares that commit it again and again. This is why these shares can be used as an objective evidence of  resource consumption of their respected producers.


EDIT:
Please pay enough attention that in contribution phase, miners don't mine blocks, they mine Contribution Shares instead that can be easily included in the Shared Coinbase Transaction in Finalization Phase. As I've already explained in the first post.

[ir.hn]

This is a very complex idea and you have defined your method but not your reasons for determining this method.  You may want to start from scratch to explain this and give the idea, most basic theoretical implementation and then what problems occur that lead you to add the new factors that you did in the design.

A lot of thought went into this and it is hard for us to get up to speed with your thought process.

What you are doing is giving someone a recipie and saying it makes a better cake but not enough explanation for why eggs whipped at 675 rpm are needed exactly. 

I'm sure it is genius and mabye a couple pepple could follow it but I'm not amoung them.  Also realize that not many of us probably even understand the underpinnings of how pools work exactly.

[aliashraf]

@ir.hn

I do admit that it is not an easy piece of cake when you get into implementation details , or even the design outlines, directly. But things are changing so fast in blockchain and developments are trending in so many false directions ... We have to be simultaneously perfect and quick. Hell of a job ...

Realising this situation, I deliberately chose to present this idea, from a technical point of view, to accelerate technical contributions. Yet, I believe you would be able find some useful general ideas following this topic.

Here I try to give a more general  and conceptual vision, I wish it would help:

In centralized pool mining that has been the dominant practice for years,  miners have no control on the blocks, they receive it from the server, ready to 'mine' which now is about finding a nonce that causes the hash of the block to be 'good enough' to be passed back as a share to the server.
The threshold of what is considered to be good enough is contracted between miner and the pool server and has almost nothing to do with the blockchain difficulty, it is just considered to be thousands to millions of times lower.

Pool server in turn, besides validating and keeping track of submitted shares, checks whether luckily, one of them is so much good (thousands or millions times better than what the trivial contracted difficulty has forced miner to find a nonce for) to be relayed as a freshly found block and added to the blockchain.
If so, the pool will get both block reward and transaction fees immediately. Shares of the miners who participated in this process, all of them and not only the one who has found the golden nonce, will be payed using one of the several common methods for pool accounting, later.

Miners desperately need such a mechanism to avoid waiting for years to find one block (if any) and pools with enough number of submitted shares are expected to be lucky very frequently.

This is a bad situation for security of the network because number of parties who actually decide about the transaction set to be included in each block and (probably) added to the blockchain is dangerously small. This is a centralization threat and any true bitcoiner should be against it, unquestionably.

Actually, as of this writing, 5 major bitcoin mining pools could easily form a cartel that would have >50% majority needed to perform occasional and intentional double spend attacks against people, for the least.

As I have briefly described in my article, I believe PoW can be improved to eliminate the miner's above mentioned need to pool mining. In this proposal this is done by reducing the difficulty hundreds of thousands times.

This difficulty reduction is achieved by tweaking classical PoW to let miners converge on a common set of transactions and submit their found results to the network primarily and wait until their submitted shares have been summarized in a finalized block to be permanently added to the network. In

In classical PoW it is done upside down, a finalized block is prepared from the first beginning, when the 'nonce hunt' is going to start,  no cooperation, no contribution, this is a gap that maliciously is filled by pools and I believe can be totally eliminated by PoCW, this proposal.

[ir.hn]

So what your saying is... basically Proof of Participation (you can steal PoP I won't be offended  ).  Everyone gets a participation trophy which is encoded into the "real" block before the block reward is distributed?  So basically there is in reality one big mining pool run by the software?  Sounds really good in theory.

Who's ledger would be the accepted ledger for the transactions in that case?  I think that is why bitcoin has one winner, so that one person's ledger becomes the accepted ledger.

Like you I figured the only way to beat pools other than coins like SpreadCoin (not sure if it is successful at it) was lowering the blocktime (which is a double edged sword in many ways) but also another idea I had was increase coinbase maturity to something like 9 months.  I figured that would throw a wrench into pools what do you think?

[GLP]

I thinked a lot about this proposition, and finally i think that:
1- It is a Decent one and i have not find any relevant technical objection against.
2- It's problem will be both social & political

So, i suggest you to make a proof of concept of it by implementing it on some testnet, ppl who embrace it can help by putting some bounty to hack it. benefits will be:
1- objectivity on the proof of work.
2- If the political resistence is umbeatable and the social insensitive is not motivated, you will have be allready done a half-step to it's productivity implementation on a new blockchain.

As you may agree, a political resistence is not something that can be forced by any rational/objective argumentation alone.

Thank you for sharing and i really valuate this work, please do not give up by a "BTC or nothing" reasoning, it deserve better horizons.

Wish you the best, i definitely will think more about it and will be glad to share any future thoughts.

[aliashraf]

@GLP

Thanks for your encouraging reply. Glad to see you are supporting the core idea and I'm looking forward for your future contributions to this project.

As I have mentioned earlier, we are in an intense situation and we need to act like a samurai, both quick and perfect. Unless,  we do need spiritual, social, economical and technical contribution unlike a samurai.

I strongly support your suggested roadmap and will do my best to make it happen. Thank you.

[aliashraf]

So what your saying is... basically Proof of Participation (you can steal PoP I won't be offended  ).  Everyone gets a participation trophy which is encoded into the "real" block before the block reward is distributed?  So basically there is in reality one big mining pool run by the software?  Sounds really good in theory.

Thanks for the explanation and yes, people who are used to pool mining, would experience the network like a huge pool. Good analogy.  

By the way, I prefer keeping the title as is, Proof of Collaborative Work.

PoCW is just an  improvement to PoW that inherits its core objective ideas and could be categorized as an implementation variant rather than a new algorithm.

Who's ledger would be the accepted ledger for the transactions in that case?  I think that is why bitcoin has one winner, so that one person's ledger becomes the accepted ledger.

No matter whose ledger it is, rational miners should contribute to the first ledger (at the start of contribution phase) and converge to the most popular one (as this phase develops). Obviously the ledger under consideration should always be examined for its validity.

[buwaytress]

Always reading with interest and happy to see that people are always trying to work on this problem of pool centralisation, although I'm not sure any of the direct threats (51% attacks for example) would really happen since pools themselves, no matter how large, have historically understood the fine balance they need to maintain to avoid miners just disconnecting from their pools to join others.

Nevertheless, it is at these finely-maintained balances that centralised pool decisions are a concern. I just wanted to share something else I read (a BIP draft) yesterday that looked instead to just change the protocol to give a bit less power to pools when it comes to creating block templates.

Your suggestion of a way to somehow make individual miners count for the work they contribute, as well, is something a lot of people would support. This makes me wonder if anyone has ever calculated the cost of wasted work (work that never counted), I'm sure it's a LOT.

[aliashraf]

@buwaytress

Thanks for the support and sharing the referenced BIP, I just posted a reply in your topic.

As you  may have noticed and I have tried to prove there, that is somehow too conservative and can't change things more than like what you say 'a bit'.

[ir.hn]

Ok thanks for your explanation it is helping me go back to your original post and make more sense out of it.

So the net merkle tree is a merkle tree without the block reward.  So if you win a nonce for that (collaboration share) you don't automatically get the block reward alla bitcoin.

I was wrong that it is a participation trophy.  It is a weighted block reward for each participant (collaborator) that varies based on how "good" their nonce was.  Is there a set time limit for collaboration shares, so a miner can basically "wait untill I get a nonce -this- good" before sending it in?  Or should they just find a bunch of low difficulty nonces?  Is there a limit as to how many collaboration shares can be included into the shared coinbase transaction?

How does the "convergence on an accepted transaction ledger" happen exactly and automatically? 

[aliashraf]

So the net merkle tree is a merkle tree without the block reward.  So if you win a nonce for that (collaboration share) you don't automatically get the block reward alla bitcoin.

Exactly.

...  Is there a set time limit for collaboration shares, so a miner can basically "wait untill I get a nonce -this- good" before sending it in?  Or should they just find a bunch of low difficulty nonces?  Is there a limit as to how many collaboration shares can be included into the shared coinbase transaction?

Yes. It is set to be at least as 0.0001 good as what the calculated network difficulty enforces and in a worst case scenario, there would be like 9,500 Coinbase Shares  packed in Shared Coinbase Transaction, in my terminology. Depending on the implementation details it implies up to 64 KB size for the transaction.

In practice you can expect much lower numbers down to just 2 transactions. So, regarding the normal distribution of probabilities, I guess 4,250 is a good assumption for the average. Hence 32 KB or so for Shared Coinbase Transaction.

How does the "convergence on an accepted transaction ledger" happen exactly and automatically?  

Once the very first Prepared Block is mined and propagated in the network, for average rational miners there is no more reasons to keep trying to mine another competitive instance of such a block, as long as they are sure that the mined block under consideration is valid and won't be rejected by peers, instead they would rationally accept the fact that they have lost the chance of gaining transaction fees and switch to Contribution Phase, producing shares for the Net Merkle Root (the transaction ledger) of the same block. It is the most innovative part of my proposal, I guess.

Obviously the above mentioned process converges exponentially because with every new Collaborative Share issued and propagated, the most popular Net Merkle Root gets even more support from the network. I think before we get to even close to 20% of the needed shares , practically the whole network would be mining the same work.

[ir.hn]

Yes. It is set to be at least as 0.0001 good as what the calculated network difficulty enforces and in a worst case scenario, there would be like 9,500 Coinbase Shares  packed in Shared Coinbase Transaction, in my terminology. Depending on the implementation details it implies up to 64 KB size for the transaction.

What I was getting at actually was this: what if there is a really good miner.  You don't want this miner giving contribution shares every time he gets a .0001 nonce, because then he would fill up the block will all his contribution shares.  You would rather have him wait until he gets a .01 nonce for example that way he can get a bigger part of the pie yet not crowd the block with tons of small value nonces. If there was a set time limit he could say "ok I can wait until I have at least a nonce of .01 difficulty before sharing it, because I know in that given time frame I should be able to find one that high". The way this can be done is to reward higher difficulty blocks a little bit disproportionately higher.  But the problem remains there still there would be the incentive to post his lower difficulty nonces too, so I'm not sure what the solution would be to get him to not share his low difficulty nonces.

It may not be vital but it would be good if a flood attack like this could be avoided.

Once the very first Prepared Block is mined and propagated in the network, for average rational miners there is no more reasons to keep trying to mine another competitive instance of such a block, as long as they are sure that the mined block under consideration is valid and won't be rejected by peers, instead they would rationally accept the fact that they have lost the chance of gaining transaction fees and switch to Contribution Phase, producing shares for the Net Merkle Root (the transaction ledger) of the same block. It is the most innovative part of my proposal, I guess

Wow very cool.  So what you are saying that the "Prepared Block" is always mined first and has it's own proof of work requirement separate from anything else?  If so we could think of this Prepared Block as the "real" block of the blockchain.  So the winner of the Prepared Block gets the transaction fees, and the Contribution block mined later is for the block reward coins?  So the only problem is a little bulkier and slower blockchain, instead of 1 block mined per transaction group, there are two.  Not a bad tradeoff though when you consider not only the risk of pools on the network but the risk of pools in terms of loosing your money from being hacked, and making accounts and passwords for different pools and whatever.

[aliashraf]

Yes. It is set to be at least as 0.0001 good as what the calculated network difficulty enforces and in a worst case scenario, there would be like 9,500 Coinbase Shares  packed in Shared Coinbase Transaction, in my terminology. Depending on the implementation details it implies up to 64 KB size for the transaction.

What I was getting at actually was this: what if there is a really good miner.  You don't want this miner giving contribution shares every time he gets a .0001 nonce, because then he would fill up the block will all his contribution shares.  You would rather have him wait until he gets a .01 nonce for example that way he can get a bigger part of the pie yet not crowd the block with tons of small value nonces. If there was a set time limit he could say "ok I can wait until I have at least a nonce of .01 difficulty before sharing it, because I know in that given time frame I should be able to find one that high". The way this can be done is to reward higher difficulty blocks a little bit disproportionately higher.  But the problem remains there still there would be the incentive to post his lower difficulty nonces too, so I'm not sure what the solution would be to get him to not share his low difficulty nonces.

It may not be vital but it would be good if a flood attack like this could be avoided.

Two points I have to clear here:

• First: A miner who finds a Collaboration Share with a nonce better than 0.0001 will be rewarded with proportional amount of the block reward eventually, given he has chosen the right Net Merkle Tree to mine for, the one that network has converged around it
• Second: A miner can mine any number of shares and publish them

Plus, it is important to note that Collaboration Shares are very small payloads (like 50 bytes) and there is no reason to set the quality limit that high.

In current Bitcoin blockchain once a block is found by a miner (being a solo miner or a pool operator) it takes a lot of communication overhead to be propagated, because the peer that is receiving the block should become aware of the exact Merkle tree path behind the Merkle root present in the header, at least the transaction ids should be transmitted and in some cases the actual transaction body needs transmission.

Also, I'm generally against hesitations to utilize networking bandwidths. I think it is very weird design strategy for a decentralized consensus based system running on an infrastructure like Internet.

Once the very first Prepared Block is mined and propagated in the network, for average rational miners there is no more reasons to keep trying to mine another competitive instance of such a block, as long as they are sure that the mined block under consideration is valid and won't be rejected by peers, instead they would rationally accept the fact that they have lost the chance of gaining transaction fees and switch to Contribution Phase, producing shares for the Net Merkle Root (the transaction ledger) of the same block. It is the most innovative part of my proposal, I guess

Wow very cool.  So what you are saying that the "Prepared Block" is always mined first and has it's own proof of work requirement separate from anything else?  If so we could think of this Prepared Block as the "real" block of the blockchain.  So the winner of the Prepared Block gets the transaction fees, and the Contribution block mined later is for the block reward coins?  So the only problem is a little bulkier and slower blockchain, instead of 1 block mined per transaction group, there are two.  Not a bad trade off though when you consider not only the risk of pools on the network but the risk of pools in terms of loosing your money from being hacked, and making accounts and passwords for different pools and whatever.

Prepared block won't go to the Blockchain ever and the 5% difficulty level set here won't be checked later. It is just a convention for lubricating the convergence mechanism . Let's dive a bit deeper into it:

Once a Prepared Block is mined and propagated, receiving miners will check its integrity and difficulty before starting to contribute, so the sole fact that some miners contribute to it and produce shares for its Net Merkle Root it is enough proof that the block has convinced them already.

One possible attack scenario by a selfish owner of a very large mining facility could be something like this:

A selfish miner may choose to bypass Preparation phase by simply producing a Net Merkle Tree and starting to produce Collaboration shares and propagating them in the network.
Receiving miners may be fooled that they have missed something and there IS actually some Prepared Block out there and decide to join the race before it is too late.
This way the selfish miner has hijacked transaction fees without consuming enough resources and more importantly he has accelerated the mining process artificially which has problematic consequences for difficulty adjustment algorithm of the network.

Mitigation:
Decent, loyal miners won't contribute to a work that can not be validated by querying the Prepared Block, so such an attack won't work unless a compromise has been made between miners that are not expected to be a 50%+1 majority. And for this minority there will always be a problem to distribute the illegitimate gains properly, thus they have to set a lower limit, say 1%, and yet they are risking to lose the main war, ever and ever because in most cases they are converging to a work that the majority just don't commit to it.

When it comes to bootstrap or partial synchronization for example, full nodes won't ask like 'From where you got this Net Merkle Tree?' because it just doesn't matter anymore.

[ir.hn]

Hmm yes we should keep in mind finding something besides the internet would be better.  Ham radio is an option if the data size is very small.

I guess to answer my own question; yes a flood attack is possible but it is ok.  Difficulty would be adjusted so, for example, the block will typically fill up full of contribution shares every 10 minutes.  Instead of taking on average 10,000 x blocktime for a small miner to win his first block, he will win a little bit every block on average.  The big miners will fill up the block with small difficulty nonces, but the small miner has 10,000x better chance of getting one of his small difficulty nonces into the block.  Nice.

I think I am with you on the preparation block (same thing as "initiation block" I presume).  The person who can win that somewhat higher difficulty nonce will gain the transaction fees which will immediately show up in their wallet.  So this block never needs to become a part of any hash to prove that it happened?  It seems a little risky in my mind that it just disappears after it is won.  But I can see that it is just needed to "prime the pump" so the collaborators have something to check their ledger with.  Seems a tad bit risky though because it seems the prep block is just a suggestion.  I think from you explaining that "preparation block skipping" attack that the network may loose preparation blocks entirely, since it appears that the prep block hash doesn't need to be a part of the collaboration share.