Mempool/mining question

[PrimeNumber7]

I can see this particular feature preventing pools from adopting v2.

I'm not so sure. Taking the example you gave - there is a large miner who has reached a private agreement with a third party to include their transactions for a fee paid directly to the miner. This miner is mining for Pool X, and is still sharing their block reward and other fees with Pool X, but is depriving Pool X of the additional private fee they have negotiated with the third party. If Pool X decide this isn't good enough, and stop using Stratum v2, then the miner in question can simply switch to Pool Y which is still using Stratum v2. Pool X have now lost the entirety of that miner's hash power, and with it, all the block rewards and other fees for the blocks they were mining. They will end up in a worse position than if they had just continued to allow Pool X to continue to mine with their private arrangement in place.

Any miner who has such an arrangement and wants to be able to select which transactions they include will just move to a pool which is willing to let them do that, and there will always be some pools which are willing to do that. Any pools which don't upgrade will be left behind by these miners.

A miner who is mining on a pool 'honestly' is providing hashes, the value of each is worth one hash. A miner who is mining on a pool in a way I described in my previous post is providing hashes that are less valuable than the hashes from an 'honest' miner. In your argument, pool X would lose n hashes, but would also not have to payout for n hashes. The value of the hashes the pool looses would be less than the value the pool is no longer paying out.

The pool would have a higher market share by allowing a dishonest miner to continue mining on the pool, but doing so will result in the pool loosing money.

I might compare this scenario to a company with a team of salesmen. If one salesman is dishonest and pockets some payments from customers, the company is going to lose the money as a result of this. The salesman doesn't pocket all of the money from customers, so continuing to employ the employee would result in higher sales revenue compared to if the salesman was fired, but the employee is still causing the company to lose money.

I would also point out that a dishonest miner will result in Pool X being able to distribute less per hash to all other miners, so allowing a dishonest miner to continue this practice will result in honest miners mining elsewhere that don't allow this practice, and can afford to pay more per hash.

[garlonicon]

The problem with private fees is one of the reasons why I think that Stratum v2 alone will be not enough. Theoretically, skipping pools entirely is possible, if all nodes will create some kind of consensus when splitting block reward between all miners. I can imagine a system where for example 5,000 miners will share their block headers. That would mean 400 kB per block, but after distributing rewards on-chain that headers could be discarded. Probably, revealing more data will be needed, for example by including the whole coinbase transaction and SPV proof for that. Assuming more detailed proof will take 1 kB per share, it would be 5 MB per block. Assuming that all rewards will mature after 100 blocks, it means 500 MB per node. As far as I know, Stratum protocol allows shares up to 2^16 times easier than the network difficulty. If so, it means 64k shares instead of 5k, so around 6.4 GB per node.

Joining hashes can be done by simply calculating (firstHash*secondHash)/(firstHash+secondHash), the first multiplication of two 256-bit values would take 512 bits, then could be divided by some 256-bit value and the result will fill no more than 256 bits, assuming all hashes having at least 16 leading zero bits and no more than 2^16 shares, they will never overflow. During division, results will be always rounded down, so joining hashes in different order should always yield the same values.

In such decentralized system, even if someone will receive fees in some private way, that miner will simply receive lower reward, because each miner will receive a fraction of its own coinbase transaction, so the more fees that miner will include, the more coins that miner will get after dividing that coinbase reward. To calculate miner's income, the value of the coinbase transaction in satoshis could be multiplied by the target, resulting in the easiest possible target that will grant single satoshi. Then, that target could be divided by miner's hash to calculate number of satoshis (if done inside Lightning Network, smaller units like millisatoshis could be used).

Because interacting with all miners may take a lot of resources when proving that all miners are honest and their shares are valid, that system could be implemented on top of the Lightning Network. In this way, mining nodes will have to validate only two blocks per coinbase transaction, reducing space requirements to 8 MB per block (two blocks fully filled with Segwit transactions will take up to 4 MB each). That means around 800 MB per node. Previously mentioned 1 kB proof would work only when transactions could be joined, so that when calculating coinbase reward it won't be needed to reveal the whole block.

[DaveF]

The reason why blocks are sometimes empty immediately after a block is found, is because miners are blindly accepting that a previously found block is valid, and is mining on top of said block prior to confirming which transactions have been confirmed. Once miners validate a recently found block, they will update their database so that their new block candidates will reflect only valid transactions.

More or less correct. The biggest thing out there is the amount of bad info being discussed.

Many / most of the larger pools have many many many nodes. They all should be in agreement. However, due to many things there may be a period of time that some are updating / processing / working on the block that was just found / just having a bad day so they do not agree.

So they work on an empty block. Once whatever agreement conditions are met, usually within seconds, they work on blocks normally.

People like to scream that they are SPV mining or some other crap. Or have misconfigured nodes. And yes some are. And some are just being paranoid.

No matter how you slice it, even at the speed of light and ultra fast CPUs & drives & networks there will be a delay.

Because, more or less, mining 1 block a week with NO FEES is better then orphaning 1 block every 2 years because that 1 node was having a bit of a slowdown and missed something.

Now some people will chime in that they can have their node validate in under 2 seconds or some crap like that. Yeah, so can mine. But if I have 20 farms each with 2 nodes that all talk back to 2 or 3 servers that make sure that they all agree that then talk back to all the farms. It's going to take a bit. For a small miner that finds blocks once a month it's one thing. For one that is mining 10 blocks a day it's another.

-Dave

[PrimeNumber7]

In such decentralized system, even if someone will receive fees in some private way, that miner will simply receive lower reward, because each miner will receive a fraction of its own coinbase transaction, so the more fees that miner will include, the more coins that miner will get after dividing that coinbase reward.

Given a scenario in which a miner has sufficient hashrate to expect to mine 4 blocks per day, average tx fees of 0.05 per block, and a 3% pool fee to cover things such as orphans, and other expenses:
If a miner receives all the transaction fees in the blocks it finds privately, they will receive:
0.2 BTC in transaction fees privately
25 BTC in block subsidies via the pool
0.2 BTC in transaction fees via the pool
-0.756 BTC in pools fees paid to the pool (to be deducted from the amounts received via the pool)

The pool will receive 25 BTC in block subsidies, but will payout 25.2 BTC to the miner. The extra .2 BTC will ultimately come from the other pool users when the pool lowers the amount it pays per hash.

The reason why blocks are sometimes empty immediately after a block is found, is because miners are blindly accepting that a previously found block is valid, and is mining on top of said block prior to confirming which transactions have been confirmed. Once miners validate a recently found block, they will update their database so that their new block candidates will reflect only valid transactions.

More or less correct. The biggest thing out there is the amount of bad info being discussed.

Many / most of the larger pools have many many many nodes. They all should be in agreement. However, due to many things there may be a period of time that some are updating / processing / working on the block that was just found / just having a bad day so they do not agree.

So they work on an empty block. Once whatever agreement conditions are met, usually within seconds, they work on blocks normally.

People like to scream that they are SPV mining or some other crap. Or have misconfigured nodes. And yes some are. And some are just being paranoid.

No matter how you slice it, even at the speed of light and ultra fast CPUs & drives & networks there will be a delay.

Because, more or less, mining 1 block a week with NO FEES is better then orphaning 1 block every 2 years because that 1 node was having a bit of a slowdown and missed something.

Now some people will chime in that they can have their node validate in under 2 seconds or some crap like that. Yeah, so can mine. But if I have 20 farms each with 2 nodes that all talk back to 2 or 3 servers that make sure that they all agree that then talk back to all the farms. It's going to take a bit. For a small miner that finds blocks once a month it's one thing. For one that is mining 10 blocks a day it's another.

-Dave

I don't see any reason why a miner would work on an empty block (assuming the mempool is not empty), other than that the miner has recently received a new block and is not yet sure which transactions are valid. Even if a pool has fully validated a block they are now mining on top of, before constructing a block with transactions, they will need to update their list of unspent outputs and compare this to inputs in unconfirmed transactions. So even if a miner is not SPV mining, there will still be a period of time in which they are mining an empty block.

I believe that most pools probably have direct connections to other major pools in which the pools send newly found blocks directly to eachother so they can save seconds in the time it takes for a block to propagate so the pool can start working on a new block earlier. I don't doubt that pools also have multiple other nodes that are not publicly known so to prevent the pool from being the subject of a sybil attack, or not receiving a block because a miner did not send said block via this private network.

The only reason why two nodes would not be in agreement is if one node received a block but another did not. It would not make sense to mine an empty block if you are not mining on top of a newly found block because you have already validated that the previous block was valid.

[DaveF]

...
The only reason why two nodes would not be in agreement is if one node received a block but another did not. It would not make sense to mine an empty block if you are not mining on top of a newly found block because you have already validated that the previous block was valid.

Remember dozens of farms and mining nodes spread out all over. You really want them to be in sync. Mostly because they are paying PPS.
Keep in mind in a PPLNS pool an orphan sucks but it costs the pool their fee and not much more.

For the big Chinese pools that tend to put out the most 0 transaction blocks, their miners are PPS so if the pool looses a block it costs them money because the miners get paid no matter what.

So you submit the block with 0 transactions and get the BTC because not doing it costs you money.

-Dave 

[PrimeNumber7]

...
The only reason why two nodes would not be in agreement is if one node received a block but another did not. It would not make sense to mine an empty block if you are not mining on top of a newly found block because you have already validated that the previous block was valid.

Remember dozens of farms and mining nodes spread out all over. You really want them to be in sync. Mostly because they are paying PPS.
Keep in mind in a PPLNS pool an orphan sucks but it costs the pool their fee and not much more.

For the big Chinese pools that tend to put out the most 0 transaction blocks, their miners are PPS so if the pool looses a block it costs them money because the miners get paid no matter what.

So you submit the block with 0 transactions and get the BTC because not doing it costs you money.

-Dave 

Think about why two nodes belonging to a miner might be out of sync.

One possibility is that one node thinks the most recent block is 692827, and this block's hash is 0000000000000000000aa...5d17c6d2 while another node thinks the most recent block is 692827 and that block's hash is 0000...c5d17d6f3. Assuming both blocks are valid, the miner would need to make a decision as to which block they are going to mine on top of, and assuming each block has a different set of transactions, including zero transactions in a block on top of either block will not prevent any new block from being invalid.

Another possibility is that one node thinks the most recent block is 692827, and this block's hash is 0000000000000000000aa...5d17c6d2 while another node thinks the most recent block is 692826 and that block's hash is 000000000000000000011...0025b5c48. If this was the case, one of the nodes simply has not yet received block 692827, and assuming the miner can trust the block is valid, it can mine on top of it with transactions if it has updated its list of unspent outputs. If the miner were to start mining an empty block in this scenario, it would need to choose which block to mine on top of -- mining on top of 826 would likely result in an orphaned block, while mining on top of 827 would result in the block being accepted provided it is otherwise valid and is propagated prior to any other miner propagating their own block on top of 827.

In both scenarios, the miner does not actually reduce the risk of having their found block be invalid. If anything, a miner with two nodes can start mining on top of a newly found block more quickly, as it can start doing so once the node receives (and validates) a block.

The only scenario in which mining an empty block would be +EV would be if the miner has received a new block, and has not yet updated their list of unspent outputs, so it cannot know which transactions are valid that can be included in a new block. One might also argue that mining an empty block would be +EV if the transaction fees are very low, but that is not the case currently. I noted above that a miner might validate a block prior to updating the unspent output table to allow it to start mining on top of a new block more quickly, so mining an empty block does not necessarily mean it is SPV mining.