Blocks are [not] full. What's the plan?

[go1111111]

This is TOTALLY useless. If the content of the junk is dynamic but deterministic (e.g. repeatedly hashing the last block), miners don't need to transfer the junk because everyone know the content. If the content is unspecified, all miners will fill it with 0s. So, again, they don't need to transfer the junk because everyone know the content.

The point is that any block with size less than the minimum size would be disallowed by the protocol. So it wouldn't matter if all the other nodes knew what the junk values would be.

[solex]

This is TOTALLY useless. If the content of the junk is dynamic but deterministic (e.g. repeatedly hashing the last block), miners don't need to transfer the junk because everyone know the content. If the content is unspecified, all miners will fill it with 0s. So, again, they don't need to transfer the junk because everyone know the content.

The point is that any block with size less than the minimum size would be disallowed by the protocol. So it wouldn't matter if all the other nodes knew what the junk values would be.

Minimum block sizes don't work as miners can pad them with transactions between their own addresses.

[jl2012]

This is TOTALLY useless. If the content of the junk is dynamic but deterministic (e.g. repeatedly hashing the last block), miners don't need to transfer the junk because everyone know the content. If the content is unspecified, all miners will fill it with 0s. So, again, they don't need to transfer the junk because everyone know the content.

The point is that any block with size less than the minimum size would be disallowed by the protocol. So it wouldn't matter if all the other nodes knew what the junk values would be.

A block of 1MB does not mean you need 1M bandwidth to transmit. If the junk is deterministic (and it will always be), no one will ever need to waste bandwidth to transmit the junk. So we are going back to the current system (i.e. bandwidth usage is proportional to the total transaction size. You save bandwidth by including less transactions)

Think carefully before you reply.

[Cryddit]

The point is that any block with size less than the minimum size would be disallowed by the protocol. So it wouldn't matter if all the other nodes knew what the junk values would be.

Doesn't matter.  If all the other nodes know what the junk values will be, then the other nodes will reconstruct the block (at the right size, with junk values) right after the block is transmitted to them (at the wrong size, without junk values). 

[go1111111]

Minimum block sizes don't work as miners can pad them with transactions between their own addresses.

That's fine though -- if a miner wants to hit the minimum size value by including transactions to themselves instead of junk, then that still neutralizes any advantage they could have gotten from broadcasting a small block.

A block of 1MB does not mean you need 1M bandwidth to transmit. If the junk is deterministic (and it will always be), no one will ever need to waste bandwidth to transmit the junk. So we are going back to the current system (i.e. bandwidth usage is proportional to the total transaction size. You save bandwidth by including less transactions)

It sounds like you are suggesting that miners will collude with each other to fix each other's invalid blocks. If that didn't happen, that any compression that a miner did to get his block under the minimum size would just result in his block being rejected.

Here's one potential argument against that: individuals running full nodes who just want everyone to play by the rules have no incentive to help miners cheat, so when their Bitcoin-QT software sees a small block get broadcast, then even if it is technically possible for them to 'fix' the block to make it larger, they don't want to, so they continue to wait for a larger block. Any cheating miners who start working off of the too-small block won't get any blocks they find accepted by users, because the cheating miners are building on a too-small block. So some corrupt miners can create their own fork of the blockchain if they want, which doesn't respect the min-blocksize protocol change, but no one will care because users will be using the chain that adheres to the protocol.

And here's a counter-argument suggesting that you and Cryddit are right: a group of cheating miners could do two things when they find a block: first, immediately broadcast the small/cheating version of the block. Then immediately broadcast the larger version of the block. Any miners who receive a cheating block will know that a valid block is probably going to follow very soon, so they fix the cheating block to make it valid, and start working off of it (otherwise they'd be worse off than other miners who did this). When they finally get the valid block, they think "yeah, I knew this block was going to come, I'm already working on it." When users of Bitcoin-QT get the cheating block, they will reject it, but they'll soon get the corresponding large block and accept it.

Did I miss anything?

Have the core developers thought much about if there is some clever trick to make it not work to broadcast an initial cheating block and then later broadcast a valid block?

[jl2012]

It sounds like you are suggesting that miners will collude with each other to fix each other's invalid blocks. If that didn't happen, that any compression that a miner did to get his block under the minimum size would just result in his block being rejected.

It must happen because everyone want to save bandwidth. Those miners who refuse to collude will simply have higher chance to get orphaned so they will collude eventually.

Here's one potential argument against that: individuals running full nodes who just want everyone to play by the rules have no incentive to help miners cheat, so when their Bitcoin-QT software sees a small block get broadcast, then even if it is technically possible for them to 'fix' the block to make it larger, they don't want to, so they continue to wait for a larger block. Any cheating miners who start working off of the too-small block won't get any blocks they find accepted by users, because the cheating miners are building on a too-small block. So some corrupt miners can create their own fork of the blockchain if they want, which doesn't respect the min-blocksize protocol change, but no one will care because users will be using the chain that adheres to the protocol.

And here's a counter-argument suggesting that you and Cryddit are right: a group of cheating miners could do two things when they find a block: first, immediately broadcast the small/cheating version of the block. Then immediately broadcast the larger version of the block. Any miners who receive a cheating block will know that a valid block is probably going to follow very soon, so they fix the cheating block to make it valid, and start working off of it (otherwise they'd be worse off than other miners who did this). When they finally get the valid block, they think "yeah, I knew this block was going to come, I'm already working on it." When users of Bitcoin-QT get the cheating block, they will reject it, but they'll soon get the corresponding large block and accept it.

Did I miss anything?

Have the core developers thought much about if there is some clever trick to make it not work to broadcast an initial cheating block and then later broadcast a valid block?

Non-mining full nodes have no stake in this issue. All you need is one single full-node to fix cheating blocks and broadcast the junk-padded blocks, then all full-nodes will accept the chain. Eventually, non-mining full-nodes will also accept cheating blocks because, again, no one wants to waste bandwidth. Non-mining full-nodes will also keep the cheating blocks because, yet again, no one wants to waste disk space. Cheating blocks will become the new standard block.

The whole idea of padding a block with junk is flawed. It couldn't be fixed.

[Peter Todd]

The whole idea of padding a block with junk is flawed. It couldn't be fixed.

It actually raises a really interesting theoretical cryptography question: Is it possible to create a long string (the padding bytes) such that you can prove it was not possible to derive the string from a smaller secondary string? (AKA a trapdoor)

You can probably come up with a scheme where the second string was some kind of secret, such that knowledge of it could be exploited, perhaps to steal the value of some fidelity bond. For instance you could computer the padding bytes as H(secret || i) with i in (0, n) and secret being some ECC privkey for a valuable txout; giving the privkey to other miners is obviously risky.

The problem is if anything I think that actually encourages centralization: I can safely give a small number of other mining pools that privkey if we have a legal agreement to only use it for the intended purpose. If my funds go missing, I have a pretty good idea who did it and can get the lawyers involved. The smaller the number of pools, the more powerful and enforcable this mechanisms is; with two pools I have 100% certainty who defrauded me. Unfortunately that's the exact opposite of what the padding idea is trying to accomplish...

[jl2012]

The whole idea of padding a block with junk is flawed. It couldn't be fixed.

It actually raises a really interesting theoretical cryptography question: Is it possible to create a long string (the padding bytes) such that you can prove it was not possible to derive the string from a smaller secondary string? (AKA a trapdoor)

You can probably come up with a scheme where the second string was some kind of secret, such that knowledge of it could be exploited, perhaps to steal the value of some fidelity bond. For instance you could computer the padding bytes as H(secret || i) with i in (0, n) and secret being some ECC privkey for a valuable txout; giving the privkey to other miners is obviously risky.

The problem is if anything I think that actually encourages centralization: I can safely give a small number of other mining pools that privkey if we have a legal agreement to only use it for the intended purpose. If my funds go missing, I have a pretty good idea who did it and can get the lawyers involved. The smaller the number of pools, the more powerful and enforcable this mechanisms is; with two pools I have 100% certainty who defrauded me. Unfortunately that's the exact opposite of what the padding idea is trying to accomplish...

Interesting. However, even you have 100% certainty who defrauded you, this is not a conclusive evidence. It is equally probable that you are the one who stole the bond. Of course you can make it a 2-of-2 multisig scheme so the signature of the other pool is needed to steal the bond. However, this would violate the original purpose of having the fidelity bond, which should allow ANYONE to steal the bond by knowing the secret key.

[Peter Todd]

Interesting. However, even you have 100% certainty who defrauded you, this is not a conclusive evidence. It is equally probable that you are the one who stole the bond. Of course you can make it a 2-of-2 multisig scheme so the signature of the other pool is needed to steal the bond. However, this would violate the original purpose of having the fidelity bond, which should allow ANYONE to steal the bond by knowing the secret key.

Note, when I said "I" that was to mean "I the pool operator" - hopefully they know what they themselves are doing! Of course, obviously it's not really 100% once you take into account that maybe I've actually had my server compromised, but it's certainly a high enough % that the idea of fidelity-bonded padding doesn't work in practice between largish pools.

[jl2012]

I still believe counting the total bitcoin-day-destroyed is the most practical way to address this issue. In this case empty blocks are disadvantaged. I guess Satoshi thought the same? How this could be exploited?

[anth0ny]

This is TOTALLY useless. If the content of the junk is dynamic but deterministic (e.g. repeatedly hashing the last block), miners don't need to transfer the junk because everyone know the content. If the content is unspecified, all miners will fill it with 0s. So, again, they don't need to transfer the junk because everyone know the content.

The point is that any block with size less than the minimum size would be disallowed by the protocol. So it wouldn't matter if all the other nodes knew what the junk values would be.

Instead of actually sending junk data why not just sleep for the time it would have taken to receive the junk data?

[anth0ny]

I still believe counting the total bitcoin-day-destroyed is the most practical way to address this issue. In this case empty blocks are disadvantaged. I guess Satoshi thought the same? How this could be exploited?

Disadvantaged how? If later blocks take precedence over earlier blocks, because they have more bitcoin-days-destroyed, it makes it easier for someone who controls lots of bitcoin-days (like the FBI) to mount an attack.

[jl2012]

I still believe counting the total bitcoin-day-destroyed is the most practical way to address this issue. In this case empty blocks are disadvantaged. I guess Satoshi thought the same? How this could be exploited?

Disadvantaged how? If later blocks take precedence over earlier blocks, because they have more bitcoin-days-destroyed, it makes it easier for someone who controls lots of bitcoin-days (like the FBI) to mount an attack.

The longest chain must always win. In case there are 2 or more branches with same length, the one with more bitcoin-days-destroyed will win.

[anth0ny]

I still believe counting the total bitcoin-day-destroyed is the most practical way to address this issue. In this case empty blocks are disadvantaged. I guess Satoshi thought the same? How this could be exploited?

Disadvantaged how? If later blocks take precedence over earlier blocks, because they have more bitcoin-days-destroyed, it makes it easier for someone who controls lots of bitcoin-days (like the FBI) to mount an attack.

The longest chain must always win. In case there are 2 or more branches with same length, the one with more bitcoin-days-destroyed will win.

So if you have a block with a lot of bitcoin-days-destroyed, you wait to release it while trying to build on it. "Selfish mining" where you win nearly 100% of the races, without the need to mount a Sybil attack.

The only tweak to "first received longest chain wins" that I can see being appropriate is to delay less than full blocks for the amount of time it would have taken to send a full block.

That said, there are other, bigger changes, which might work better. What if the block headers are propagated immediately, and the winner becomes based on the first header seen, so long as the entire block corresponding to that header is seen within X seconds (maybe 30 seconds)?

An even bigger change which is worth considering is to propagate blocks in erasure-coded pieces over UDP, allowing peers to download pieces from multiple peers at once a la bittorrent. (This could also be done using TCP, making some of the flow-control issues easier, but at the expense of significant inefficiency and rendering multicasting out of the question. Ideally I think I'd move everything or almost everything to UDP, with TCP being a fallback for those behind restrictive firewalls.)

[zebedee]

Agreed.  Hopefully pools and major solo operators can see that their long term profitability is based MORE on the growth of Bitcoin than the short term orphan cost.   Hopefully large pools are aware they don't need universal support.  If 70% of hashpower agrees to produce larger blocks (say 500KB avg) for the good of the network.  It cuts the orphan cost by 70%.   Still I think the orphan cost does highlight the reality that miners are going to be increasingly reluctant to devote more space to free tx.  This is something users will have to come to grips with.   Including a fee and waiting hours or days is unacceptable (although it is often for non-fee reasons) however no fee tx should be considered charity by users and if someone does you an act of charity in an hour, or day, or week well you got what you paid for.  The default behavior of most clients should probably be changed to include the "min fee" on ALL txs not just low priority ones.  If users want to they could change this but they should be warned "Including no tx fee may result in delayed confirmation times".  Enforcement for relaying at node level should still only be on low priority. 

It is somewhat ironic that this is more of an issue due to the higher exchange rate.  The min fee on low priority tx was lowered due to rising exchange rate.  Today 3.3 mBTC is ~$1.50.  Ouch.  However if Bitcoins were worth less it would be less of a cost.  Since Bitcoin is often used as a proxy for USD a 5 mBTC fee (which more than covers orphan costs) is more viable at a lower exchange rate.

I love this thread.  Only 1 year ago the likes of Mister Bigg were going nuts saying miners would create humungous blocks full of 1-satoshi paying transactions, because there was no reason not to.  Therefore block size shouldn't be lifted.

Now people are moaning block size isn't increasing fast enough.

There's no better evidence that a block size limit is no longer necessary - no-one apart from the odd Eligius block is even approaching 1MB.  Miners are forgoing the fees.  Just scrap the cap.  No-one is forcing miners to build on some 4MB block full of spam transactions; they can always ignore it and teach the spammer a lesson.  Free markets will sort it out.

And with 1 BTC worth $800 or more, screwing around is no longer cheap.

[DeathAndTaxes]

Agreed.  Hopefully pools and major solo operators can see that their long term profitability is based MORE on the growth of Bitcoin than the short term orphan cost.   Hopefully large pools are aware they don't need universal support.  If 70% of hashpower agrees to produce larger blocks (say 500KB avg) for the good of the network.  It cuts the orphan cost by 70%.   Still I think the orphan cost does highlight the reality that miners are going to be increasingly reluctant to devote more space to free tx.  This is something users will have to come to grips with.   Including a fee and waiting hours or days is unacceptable (although it is often for non-fee reasons) however no fee tx should be considered charity by users and if someone does you an act of charity in an hour, or day, or week well you got what you paid for.  The default behavior of most clients should probably be changed to include the "min fee" on ALL txs not just low priority ones.  If users want to they could change this but they should be warned "Including no tx fee may result in delayed confirmation times".  Enforcement for relaying at node level should still only be on low priority.  

It is somewhat ironic that this is more of an issue due to the higher exchange rate.  The min fee on low priority tx was lowered due to rising exchange rate.  Today 3.3 mBTC is ~$1.50.  Ouch.  However if Bitcoins were worth less it would be less of a cost.  Since Bitcoin is often used as a proxy for USD a 5 mBTC fee (which more than covers orphan costs) is more viable at a lower exchange rate.

I love this thread.  Only 1 year ago the likes of Mister Bigg were going nuts saying miners would create humungous blocks full of 1-satoshi paying transactions, because there was no reason not to.  Therefore block size shouldn't be lifted.

Now people are moaning block size isn't increasing fast enough.

There's no better evidence that a block size limit is no longer necessary - no-one apart from the odd Eligius block is even approaching 1MB.  Miners are forgoing the fees.  Just scrap the cap.  No-one is forcing miners to build on some 4MB block full of spam transactions; they can always ignore it and teach the spammer a lesson.  Free markets will sort it out.

And with 1 BTC worth $800 or more, screwing around is no longer cheap.

I think a cap is still useful.  The cost to massively bloat the blockchain remains a viable threat, and at a tiny fraction of what it would cost to 51% the network.  Bitcoin has a somewhat unique cost vs benefit scenario.  There is actually little direct cost to put a tx in a block however the true cost is the total storage, bandwidth, and memory requirements of all the full nodes combined and most nodes are not miners.

While miners who are economically motivated are unlikely to do something stupid (like create a 1 GB block full of millions of txs with a 1 satoshi fee), an attack may not be economically motivated and at this point dumping hundreds of thousands of GB of additional blockchain size could slow adoption.

That being said I am much more in favor of raising the cap possibly to 10 MB (or even 5MB) to buy us the community the time to fully analyze all the possible implications of future block sizes (no cap, floating cap, high fixed cap, etc).  

[solex]

I think a cap is still useful.  [...] I am much more in favor of raising the cap possibly to 10 MB (or even 5MB) to buy us the community the time to fully analyze all the possible implications of future block sizes (no cap, floating cap, high fixed cap, etc). 

I still like this...

If unlimited size is too worrying then the simplest safety net would be max block size = 10x median block size during previous 2016 blocks or previous difficulty period.

[niniyo]

What's wrong with the minimum block size idea?  If dummy data doesn't work then make it require real transactions.

Miners will only fill blocks with dummy transactions if they don't have enough available transactions in their mempool.  Otherwise they'll fill it with transactions that earn them fees.  If we can expect a certain transaction rate on the bitcoin network then the min block size could be set accordingly and so we shouldn't see much use of dummy transactions.

[jl2012]

What's wrong with the minimum block size idea?  If dummy data doesn't work then make it require real transactions.

Miners will only fill blocks with dummy transactions if they don't have enough available transactions in their mempool.  Otherwise they'll fill it with transactions that earn them fees.  If we can expect a certain transaction rate on the bitcoin network then the min block size could be set accordingly and so we shouldn't see much use of dummy transactions.

No, it won't work. Miners will simply create an extra output in the reward transaction, sending 0 bitcoin to OP_TRUE. They will then use it as an input for another OP_TRUE output, and repeat this process. Since the process is totally deterministic, miners won't need to broadcast these dummy transactions. Other miners will recreate the full block locally. So we go back to the current system: with less (real) transactions, the orphan rate is lower.

[anth0ny]

What's wrong with the minimum block size idea?  If dummy data doesn't work then make it require real transactions.

Miners will only fill blocks with dummy transactions if they don't have enough available transactions in their mempool.  Otherwise they'll fill it with transactions that earn them fees.  If we can expect a certain transaction rate on the bitcoin network then the min block size could be set accordingly and so we shouldn't see much use of dummy transactions.

No, it won't work. Miners will simply create an extra output in the reward transaction, sending 0 bitcoin to OP_TRUE. They will then use it as an input for another OP_TRUE output, and repeat this process. Since the process is totally deterministic, miners won't need to broadcast these dummy transactions. Other miners will recreate the full block locally. So we go back to the current system: with less (real) transactions, the orphan rate is lower.

Seems like quite a lot of software engineering work with relatively little benefit. If you really want to get together with other miners to speed up transaction propagation there are much better solutions. If you know the identity of the miner you can always start working on the new block as soon as you see the header, and check the validity of it a few seconds later when you receive the whole block. If the block winds up being invalid then you ignore those few seconds of work, and punish the miner who sent you the bad block. Alternatively, a trusted third party can verify blocks and sign the headers. Miners could send blocks to this third party ahead of time, so at the time the block is actually found very little needs to be transferred.

That said, introducing a minimum block size seems to me to be a drastic change, and I haven't seen enough evidence of a drastic problem to implement it. I think the focus at this point should be on raising the maximum block size and speeding up propagation.