Block size increase proposal: Consensus based on mempool

[Schnibble]

As we know, the 1MB limit on the block size was set to counter spam. I want to suggest an alternative solution to this problem.

Let's consider all transactions with a fee below some fixed value (say, 20 satoshis/byte) as spam. In this case, the problem can be solved very easily. Just remove the limit on the block size and prevent miners to include transactions(their own too) with a fee below this value into the block… Actually, let's not prohibit all these transactions, but give some space for them (let it be 5%, tentatively):

• the limit on the block size should be removed^*
• full nodes should temporarily reject^* blocks that consist more than 5% of transactions considered as undesirable (received less than 15 seconds ago | with a fee below 20 satoshis/byte)
• miners should follow the same rules, but in addition, they have to relay and then hold transactions at least for 15-20 seconds before including them into the block
• minTxRelayFee parameter becomes part of the consensus and cannot be higher than 20 satoshis/byte, otherwise, nodes won't be able to check the block for spam

The main idea is that transaction must get into the most nodes mempools and stay there for some time before it can be included in the block. Thus miners are unable to include their own transactions for free and have to broadcast them over the network as all do.

By reserving 5% of space for undesirable transactions we make the condition softer. This is a very large reserve to ensure that nodes always come to consensus. Even if mined block obtained before some transactions that included to it.

* Anyway, the best chain is selected by the amount of work in it. Thus rejected undesirable blocks potentially could be included into the blockchain, but only if most of the miners (>50%) break the rules and accept these blocks with spam. Therefore, during the evaluation of PoW for a chain, nodes and miners shouldn't count these undesirable blocks until they are followed by two or three normal blocks. For the same reason, it makes sense to limit one-time increase of block size in comparison to the average size of the previous several blocks, say not more than twice.

[1714071334]

1714071334

[DannyHamilton]

• miners should follow the same rules, but in addition they need to hold transactions at least for 15-20 seconds before including them into the block

It is impossible to know with any level of certainty how long a miner (or pool) has had a transaction in their mempool.  Therefore, it is impossible to enforce a rule that requires them to have a transaction in their mempool for a specified amount of time.

• full nodes should reject blocks that consists more than 5% of transactions considered as non-standard (using non-standard scripts | being in the mempool less than 15 seconds | with a fee below 20 satoshies/byte)

If my node receives new block "A" which builds on the current best chain but includes more than 5% of transactions that have been in my mempool less than 15 seconds, and then half a second later receives block "B" which builds on block "A" which follows your rules, what should it do?

• Accept the invalid block "A" so that it can accept valid block "B"?
• Reject invalid block "A", and therefore need to reject valid block "B"?

The main idea is that transaction must get into the most nodes mempools and stay there for some time before it can be included into the block.

Have you figured out a reliable way to know what most nodes have in their mempool?  Right now, a node only knows what it has in its own mempool, and it has no way of knowing if its missing anything or if any other node has any of those same transactions in their mempool.

[achow101]

* Anyway, the best chain is selected by amount of work in it. Thus rejected blocks potentially could be included into the blockchain, but only if most of miners (>50%) break the rules and accept them. Therefore, during the evaluation of PoW for a chain, nodes shouldn't count these blocks until they are followed by two or three valid blocks. For the same reason, it makes sense to limit one-time increase of block size in comparison to the average size of the previous several blocks, say not more than twice.

No. This is a horrible idea. It gives miners absolute control over the consensus rules. The consensus rules will stop being consensus rules and suddenly become Miners Want This Rules. Making such a drastic change will allow miners to change whatever rules whenever they want. That is completely antithetical to Bitcoin and completely defeats the purpose of full nodes. The point of running a full node is to ensure that miners follow the consensus rules, not just to have the entire blockchain. Your proposal completely removes that check on miners' power.

By your proposal, miners could decide that they can make a transaction which spends all of the Bitcoin back to themselves, put it in a block, and make a ton of money. Or miners can remove the block subsidy schedule and just make as much Bitcoin as they want out of thin air, and everyone will follow. This idea is just plain stupid and simply hands over all control of Bitcoin to the miners, instead of splitting it between users and miners.

[Schnibble]

It is impossible to know with any level of certainty how long a miner (or pool) has had a transaction in their mempool.  Therefore, it is impossible to enforce a rule that requires them to have a transaction in their mempool for a specified amount of time.

Obviously, miners as well as other full nodes must relay all standard transactions before including them into the block. Do you think that small transaction can propagate over the network 5 seconds slower than full block does? Not a problem - miners can adjust this delay as much as needed.

If my node receives new block "A" which builds on the current best chain but includes more than 5% of transactions that have been in my mempool less than 15 seconds, and then half a second later receives block "B" which builds on block "A" which follows your rules, what should it do?

Reject invalid block "A", and therefore reject valid block "B". But if afterwards it also receives valid block "C" which builds on block "B", the chain becomes valid. But this situation is possible only in case of 51% attack because honest miners will do their work on block preceding "A".

Have you figured out a reliable way to know what most nodes have in their mempool?

I suggest to define which transactions should be considered as standard and tighten the protocol rules to make nodes required to relay all these transactions.

[Schnibble]

Your proposal completely removes that check on miners' power

Ok. You just misunderstood me. I've changed "rejected blocks potentially could be included" to "temporary rejected blocks ..." in my first post.
Actually my solution doesn't solve the 51% attack problem. But of course the best chain is selected only when it meets the consensus rules.

The point of running a full node is to ensure that miners follow the consensus rules, not just to have the entire blockchain.

I don't think that full nodes have any significant power. But they can try to continue to reject undesirable blocks more than after 2-3 following blocks.

[jonald_fyookball]

The problem is economic/political, not technical.

Many solutions would be technically feasible and sound,
but there is a faction of Bitcoin that seeks to limit
on chain scaling in whatever form it comes.

[achow101]

Obviously, miners as well as other full nodes must relay all standard transactions before including them into the block. Do you think that small transaction can propagate over the network 5 seconds slower than full block does? Not a problem - miners can adjust this delay as much as needed.

But then how do you know that a transaction in a block was actually relayed beforehand? There is no universal mempool, the mempool varies from node to node. Furthermore, you are assuming that every node has a mempool, but people can run nodes that are blocks only. They would not be able to validate a block under this rule. You are also assuming that all nodes will keep all transactions and receive them at roughly the same time. However you cannot assume that. Many nodes will evict transactions or not accept them if they violate local node policy (which does not have to match the node policies of anyone else). For example, there could be a race condition where a bunch of low fee transactions are evicted just before a block is received by the node so this evicted transactions are no longer in the mempool and the block considered invalid. Furthermore, a newly started node will fail to properly sync because it would consider all blocks invalid (after all, it still has to validate consensus rules) because it has none of the transactions in those blocks in its mempool. A block that is found very shortly after a node starts up would also be rejected as that node's mempool has not filled up with transactions yet.

Reject invalid block "A", and therefore reject valid block "B". But if afterwards it also receives valid block "C" which builds on block "B", the chain becomes valid. But this situation is possible only in case of 51% attack because honest miners will do their work on block preceding "A".

Suppose the miner who found A is super lucky, and found B, then C very quickly. Now all miners will be using the chain including the invalid block, and that invalid block could be maliciously crafted to create transactions which allows the miner who found the block to spend the Bitcoin.

You are also assuming that all miners are fully validating every single block before they build on top of it. Unfortunately, we know for a fact that this is absolutely not the case. Many miners are SPV mining; they will begin mining on top of a block before it is fully validated. With SPV mining, it is very possible that miners will end up mining on top of an invalid block and thus extending the chain for that block and find enough blocks so that its is valid.

I suggest to define which transactions should be considered as standard and tighten the protocol rules to make nodes required to relay all these transactions.

That does nothing to help with figuring out what transactions are in nodes' mempools. Standardness  rules are not consensus rules, they are local node policy rules. Anyone can change their local node policy. Even so, there are transactions which are standard, but still rejected by other node policies. Relying on standardness rules is not enough to say that you know what transactions are in everyone's mempools.

Ok. You just misunderstood me. I've changed "rejected blocks potentially could be included" to "temporary rejected blocks ..." in my first post.
Actually my solution doesn't solve the 51% attack problem. But of course the best chain is selected only when it meets the consensus rules.

The miners are then redefining the consensus rules still. They can do so every 3 blocks; every 3 blocks can basically have whatever consensus rules it wants then so long as everyone mines a block on top of the invalid one, which is entirely possible given that many miners do SPV mining.

I don't think that full nodes have any significant power. But they can try to continue to reject undesirable blocks more than after 2-3 following blocks.

But miners shouldn't have full and absolute power over everything. Full nodes prevent miners from changing the consensus rules willy-nilly. They provide an incentive for miners to not produce invalid blocks. With your proposal, they now have an incentive to produce invalid blocks because in that invalid block they can make a ton of money and have a fairly high likelihood of making that invalid block a valid accepted block.

[Schnibble]

people can run nodes that are blocks only. They would not be able to validate a block under this rule.

These people aren't running full nodes. They would not be able to temporarily reject undesirable block but still can validate it.
In a good way this node should be considered as misbehaving node unless it reported to all connected nodes that have an intention to evict transactions.

You are also assuming that all nodes will keep all transactions

I don't assume that. I've proposed to make them required to keep and relay all transactions.

and receive them at roughly the same time

I don't assume that. Miners can adjust their delay parameter as much as needed.

there could be a race condition where a bunch of low fee transactions are evicted just before a block is received by the node so this evicted transactions are no longer in the mempool

minTxRelayFee parameter becomes part of the consensus and cannot be higher than 20 satoshies/byte

a newly started node will fail to properly sync because it would consider all blocks invalid

The chain becomes valid anyway, when it receives two or three valid blocks. Also we can bypass this rule for some time after the launch.

Suppose the miner who found A is super lucky, and found B, then C very quickly. Now all miners will be using the chain including the invalid block

Not invalid, just undesirable.

and that invalid block could be maliciously crafted to create transactions which allows the miner who found the block to spend the Bitcoin

No. The worst possible thing that could occur is acceptance the undesirable block consisting of spam.

With SPV mining, it is very possible that miners will end up mining on top of an invalid block and thus extending the chain for that block and find enough blocks so that its is valid

Nothing serious. Besides, nobody will mine on top of invalid (or undesirable) block unless 51% attack is performed...

Standardness  rules are not consensus rules, they are local node policy rules.

Why do you think so? What about 1MB block size limit rule?

Anyone can change their local node policy

If node violates some mandatory rule and for example sets minTxRelayFee parameter to 30 sat/byte it should be considered as misbehaving node, unless it reported to all connected nodes that have an intention to evict transactions.

Relying on standardness rules is not enough to say that you know what transactions are in everyone's mempools

It is enough to always come to consensus. Miners just need to adjust their delay parameter.

which is entirely possible given that many miners do SPV mining

My proposal assumes hard fork, so miners/pools/SPV servers will have to update their soft.

With your proposal, they now have an incentive to produce invalid blocks because in that invalid block they can make a ton of money

No. The worst possible thing that could occur is acceptance the undesirable block consisting of spam.

[achow101]

These people aren't running full nodes.

Yes they are. A full node is a node which fully validates every single block and transaction that it receives. Blocksonly nodes are still full nodes, they fully validate every single block and transaction it receives. In fact, as an outside observer, you would absolutely not be able to tell whether someone is in blocksonly mode or not. There is no indication that they are not just a regular node.

They would not be able to temporarily reject undesirable block but still can validate it.
In a good way this node should be considered as misbehaving node unless it reported to all connected nodes that have an intention to evict transactions.

Many people run blocksonly nodes because it significantly reduces the bandwidth requirements for running a full node. You would completely make it impossible for people to run full nodes in a low bandwidth mode such as blocksonly. This is not something that we want to do as some people may not have the resources to run a non-blocksonly full node but also want the security and privacy afforded by a full node.

I don't assume that. I've proposed to make them required to keep and relay all transactions.

How do you enforce that? How do you prevent someone from making a ton of super low fee transactions just to spam up mempools and take up more memory, at some point causing nodes to crash? If you require every single transaction to be kept, then you open up an entirely new attack vector.

I don't assume that. Miners can adjust their delay parameter as much as needed.

What the hell does that mean? As you described it earlier, the "delay parameter" is a consensus rule. You can't just "adjust a consensus rule" as you are proposing without causing invalid blocks.

minTxRelayFee parameter becomes part of the consensus and cannot be higher than 20 satoshies/byte

How do you enforce that? minTxRelayFee will still be part of node policy. How do you enforce that the minTxRelayFee is no higher than 20 sat/byte. Furthermore, suppose someone were to spam up the mempool with transactions that are 20 sat/byte so much that nodes crash because the mempools are too big. Now what? Additionally, Bitcoin Core currently implements a floating minTxRelayFee because it prevents such crashes. It limits the mempool to a certain size and bumps up the minrelayfee as it gets fuller.

Not invalid, just undesirable.

Suppose Miner A was malicious and got lucky. They made a block which instead of awarding them 12.5 BTC as the subsidy, awarded them 125000000000 BTC as the subsidy. Now what? Everyone is just going to follow along that chain and let that miner have the 125000000000 BTC because he just got lucky and other miners were SPV mining?

No. The worst possible thing that could occur is acceptance the undesirable block consisting of spam.

I'm not talking about spam. I'm talking about a miner who awards themselves a massive amount of Bitcoin in the block subsidy. Or creates a bunch of transactions which spends UTXOs that they normally are not allowed to spend thus stealing coins.

The worst possible thing that could occur is acceptance the block consisting of spam. But nobody will mine on top of invalid (or undesirable) blocks unless 51% attack is performed...

With SPV mining, a 51% attack is not necessary. Forks have happened before where SPV miners (which consist of >51% of the network as most large pools do this) have mined on top of invalid blocks and in fact made that chain longer than the valid chain until the pool operators intervened and switched the pool back to the valid chain.

Why do you think so? What about 1MB block size limit rule?

The block size limit is a consensus rule. A standardness rule applies to transactions, and is defined by the IsStandard function in the Bitcoin Core source code. That standardness rules are not the same as consensus rules is not just something I think, but is by definition a local node policy, as defined by the reference implementation. There is no such thing as a standard block, only that there are non-standard transactions, which can still be valid and will be accepted by all full nodes if included in a block.

If node violates some mandatory rule and for example sets minTxRelayFee parameter to 30 sat/byte it should be considered as misbehaving node, unless it reported to all connected nodes that have an intention to evict transactions.

How would you know? How would you enforce this? Local node policies (including standardness rules) are not broadcast among nodes and even if they were, can be easily faked.

It is enough to always come to consensus. Miners just need to adjust their delay parameter.

As I said earlier, standardness rules are not the same as consensus rules.

My proposal assumes hard fork, so miners/pools/SPV servers will have to update their soft.

Regardless of what type of fork this is, SPV mining will still happen and is still a problem. Those two are completely unrelated things.

No. The worst possible thing that could occur is acceptance the undesirable block consisting of spam.

No, the worst possible thing that could happen is the acceptance of a completely invalid block. Except under your proposal, such an invalid block can and will be accepted.

[Schnibble]

Yes they are. A full node is a node which fully validates every single block and transaction that it receives.

In your opinion, node is still full node even if it doesn't relay anything.

Blocksonly nodes are still full nodes

Don't want to argue but I wouldn't call them full nodes. Because the system is not designed to operate on such nodes.

In fact, as an outside observer, you would absolutely not be able to tell whether someone is in blocksonly mode or not.

I'm not sure that we need it. How does it related to the topic? My proposal does not imply that all existing nodes relay transactions. Similarly, miners does not imply that all existing nodes relay blocks.

You would completely make it impossible for people to run full nodes in a low bandwidth mode such as blocksonly.

They can do. I just don't call them full nodes. They give a little advantage for the network.

How do you prevent someone from making a ton of super low fee transactions just to spam up mempools and take up more memory, at some point causing nodes to crash? If you require every single transaction to be kept, then you open up an entirely new attack vector.

Good point. This problem can be solved in different ways.
First of all, nodes aren't required to keep full transactions (TxID will be enough). Also we don't need to keep them forever. I propose that nodes can forget about transactions received over an hour ago. Miners should be careful to include these transactions in block.

What the hell does that mean? As you described it earlier, the "delay parameter" is a consensus rule.

This parameter is a consensus rule:
  full nodes should temporarily reject* blocks that consists more than 5% of transactions considered as undesirable (being in the mempool less than 15 seconds | with a fee below 20 satoshies/byte)

This parameter is adjustable by miners and not a consensus rule:
  miners should follow the same rules, but in addition they need to hold transactions at least for 15-20 seconds before including them into the block

How do you enforce that the minTxRelayFee is no higher than 20 sat/byte

What for? My proposal does not imply that all existing nodes will follow the rules. Do we need to enforce that the MAX_BLOCK_BASE_SIZE is no lower or higher than 1MB?

They made a block which instead of awarding them 12.5 BTC as the subsidy, awarded them 125000000000 BTC as the subsidy. Now what?

All honest nodes will reject it because the best chain is selected only when it meets the consensus rules.

have mined on top of invalid blocks and in fact made that chain longer than the valid chain

How will it happen?

only that there are non-standard transactions, which can still be valid and will be accepted by all full nodes if included in a block

Yes. But they couldn't be redeemed until soft-fork is performed and they become considered as standard.
However, I think you're right, we may not limit non-standard transactions. I've removed this part from my proposal to keep things simple.

SPV mining will still happen and is still a problem.

What's wrong with SPV mining?

Except under your proposal, such an invalid block can and will be accepted.

How will it happen?

[achow101]

I'm not sure that we need it. How does it related to the topic? My proposal does not imply that all existing nodes relay transactions. Similarly, miners does not imply that all existing nodes relay blocks.

Your proposal requires that all nodes receive and store transactions in order to verify that a block contains "desireable" transactions. Yet blocksonly nodes do not receive and store transactions, thus they cannot receive, verify, and relay blocks.

Even if you don't call blocksonly nodes full nodes, you are still preventing people who want to have the security of a full node (and they are still getting that with a blocksonly node) but don't have enough bandwidth for a normal full node from being able to use their wallets.

Good point. This problem can be solved in different ways.
First of all, nodes aren't required to keep full transactions (TxID will be enough). Also we don't need to keep them forever. I propose that nodes can forget about transactions received over an hour ago. Miners should be careful to include these transactions in block.

Then you run into other issues like people paying too low of a transaction fee and thus getting their transaction evicted after the hour is up.

This parameter is a consensus rule:
  full nodes should temporarily reject* blocks that consists more than 5% of transactions considered as undesirable (being in the mempool less than 15 seconds | with a fee below 20 satoshies/byte)

What if a transaction was in a miner's mempool for over 15 seconds but not in a node's mempool for 15 seconds (due to latency)? What if I just restarted my node (so it's mempool is now empty) and then I receive a block? Do I reject it? I don't have any of the transactions in the block in my mempool. This time rule assumes that mempools will be in sync with each other, and they simply are not.

I know you are going to say "miners will adjust their delay" but miners can't just "adjust their delay" to accommodate every node. Again, what if a new node just started up (which miners won't know about anyways) and has no transactions in its mempool? How do you handle that?

All honest nodes will reject it because the best chain is selected only when it meets the consensus rules.
...
How will it happen?
...
How will it happen?

Again, SPV miners cause an issue here, and malicious miners. SPV miners don't verify blocks before they begin building upon them. Given that a majority of the hashrate does this (because mining pools) combined with the malicious miner, they could, with some luck, find the 3 blocks necessary in order to make this invalid block accepted. The blocks would be found before the SPV miners have finished verifying that the invalid block is in fact invalid, and once the three blocks are found, that invalid block is now valid. That is how it would happen.

What's wrong with SPV mining?

SPV mining is also called validationless mining. SPV miners are building blocks on top of blocks which they have not yet verified to be valid. This is a problem because you can end up with a situation where SPV miners have extended a blockchain on top of the invalid block enough so that the invalid block would be considered valid under your proposal. SPV miners currently consist of a majority of the hashrate, and having SPV miners extend on an invalid block has happened in the past.

[Schnibble]

Your proposal requires that all nodes receive and store transactions in order to verify that a block contains "desireable" transactions

Actually, node requires to be connected at least with one node that relays all desireable transactions.

Yet blocksonly nodes do not receive and store transactions, thus they cannot receive, verify, and relay blocks

They would not be able to temporarily reject undesirable (possibly orphan) block but still can receive, validate and relay it.

Even if you don't call blocksonly nodes full nodes, you are still preventing people who want to have the security of a full node (and they are still getting that with a blocksonly node) but don't have enough bandwidth for a normal full node from being able to use their wallets.

They still can use their wallets in blocksonly mode. But what's the point if it doesn't receive transactions?

Then you run into other issues like people paying too low of a transaction fee and thus getting their transaction evicted after the hour is up.

It's not a problem. Miners could keep and rebroadcast these transactions shortly before including them into the block.

What if I just restarted my node (so it's mempool is now empty) and then I receive a block? Do I reject it?

The chain becomes valid anyway, when it receives two or three valid blocks. Also we can simply bypass this rule for some time after the launch. Also we can add a new type of message for the purpose of mempool synchronization between the nodes.

miners can't just "adjust their delay" to accommodate every node

Yes. But it is not a miners' problem. Actually, I doubt whether node even without additional delay can receive full block before small transaction. Furthermore, given that transactions are relayed before.

Given that a majority of the hashrate does this (because mining pools) combined with the malicious miner, they could, with some luck, find the 3 blocks necessary in order to make this invalid block accepted.

Malicious miner will generate a lot of orphan blocks before that happens.

The blocks would be found before the SPV miners have finished verifying that the invalid block is in fact invalid, and once the three blocks are found, that invalid block is now valid

Anyway, the worst possible thing that could occur is acceptance the undesirable block consisting of spam. The best chain is selected only when it meets the consensus rules.

[Quickseller]

You are also assuming that all miners are fully validating every single block before they build on top of it. Unfortunately, we know for a fact that this is absolutely not the case. Many miners are SPV mining; they will begin mining on top of a block before it is fully validated. With SPV mining, it is very possible that miners will end up mining on top of an invalid block and thus extending the chain for that block and find enough blocks so that its is valid.

The miners who engage in "SPV" mining will only mine on top of an unvalidated block for as long as it takes them to validate the block. These miners are relying on the valid economic assumption that it is not economical to build/broadcast an invalid block, and it will cost a troll miner much more to broadcast an invalid block than it would cost the rest of the network to build upon an invalid block for only as long as it takes to determine that said block is invalid. It would cost a troll miner roughly 13.5-14BTC to broadcast an invalid block, while it would cost 90% of the network collectively between ~.6BTC-~.62BTC to build on top of that block for the amount of time it takes to figure out it is invalid.

The result of this is greater security as this will reduce the orphan rate.

OP - You are describing something very similar to EC (emergent consensus). It should be important to point out that only certain consensus rules are subject to EC in your proposal, otherwise the miners will be able to make *all* the rules, which is probably not something that is desirable. 

I would also point out that not all nodes' clocks are on the exact same time (when converted to GMT), so it would be difficult to determine if a transaction has in fact been in a miner's mempool for at least 15 seconds prior to broadcasting a block that confirmed said transaction, so I don't think it would be possible for a node that had received a transaction at the exact same time as the miner to validate the miner had the transaction in it's mempool for at least 15 seconds. Also, as achow101 pointed out (somewhat), if a node receives a transaction 5 seconds prior to receiving a block that includes said transaction, they believe said block is invalid -- this is however resolved by your EC clause.

Another problem is that if someone broadcasts a transaction that gets to miner "A" located in China, and a conflicting transaction that gets to miner "B" located in New York, US, then exactly one of those transactions will get rejected by nodes. This will possibly result in a miner creating a block that gets rejected by the rest of the network, and a potentially long orphan race, both of which is undesirable. The winning chain would be determined by EC after the fact.

I think that EC would probably be better for things like long term changes to demand for block space, and the potential for a one-time large block to account for a temporary increase in the need to confirm many transactions. I don't think that EC is appropriate for the minute-to-minute decision regarding which transactions should get included in specific blocks because miners will too frequently not be on the same page.

[achow101]

Actually, node requires to be connected at least with one node that relays all desireable transactions.

Even if they are connected to a node that relays all "desireable" transactions, that does not necessarily mean that the node will have all of those transactions stored somewhere in order to properly validate a block. It needs to be able to lookup that a transaction was in its mempool for at least X seconds, and if it doesn't have a mempool, then this is impossible.

They still can use their wallets in blocksonly mode. But what's the point if it doesn't receive transactions?

A blocksonly wallet still receives transactions when they come in a block, i.e. the transactions have confirmed. You are reducing the security of people running blocksonly nodes as they are trusting that blocks that they receive are valid with regards to the transactions inside of them being in the mempool at least X seconds.

The chain becomes valid anyway, when it receives two or three valid blocks. Also we can simply bypass this rule for some time after the launch.

No, bypassing the rule for some time after the proposal deploys is not going to help at all. New nodes don't just come online after a proposal activates; nodes are being restarted all the time and this rule will force a node which just started to reject the blocks because it doesn't have a populated mempool to begin with.

Yes. But it is not a miners' problem. Actually, I doubt whether node even without additional delay can receive full block before small transaction. Furthermore, given that transactions are relayed before.

A node can most certainly receive a block before transactions are received. There is no stipulation that blocks cannot be sent before transactions (it's asynchronous). A newly started node with NO MEMPOOL is not likely to receive the transactions that are included in a block if the block is found shortly after the node starts up. Furthermore, new nodes don't populate their mempools with the transactions in the mempool's of other nodes. Nodes only relay a transaction once, right after they receive and validate it.

Malicious miner will generate a lot of orphan blocks before that happens.

Even so, it is an attack that is entirely possible and potentially has huge payout.

Anyway, the worst possible thing that could occur is acceptance the undesirable block consisting of spam. The best chain is selected only when it meets the consensus rules.

No, you don't seem to grasp the idea that ALLOWING INVALID BLOCKS MEANS THAT MALICIOUS THINGS CAN BE DONE. Your proposal allows the absolute worst possible thing that can happen in Bitcoin, allowing an invalid block become completely valid. By allowing invalid blocks, you are allowing blocks completely circumvent the consensus rules. What part of that do you not understand? Do you not understand that invalid blocks include more than just being larger than the block size limit?

The miners who engage in "SPV" mining will only mine on top of an unvalidated block for as long as it takes them to validate the block. These miners are relying on the valid economic assumption that it is not economical to build/broadcast an invalid block, and it will cost a troll miner much more to broadcast an invalid block than it would cost the rest of the network to build upon an invalid block for only as long as it takes to determine that said block is invalid. It would cost a troll miner roughly 13.5-14BTC to broadcast an invalid block, while it would cost 90% of the network collectively between ~.6BTC-~.62BTC to build on top of that block for the amount of time it takes to figure out it is invalid.

Not necessarily. As I mentioned earlier, an invalid block can be invalid in many ways. It is possible that a miner could make an invalid block which has a massive payout, such as changing the block subsidy to 1250000000000 BTC for that block. Even though the consensus rule is that coinbases need to have 100+ confirmations before they can be spent, this is an invalid block, so consensus rules go out the window. They can make a block which has that huge reward and make and include a transaction which spends that reward to an exchange address thus once they get their invalid block considered valid, they can cash out and make a ton of money to recover their costs.

OP - You are describing something very similar to EC (emergent consensus). It should be important to point out that only certain consensus rules are subject to EC in your proposal, otherwise the miners will be able to make *all* the rules, which is probably not something that is desirable.

Has OP stipulated that his "temporarily invalid" block thing is only applicable to certain consensus rules? He has not mentioned that explicitly thus far in this conversation, and it was not mentioned in the OP when I first responded to it.

I would also point out that not all nodes' clocks are on the exact same time (when converted to GMT), so it would be difficult to determine if a transaction has in fact been in a miner's mempool for at least 15 seconds prior to broadcasting a block that confirmed said transaction, so I don't think it would be possible for a node that had received a transaction at the exact same time as the miner to validate the miner had the transaction in it's mempool for at least 15 seconds. Also, as achow101 pointed out (somewhat), if a node receives a transaction 5 seconds prior to receiving a block that includes said transaction, they believe said block is invalid -- this is however resolved by your EC clause.

But waiting for the EC clause to take effect takes time, and this case is a usability issue. Suppose you received a transaction, and that transaction is included in a block. But you perceive that block as invalid because some other transaction in that block was received less than 15 seconds before you received the block, so you mark it as invalid. This is can make Bitcoin less easy to use as some users won't know that their transaction has actually confirmed since the wallet considers the block it was confirmed in to be invalid until ~30 minutes after the confirmation. In fact, a miner could perpetually prevent the EC clause from taking effect for a specific node by always broadcasting an old transaction (i.e. one that was discarded by being older than 1 hour, as was mentioned previously) shortly before broadcasting the block including it. You could do this an effectively take nodes offline for a while.

[Quickseller]

The miners who engage in "SPV" mining will only mine on top of an unvalidated block for as long as it takes them to validate the block. These miners are relying on the valid economic assumption that it is not economical to build/broadcast an invalid block, and it will cost a troll miner much more to broadcast an invalid block than it would cost the rest of the network to build upon an invalid block for only as long as it takes to determine that said block is invalid. It would cost a troll miner roughly 13.5-14BTC to broadcast an invalid block, while it would cost 90% of the network collectively between ~.6BTC-~.62BTC to build on top of that block for the amount of time it takes to figure out it is invalid.

Not necessarily. As I mentioned earlier, an invalid block can be invalid in many ways. It is possible that a miner could make an invalid block which has a massive payout, such as changing the block subsidy to 1250000000000 BTC for that block. Even though the consensus rule is that coinbases need to have 100+ confirmations before they can be spent, this is an invalid block, so consensus rules go out the window. They can make a block which has that huge reward and make and include a transaction which spends that reward to an exchange address thus once they get their invalid block considered valid, they can cash out and make a ton of money to recover their costs.

I was speaking strictly in terms of how SPV mining works today, under today's rules.

OP - You are describing something very similar to EC (emergent consensus). It should be important to point out that only certain consensus rules are subject to EC in your proposal, otherwise the miners will be able to make *all* the rules, which is probably not something that is desirable.

Has OP stipulated that his "temporarily invalid" block thing is only applicable to certain consensus rules? He has not mentioned that explicitly thus far in this conversation, and it was not mentioned in the OP when I first responded to it.

No, he has not, however without EC being applicable to only certain rules, the OP's proposal would have too many technical flaws to even be considered.
 

I would also point out that not all nodes' clocks are on the exact same time (when converted to GMT), so it would be difficult to determine if a transaction has in fact been in a miner's mempool for at least 15 seconds prior to broadcasting a block that confirmed said transaction, so I don't think it would be possible for a node that had received a transaction at the exact same time as the miner to validate the miner had the transaction in it's mempool for at least 15 seconds. Also, as achow101 pointed out (somewhat), if a node receives a transaction 5 seconds prior to receiving a block that includes said transaction, they believe said block is invalid -- this is however resolved by your EC clause.

But waiting for the EC clause to take effect takes time, and this case is a usability issue. Suppose you received a transaction, and that transaction is included in a block. But you perceive that block as invalid because some other transaction in that block was received less than 15 seconds before you received the block, so you mark it as invalid. This is can make Bitcoin less easy to use as some users won't know that their transaction has actually confirmed since the wallet considers the block it was confirmed in to be invalid until ~30 minutes after the confirmation. In fact, a miner could perpetually prevent the EC clause from taking effect for a specific node by always broadcasting an old transaction (i.e. one that was discarded by being older than 1 hour, as was mentioned previously) shortly before broadcasting the block including it. You could do this an effectively take nodes offline for a while.

Yes, in the case of the OP's proposal, to use EC would result in chaos because nodes would be broadcasting potentially conflicting transactions, and more importantly, miners would be building on top of potentially several blockchains, which would lead to frequent reorgs. You could even have multiple reorgs within a reorg.

I would be more comfortable with a consensus rule change so that transactions must have a tx fee rate above a certain amount (with no exceptions >.<), and EC in regards to block size only. The reason for this is because there is a higher chance there would be a "business" reason to create a larger block (eg the miners are clearing a spam attack backlog, or a backlog that results as a result of some other temporary reason), than to create a block that violates transaction ordering rules. I also suspect that the miners would mostly be on the same page about a pending "large" block, so they would not try to orphan said block, and there would not be many reorgs because of large blocks as opposed to the large number of reorgs that would result from the OP's proposal.

[Schnibble]

Actually, node requires to be connected at least with one node that relays all desireable transactions.

Even if they are connected to a node that relays all "desireable" transactions, that does not necessarily mean that the node will have all of those transactions stored somewhere in order to properly validate a block

Ok. Node requires to be connected at least with one honest node (that relays all desireable transactions and blocks).

A blocksonly wallet still receives transactions when they come in a block

Ok. That makes sense.

BTW, I have an alternative proposal. Just an idea:

In the future, size of block can reach several hundred megabytes, that imminently cause unacceptable large delays from the moment of block generation until it is completely propagated over the network. At the same time, everybody has a large part of transactions in mempool, so there is no need to transfer them again. It is sufficient to transmit only the header, missing transactions (usually it is coinbase TX) and to define a standard algorithm to construct the block from existing transactions. For example, miner could relay just an array of TxIds.

You are reducing the security of people running blocksonly nodes as they are trusting that blocks that they receive are valid with regards to the transactions inside of them being in the mempool at least X seconds

Yes. But security is reduced very insignificant. Similarly, node can receive a regular orphan block and consider it valid.

bypassing the rule for some time after the proposal deploys is not going to help at all

Why?

this rule will force a node which just started to reject the blocks because it doesn't have a populated mempool to begin with

This behavior for the case if nothing is done at start. What the problem? Just wait a few blocks and rejected chain becomes valid.

A node can most certainly receive a block before transactions are received.

How often? Miners have to relay transactions before including them into the block.

ALLOWING INVALID BLOCKS MEANS THAT MALICIOUS THINGS CAN BE DONE

I don't propose to allowing invalid blocks. Undesirable blocks are valid but they simply becomes orphan because most of nodes reject them.

Has OP stipulated that his "temporarily invalid" block thing is only applicable to certain consensus rules?

Initially, I called it "temporarily rejected block that consists transactions considered as non-standard"
Sorry, if misled. Now changed this to "undesirable" or "block with spam". But I've never said that chain is selected only by amount of work in it.

[achow101]

BTW, I have an alternative proposal. Just an idea:

In the future, size of block can reach several hundred megabytes, that imminently cause unacceptable large delays from the moment of block generation until it is completely propagated over the network. At the same time, everybody has a large part of transactions in mempool, so there is no need to transfer them again. It is sufficient to transmit only the header, missing transactions (usually it is coinbase TX) and to define a standard algorithm to construct the block from existing transactions. For example, miner could relay just an array of TxIds.

You're late to the party. "Your idea" has been discussed for years now, and already implemented and already in use. Look up BIP 152 Compact Blocks or BUIP 10 XThin blocks (both operate conceptually the same, but are wholly different from each other).

Why?

It does nothing about the fact that months after your proposal activates, when I restart my node, it will have either a completely empty mempool, or a mempool that is completely outdated. Either way, if a new block is found immediately after I start my node and I receive that block, the mempool on my node is not likely to have all of the transactions in that block. Even if it did, those transactions would not be there for more than 15 seconds if I received the block less than 15 seconds after starting my node. It will still consider that block invalid, and I will still have to wait for 3 more blocks to be found on top of it before I can actually use my node properly.

This behavior for the case if nothing is done at start. What the problem? Just wait a few blocks and rejected chain becomes valid.

"A few blocks" can take anywhere from 30 minutes to a few hours. Do you realize how long you would have to wait in order for the wallet to actually become usable? That is a huge usability problem. Also, a lot of people may simply run their wallet for a few minutes a day to receive or send some Bitcoin. It takes a couple of seconds to sync the last day of blocks and then the wallet is immediately useable. They use it for a few minutes, and then shutdown. These people (and most people for that matter), don't have time to wait 30 minutes to a few hours for their wallet to actually become usable. There is a possibility that they would have to wait this long many times that they start their wallet, not just the one time wait for initially syncing a wallet.

How often? Miners have to relay transactions before including them into the block.

No they don't. Miners don't have to relay transactions at all. They can just assume that a transaction has already been propagated. Even if they do attempt to relay a transaction, a node that already has a transaction isn't going to re-relay a transaction just because another node relayed the transaction again to them. If they did, that would be a huge DoS vector.

Initially, I called it "temporarily rejected block that consists transactions considered as non-standard"

No, initially you called it "invalid blocks" which encompasses more than just your "undesireable blocks"

Sorry, if misled. Now changed this to "undesirable" or "block with spam". But I've never said that chain is selected only by amount of work in it.

Oh really?

Anyway, the best chain is selected by amount of work in it.

You are not specifying that it is the best VALID chain, just the BEST CHAIN which can include invalid blocks.

[Schnibble]

I would also point out that not all nodes' clocks are on the exact same time (when converted to GMT), so it would be difficult to determine if a transaction has in fact been in a miner's mempool for at least 15 seconds prior to broadcasting a block that confirmed said transaction

Actually, we don't need to determine how long it has been in a miner's mempool.

The idea is that transaction must get into the most nodes' mempools and stay there for some time before it can be included into the block. Thus miners are unable to include their own transactions for free and have to broadcast them over the network as all do.

So nodes should simply reject blocks that include these undesirable transactions:
- full nodes should temporarily reject blocks that consists more than 5% of transactions considered as undesirable (received less than 15 seconds ago | with a fee below 20 satoshies/byte)
- miners should follow the same rules, but in addition they need to hold transactions at least for 15-20 seconds before including them into the block

Nodes check that every transaction in a new received block presents in their mempool at least for 15 seconds. If the block contains too much undesirable transactions, most of nodes simply reject it. Thus miners have to relay transactions(their own too) and then hold them at least 15-20 seconds before including into the block. That's all.

Another problem is that if someone broadcasts a transaction that gets to miner "A" located in China, and a conflicting transaction that gets to miner "B" located in New York, US, then exactly one of those transactions will get rejected by nodes

It is a very good question.
I propose to reject all transactions that conflicting any other transaction in node's mempool. But we have to keep original transaction and reset time when it was received. So it cannot be included into the block another 15 seconds. In addition we need to keep the transaction's size and update it every time when we are rejecting a conflicting transaction that larger then all previous such transactions.

For example. If node receives a transaction "A" which size is 400 bytes, it should keep this transaction and two additional values: { tx_A, 400, time }. If then after 10 seconds it receives a conflicting transaction "B" which size is 500 bytes, it should update these values as follows: { tx_A, 500, time+10 }. If it receives another conflicting transaction "C" after another 5 seconds which size is 550 bytes, it should update the values: { tx_A, 550, time+15 }.

When node is validating the block and found some transaction that conflicts with a transaction in it's mempool, it should check that size of this transaction is not more then stored size. If it is so, the transaction shouldn't be considered as spam. Thus miners cannot include their own transactions into the block for free and cannot spam nodes' mempools with conflicting transactions.

The miners who engage in "SPV" mining will only mine on top of an unvalidated block for as long as it takes them to validate the block. These miners are relying on the valid economic assumption that it is not economical to build/broadcast an invalid block, and it will cost a troll miner much more to broadcast an invalid block than it would cost the rest of the network to build upon an invalid block for only as long as it takes to determine that said block is invalid. It would cost a troll miner roughly 13.5-14BTC to broadcast an invalid block, while it would cost 90% of the network collectively between ~.6BTC-~.62BTC to build on top of that block for the amount of time it takes to figure out it is invalid.

I was speaking strictly in terms of how SPV mining works today, under today's rules.

Your thoughts have meaning under my proposal as well.

I think that EC would probably be better for things like long term changes to demand for block space, and the potential for a one-time large block to account for a temporary increase in the need to confirm many transactions

I have another similar idea but only in Russian so far: https://bitcointalk.org/index.php?topic=1840975.msg18325913#msg18325913

[jonald_fyookball]

OP - You are describing something very similar to EC (emergent consensus). It should be important to point out that only certain consensus rules are subject to EC in your proposal, otherwise the miners will be able to make *all* the rules, which is probably not something that is desirable.

right.  The 'original' EC was just about blocksize. fees already have an EC to them in a sense. 

This thing has gotten so political that something as simple as EC for blocksize is made out to be way more radical than it really is, imo.

[Schnibble]

You're late to the party.

Cool! Didn't dig in details of CMPCT_BLOCK.

It does nothing about the fact that months after your proposal activates, when I restart my node

The idea was to temporarily bypass the rule. So when you restart your node, it will accept all new blocks without checking them for spam.

Do you realize how long you would have to wait in order for the wallet to actually become usable?

Also node can simply bypass this rule for some time at start. Also we can add a new type of message for the purpose of mempool synchronization between the nodes.

Even if they do attempt to relay a transaction, a node that already has a transaction isn't going to re-relay a transaction

I don't propose miners to re-relay a transaction. They just have to relay it once or at least "assume that a transaction has already been propagated". Anyway, transactions always propagated before miners include them into the block (according to my proposal)