One proposal for hindering antisocial blocks

[kjj]

I would like this thread to discuss only this one proposal for hindering empty block mining.  We already have two threads full of speculation about who is mining the empty blocks, and brainstorming other ideas for stopping them.  I'd prefer that this not become a third.

The idea has floated several times to reject blocks that don't contain at least some portion of the transactions that you node is aware of.  Might be time to reconsider it

Of course, it wouldn't be anything as simple as "reject empty blocks" or "reject blocks with less than X transactions".  Those won't work.

It will need to be more like "I see 100 pending transactions that are valid and should be included in the next block.  This block I just got from the network contains less than 25% of them, so I will reject it."  There will probably need to be a threshold too, like "I only see 10 pending transactions, and that isn't enough to enforce a minimum number on this incoming block, so I will accept it, even though it has none".

As a first guess, I would say that a threshold of 10 or 20 and a fraction of 1/4 to 1/2 would work well. 

Method b) Nodes invidually determine which nodes are invalid (seems to be the method advocated by kjj )

The problem is that at any moment in time your node my not see every transaction.  This is important:  CHAIN SELECTION BY NODES IS DETERMINISTIC AND MUST ALWAYS BE TO AVOID FATAL FORKS.  Now matter how a node is made aware of a block, how delayed, or what other data it has at the time it currently always picks the same chain.  This keeps the entire network operating on the same chain and avoids fatal forks (forks which can't be re-organized by simply allowing dominant chain to grow longer). 

If nodes determine which block is valid based on information it has (which may be incomplete) then it is possible (actually very probable given enough time) that some nodes will consider block X valid and some consider block X invalid.  Deterministic chain selection is now fatally broken.  Once a node considers a block invalid it won't change its mind and thus any blocks built on it also become valid.  The two fragments keep diverging.  Forks will build upon forks upon forks until the network a fragmented mess of sub networks each with a different view of the current coin balances.  An attacker could exploit this fact by broadcasting transactions (not in the block it just mined) selectively right before broadcasting the block to ensure some nodes will consider a block valid and some not.  Even without malicious intent the network would destroy itself if chain selection isn't deterministic.

It is a non-issue.  Over time transaction fees will make up a larger and larger share of the revenue a miner needs to stay profitable.

You've added a number of assumptions.  The biggest one is that you assume that block rejection is permanent.

Taking out just that one added assumption, and the system works again.  If my node has block X, and another block comes in (call it Y) that has the hash of block X-1 in it, that new block is rejected.  Unless when block Y+1 comes in and has the hash of block Y in it instead of X.  When that happens, block X is tossed out, and block Y gets unrejected.

In this case, if block Y+1 meets the acceptance criteria, block Y becomes valid, even though I didn't like it when I first saw it.  However, if block Y+1 is also an antisocial block, they both remain on the lonely pile until a good miner builds on top of it, or until the X chain grows longer than the Y chain, making the issue moot.

There may be holes in my idea, but I don't believe that this is one.

16 or 20 years from now, when the subsidy has dropped and hopefully the fees have increased, this may be a non-issue.  But today, it is a real issue.  We are paying full price for these blocks, but only getting part of the value.

For the record, I have no problem with botnets contributing to bitcoin, as far as bitcoin is concerned (i.e. ignoring the moral issues of theft of service or whatever you want to call it), as long as they are contributing to the security of both past and present transactions.

Seems like a bad idea. There's going to be a lot of wasted work if half the miners are working on a different block. You will effectively halve the total network hashrate and make it that much easier to 51% the network. Plus if X+1 and Y+1 are found around the same time, it gets worse. This also makes it easier for some to try to double spend their coins.

Probably best to keep things as they are and things will sort themselves out as designed.

We get network splits all the time.  And good miners would have an incentive to build off of the blocks other good miners, so the forks would rarely be anything near 50/50.

And most importantly, this gives the antisocial miners powerful incentives to become good miners, without waiting for however many years it takes for fees and subsidies to do that automatically.

Let me be very clear.  I propose that:
1. nodes temporarily reject blocks that they can identify as antisocial.  Nodes already temporarily reject blocks, but for different reasons.
2. nodes have a method whereby a block that was temporarily rejected can become accepted.  Again, nodes already do this.

Further, I assert that:
1. these will not lead to permanent or even long-lived chain forks, for exactly the same reason that the ordinary chain forks that we get about once a week are not permanent, and can't become permanent.
2. any method used to game the system to create blocks that include only dummy transactions created by the attacker to circumvent rejection will necessarily require more effort than simply doing the right thing (including ordinary transactions), giving the current antisocial miner(s) a strong incentive to become a positive part of the network.

The two main objections to my proposal are:

1.  It would cause forks, which would be bad.  I think that I've addressed this sufficiently.  It will indeed cause local forks, but those forks will not be global, and will not persist any more than the forks we already have.

2.  Miners would lose quite a bit of autonomy over which transactions they include.  Sadly, this is true, and is actually the entire point of the proposal.  But, the community at large is paying for the work done.  I think that we have the right to set standards for the quality of work which we will accept.

My proposal also has some less obvious advantages.

1.  It requires no new protocol and no new centralized authority.  Each node can make the decision to defer an incoming block based on the data it has locally.

2.  Each node would be able to set their own policy.  The network would reflect something like an average of the policies of all miners.

[Revalin]

I support everything said above.  Here are some initial thoughts for more refined social acceptability policies:


Proposal 1) "blocks must include ($p percent - $o offset) of all pending transactions as measured by fees".

$p can be a low number (say, 10%), and $o could be 5, to give miners benefit of the doubt when there are few transactions available.  If many blocks pass with miners only including the minimum 10%, the backlog will grow.

Since the 10% includes the backlog, they are still guaranteed to eventually be included - they have a minimum 10% chance each block; and transactions paying higher fees will be processed faster since they're more valuable for meeting the quota.  Free transactions are processed only at the whim of miners.


Proposal 2)  "blocks must include ($p percent - $o offset) of all pending transactions as measured by $scoring_algorithm".

An  example scoring algorithm might be: $transaction_score = (0.001 + $transaction_fees) * $time_transaction_has_waited.

The 0.001 ensures that even free transactions will be cleared eventually; scaling by the time the transaction has waited means that minimum-fee transactions will eventually become valuable enough to clear.


Generally I am trying to create as little economic disturbance to the system while still enforcing some minimum inclusion.  The downside to both of the above is the $o offset will create a brief window after each new block (assuming it includes 100%) where Mystery (or anyone) can generate null blocks.  With the example 10% and 5 offset, null blocks would be valid until there are at least 50 fee-paying transactions available (assuming equal fees).  This is likely overly-conservative.  Different values or a less linear relationship (for instance, setting $o to the number of txes seen in the last 10 seconds instead of a static 5) may work better.

Since no protocol change is required, any miner can set their own values or even their own algorithm.  A heterogeneous policy would likely create a smoother response to miners' include policies, but it increases the risk of a blockchain split.  These effects should be modeled.

Edit: One additional rule may be: "all blocks must include at least one previous transaction".  It wouldn't help in a DOS situation (the miner can generate some junk transactions), but it does help with the Mystery situation: it would require miners to actually have a copy of previous blocks.  I'm on the fence whether this is a good idea or not.

[etotheipi]

I like the general idea of this proposal.  To me, it sounds like it's a very simple two steps:

(1) Create black-box algorithm isAntiSocial(newBlock, currMemPool); returns true or false.
(2) Only build on top of blocks that return false from (1)  (not antisocial)

Number (2) means that if we are a miner and we see an anti-social block extending our current longest chain, we will continue building on the previous block and just ignore it.  If someone builds on top of that with a non-anti-social block, then we will switch to it.   In the end it, doesn't really matter which block a miner builds off of, he doesn't get any more or less generation fees.  The only thing he's doing is leading the invalidation of blocks of a miner that isn't doing its job.

One big problem I see with this is that all nodes have conflicting interpretations of how many confirmations a given transaction has, and now when you have a chain of 3 anti-social blocks and then someone builds a non-anti-social block on top of it, your node instantly accepts it as having four confirmations.  Is it a problem?  Probably not, but I haven't thought too much about it.

It certainly could be confusing, as nodes can't actually agree on this metric (though, how important is the metric?).  Forget the black&white case of some nodes producing empty blocks, and other nodes producing "normal" blocks.  Perhaps the anti-social blocks still include some transactions, just not many, and half the network decides social and the other half decides anti-social.  Then new blocks come in extending it, which leads to a new fragmentation of the network.  You've now got miners all different levels:  some are working to extend block X, some extending X+1, some X+2... and the node working on X will create a block that the X+1 and X+2 nodes don't recognize as the longest chain and simply ignore -- dramatically increasing the probability that miners end up with invalid blocks. 

Perhaps there's even a way to trick nodes into believing certain things as a form of attack.  You feed them lots of transactions to change the makeup of their memory pools so that they are likely to view more blocks as anti-social and then subsequently produce lots of blocks that end up being invalid.

[twobitcoins]

Miners have a financial incentive to make sure their blocks are accepted so they get paid.  This proposal tries to use that incentive to force miners to include transactions, but it ignores the impact of that incentive on miners who must decide whether or not to reject blocks that contain too few transactions.

As a miner, if I choose to ignore the longest chain because it contains too few transactions and I instead build a block parallel to that chain, I face the risk that someone else may build on the other chain instead of mine, invalidating my block.  If I instead build a good, transaction-containing block on top of the longest chain, I can avoid that risk.  Therefore, I am unlikely to choose to participate in the proposed scheme, and if code that implements the scheme is added to the standard client, I am likely to disable it for my mining operation.

I also think this is not really true:

2.  Each node would be able to set their own policy.  The network would reflect something like an average of the policies of all miners.

Because of the risk of generating an orphaned block, the only time it is safe to reject a block is if you can be sure that almost everyone else will also reject it.  It's not enough to have a majority (as with a change like P2SH).  Even if 90% of miners will reject a particular block, choosing to reject that block yourself will result in wasted work 10% of the time.  The system can only work if everyone follows the same rules.  It seems nearly impossible to get such a system off the ground.  Any proposal must address these incentive issues.

[etotheipi]

There's an interesting dynamic here... the system only needs to be effective at the beginning to clean up what is there and prevent future abuses.  After the system reaches steady-state, it would be as if it wasn't there -- no one would bother building anti-social blocks anymore, and the logic would remain unused (assuming it's calibrated to only handle the most anti-social cases).  As kjj said, it only needs to be enough to make it easier to follow the rules than work around it.  I think that's a low threshold to meet, and you could set the algorithm to have basically zero-false alarm rate for regular miners.

As twobitcoins said, it is purely, financially negative to follow this rule unless 100% of other honest miners are following it.   But it seems like three conditions at least make the theory plausible:

(1)  isAntiSocial() algorithm is very reliable -- I don't want to discuss the details of this, but I think it's very feasible to separate out the two classes of blocks:  social and anti-social -- which means you could very reliably classify them.
(2)  The logic required for the botnet to get around it would be harder than the rules to just operate honestly. 
(3)  The network well-connected and efficient enough that if 100% of honest miners implement the logic, that that 99%+ of them will also agree on isAntiSocial for each new block -- thus the cost to them of improving the network costs them less than 1% of their mining rewards -- perhaps an efficient network is worth 1% of their total income...

On point number three -- in a nominal scenario, the negative impact to their mining rewards is only transient:  if the system is effective, anti-social blocks will stop happening, and then it will become irrelevant.  On the other hand, if it really is a botnet, then the operator really doesn't care.  His botnet produces only 20% valid blocks (because miners will accept empty blocks that were found quickly), but he's still making a ton of money from it for zero cost.  Perhaps it's worth it to him to implement the honest logic so that he can up it to 80%, but maybe he won't...

[vv01f]

I am not sure if it really is necessary to give new rules *to miners*. Instead I would love to give *the non-mining nodes* a more sustaining role by their role as relay. Those shall not broadcast "antisocial" blocks and by this miners would have to arrange with with nodes which could help to solve the imbalance between these two roles in the network. This way to harm the network you not only need 50% hashing-power but also 50% broadcast-power as security.

[DeathAndTaxes]

I am not sure if it really is necessary to give new rules *to miners*. Instead I would love to give *the non-mining nodes* a more sustaining role by their role as relay. Those shall not broadcast "antisocial" blocks and by this miners would have to arrange with with nodes which could help to solve the imbalance between these two roles in the network. This way to harm the network you not only need 50% hashing-power but also 50% broadcast-power as security.

That wouldn't do anything.  What matter is will another miner build upon a "anti-social" block.  If all non-mining nodes agree that a block is anti-social but all miners build upon it then the distinction is useless.

Finding the nodes of other major miners isn't that difficulty and one could simply send their anti-social blocks directly to other major miners.
There is no such thing as broadcast power.

[vv01f]

Finding source of other major miners is easy and you can simply send directly to them.  If 99% of network refuses to forward a block but it is built on by other miners you haven't accomplished anything.

Yes, got it. The subnet is to strong and doesn't rely on the nodes.

[Killdozer]

Your proposal looks like it just might work, I see no obvious holes in this.

About

2.  Miners would lose quite a bit of autonomy over which transactions they include.  Sadly, this is true, and is actually the entire point of the proposal.  But, the community at large is paying for the work done.  I think that we have the right to set standards for the quality of work which we will accept.

The most important reason for why miners would want any autonomy of decision is for later days when their revenue will come from the transaction fees instead of block rewards. Then they must have the power to only include transactions that have enough fees.

But if the percentage of transcations that are needed to be included in order for the block to be valid is set pretty low (like 20%)(which will be enough to fix the problem), the miners would still have the freedom to not include 60-70% of all transactions, and obviously only choose the best paying ones, so it could still work!


I also think that keeping the block rewards constant for all eternity is beneficial in many ways and is going to be discussed more and more when we get near it's planned elimination, and if that is how it is done (that we do change the protocol to keep block reward for all eternity), the miners will not need that much freedom, they will just have to include all the transactions they see. A way to think about this is that instead of each transaction paying a small amount to the miners, the whole community pays to them through constant block rewards and thus a very slow inflation rate. This is even more logically correct, because the mining is not only beneficiary for the people transferring bitcoins right now, but also to the people keeping bitcoins, because mining causes all the bitcoins being more secure and more stable, thus helping even people sitting on bitcoins.


Still, I think it is too early to be implementing anything like this, but it is certainly beneficial to start the process of evaluating this thing as a community.

(1)  isAntiSocial() algorithm is very reliable -- I don't want to discuss the details of this, but I think it's very feasible to separate out the two classes of blocks:  social and anti-social -- which means you could very reliably classify them.

This is absolutely a requirement of course, but even there would be small errors now and then - they would get eaten up by the fact that those blocks become temporarily blocked. So even if a small percentage of miners would get some block wrong sometime, the whole thing would figure itself out at the next block, in the true decentralized spirit . No permanent damage.

[DeathAndTaxes]

I also think that keeping the block rewards constant for all eternity is beneficial in many ways and is going to be discussed more and more when we get near it's planned elimination, and if that is how it is done, the miners will not need that much freedom, they will just have to include all the transactions they see.

Ok this doesn't make any sense.  As block reward declines the need for miners to exclude unprofitable tx becomes even MORE not less important.  

To combine your the two underlined portions, so hypothetically when the block subsidy is 0 BTC miners should be required to include all tx.  Even if they are mostly 0 fee?  If miners must include all tx (even those w/ no fee) why would any user include a fee?  The simple answer is they won't and thus both block subsidy and fees will be roughly ~0.

So when block subsidy is 0 and fees are 0 how will the network remain secure?

[etotheipi]

I think 20% is too much -- in regards to what is needed to ensure a miner is doing his job versus the autonomy desired by miners.  As mentioned before, there only needs to be enough tx required that the rogue miner has to start paying attention to other tx on the network:  it sounds like he's not paying attention to any of them: 0%.  If he has to start paying attention an including any of them, he might as well include all of them.  Problem solved.

Well not actually solved yet:  but my point is there's a very low threshold for what is needed to get him to start paying attention.

EDIT: on second thought, what's stopping him from just creating 20 tx per block himself and broadcasting them, just to make sure he can meet the X% req't?  Seems like we might accidentally encourage the rogue miners to start adding bloat just to get around the rule (which might be easier than starting to pay attention).

[Killdozer]

Ok this doesn't make any sense.  As block reward declines the need for miners to exclude unprofitable tx becomes even MORE not less important.

I guess I wasn't clear enough , sorry
by this

and if that is how it is done

I mean of course that the block reward is kept constant. (That we do take this protocol change in the future.)

[DeathAndTaxes]

I mean of course that the block reward is kept constant. (That we do take this protocol change in the future.)

Nobody (I guess you are?) is even suggesting block reward will be kept constant.  The fixed # of coins is a cornerstone of Bitcoin.  Changing that now (even if desirable) will never happen.

[kjj]

I also think that keeping the block rewards constant for all eternity is beneficial in many ways and is going to be discussed more and more when we get near it's planned elimination, and if that is how it is done, the miners will not need that much freedom, they will just have to include all the transactions they see.

Ok this doesn't make any sense.  As block reward declines the need for miners to exclude unprofitable tx becomes even MORE important. 

So hypothetically when the block subsidy is 0 BTC miners should be required to include all tx.  Even if they are mostly 0 fee?  If miners must include all tx (even those w/ no fee) why would any user include a fee?

So when block reward is 0 and fees are 0 how will the network remain secure?

He is talking about keeping the subsidy.  Miners would have to include free transactions because they were paid by the subsidy, not by fees.

This is actually the best reason I've ever seen suggested for potentially keeping the subsidy.  But I still don't think it is a good idea, and I don't think it'll gain any traction.

[Killdozer]

Nobody is even suggesting block reward will be kept constant.  The fixed # of coins is a cornerstone of Bitcoin.  Changing that now (even if desirable) will never happen.

I am suggesting it, and some other people seem to have the same idea. And it can happen, that's ultimately up to the miners. Protocol changes have been done before. But that's really not the topic here, I just said that if that was to be done, the miners would not need that much freedom and the proposal discussed here would work even better.

[DeathAndTaxes]

I am suggesting it, and some other people seem to have the same idea. And it can happen, that's ultimately up to the miners. Protocol changes have been done before. But that's really not the topic here, I just said that if that was to be done, the miners would not need that much freedom and the proposal discussed here would work even better.

While anything can happen you will never get 70% to 80% of miners to support it.  Notice I said 70% not 50.001%.  While technically you could fork the chain with any amount of support (even 1%) a 50% split would likely be the death of Bitcoin.  The original and modified forks would both exist, people would have coins in both forks and when sending coins they would need to specify which Bitcoin they were talking about.  Some exchanges/merchants might support "new Bitcoin" and some "old Bitcoin" and some would support both.  Users would have differing values in each chain and those supporting one side would see no value in the other resulting in massive selling. 

It would be beyond chaos, and the negative PR alone would be catastrophic.

It won't happen.  Even many miners who would support it in theory won't support it for the massive disruptions it would cause.  Discussion it is just pointless.

[kjj]

I think 20% is too much -- in regards to what is needed to ensure a miner is doing his job versus the autonomy desired by miners.  As mentioned before, there only needs to be enough tx required that the rogue miner has to start paying attention to other tx on the network:  it sounds like he's not paying attention to any of them: 0%.  If he has to start paying attention an including any of them, he might as well include all of them.  Problem solved.

Well not actually solved yet:  but my point is there's a very low threshold for what is needed to get him to start paying attention.

EDIT: on second thought, what's stopping him from just creating 20 tx per block himself and broadcasting them, just to make sure he can meet the X% req't?  Seems like we might accidentally encourage the rogue miners to start adding bloat just to get around the rule (which might be easier than starting to pay attention).

Nothing would stop him from creating his own transactions.  But he still has to distribute them to his nodes, and even one transaction would be much more data than he is distributing now.  Also, the data he needs now he can get from a variety of public sources.

[Killdozer]

DeathAndTaxes, what you are describing is simply called a split chain and these problems arise with any big protocol change, and there are numerous ways of addressing them. But you are right, it's pointless to discuss that in this topic, I simply mentioned that as a possibility and how that would connect to the proposal in this topic.

[Haplo]

I mean of course that the block reward is kept constant. (That we do take this protocol change in the future.)

Nobody (I guess you are?) is even suggesting block reward will be kept constant.  The fixed # of coins is a cornerstone of Bitcoin.  Changing that now (even if desirable) will never happen.

Actually I've considered that alternative from time to time. A very low (long run < 1%/y) inflation rate is basically just as good as zero inflation, with the added benefit of replacing lost coins. The main problem is that there's no guarantee that the block reward will be enough to pay for mining, but similarly there is no guarantee that it won't completely pay for mining and make tx fees basically superfluous. That would require predicting the future market price for BTC, the cost of a future mining rig, the number of rigs required to handle the future tx throughput, which you'd also need to predict, and so on.

The same limitations of multiple, unpredictable variables makes it somewhere between difficult and impossible to make any determinations about efficient mining incentives. The major advantage to being able to predict something like that is that then you could charge a standard fee rate or no fees and basically every tx could still be verified by the next block. Given that "change" can't be spent till at least 1 confirmation (I assume anyway), the time consideration can become a major bottleneck for the use value of BTC.