Miners that refuse to include transactions are becoming a problem

[guruvan]

And to that end, this is what I dug up researching this today:

Right now the transaction fee address is left "blank" and the block generator fills it out.
Now you would fill it in with the address of the person you are asking to build the block.  

If you're only going to have one person work on building the block, that could take days.  Oh, do you mean send a different variation to each node with the tx fee written to them?

The way it is now, it's whoever builds this gets it.

If we needed to, we could have a BitTorrent-esque tit-for-tat for transaction broadcast.  Relay paying transactions to me, or I won't relay them to you.  It probably won't be an actual problem though.  It only takes one node relaying like it should to cancel out 7 others greedily not relaying.

Some food for thought.

[theymos]

Then Haplo's clarification is correct?

I think so.

So it means instead of forcing miners to include transactions, they would instead be forced to access recent blocks and extract data from them.

What's still a little confusing about that on a conceptual level is this: mining ALREADY requires access to recent blockchain data, in the form of a hash of the prior block. Even if it required new code, couldn't the method already being used to distribute that prior-block-hash to all the nodes on a botnet be copied/modified to also distribute the newly-required "proof of verification?"

It's not just recent blocks: they'd need to be able to access random unspent transactions in the chain.

The proof of verification could be distributed, but then at least you know that someone is doing the proper verification and miners aren't blindly building onto invalid blocks.

[jgarzik]

Why would a rational miner want to allow free tx forever?

To encourage users to use the network.

A network's value increases as its userbase increases (and vice versa).

[DeathAndTaxes]

reread what you quoted.

[jgarzik]

Personally, I favor a drastic rule such as

* Build a list of "likely to be in next block" transactions, from memory pool
* Do not relay blocks unless they contain at least 50% of the transactions in the "likely to be" list

This is "drastic" in my estimation because such a rule has notable downsides,

* Increases possibility of short term fork
* Creates de facto requirement that at least 50% of each block are standard transactions (as defined by isStandard)
* and some other minor fallout

However, it is a strong rule that does address the issue at hand, while permitting valid, no-tx-activity zero-transaction blocks to exist.

[DeathAndTaxes]

Personally, I favor a drastic rule such as

* Build a list of "likely to be in next block" transactions, from memory pool
* Do not relay blocks unless they contain at least 50% of the transactions in the "likely to be" list

This is "drastic" in my estimation because such a rule has notable downsides,

* Increases possibility of short term fork
* Creates de facto requirement that at least 50% of each block are standard transactions (as defined by isStandard)
* and some other minor fallout

However, it is a strong rule that does address the issue at hand, while permitting valid, no-tx-activity zero-transaction blocks to exist.

So miners are required to include tx that don't meet their fee requirements?  If they don't they risk having the valid blocks not forwarded?

[Haplo]

Personally, I favor a drastic rule such as

* Build a list of "likely to be in next block" transactions, from memory pool
* Do not relay blocks unless they contain at least 50% of the transactions in the "likely to be" list

This is "drastic" in my estimation because such a rule has notable downsides,

* Increases possibility of short term fork
* Creates de facto requirement that at least 50% of each block are standard transactions (as defined by isStandard)
* and some other minor fallout

However, it is a strong rule that does address the issue at hand, while permitting valid, no-tx-activity zero-transaction blocks to exist.

That's similar to what Gavin and Revalin proposed, the downsides of which have been discussed at length .

Gmaxwell's proposal should achieve the same effect without the downsides of a threshold-based approach. The main advantage of an approach like yours is that if the bot owner finds a way to include verification without including current tx, he still gets shut out for non-cooperation.

I think if you were to do it that way, a much lower threshold would suffice. AFAIK our mystery miner only includes a single token tx in his blocks, or none. With ~80tx average per block, a 5% threshold would neatly wipe them out. At that rate, miners have little to complain about, and the likelihood of a fork is lower. At 50%, a pool might end up having an otherwise valid block excluded if they happen to charge on the high end of fees. At 5%, a pool charging enough to get excluded would not be providing anything extra to its users (if any).

[Nim]

Going forward (once I have modified bitcoind and tested it) I will be excluding ALL transaction with < 0.01 BTC in fees.  I will also share the patch as open source and encourage other miners to adopt similar minimum fee rules (although each miner could choose a different amount).

What would happen if everyone did this and I send out a transaction tomorrow with no tithe to the system. Where does that transaction go? How would the user have any idea what has happened? How would they know if it still floating around out there, maybe waiting on a generous miner who has a long pro bono publico wait list, and that they don't need to resend it?

What if the fee requiring miners are in the minority but each of the 8 node connections that my client has just happens to have this fee rule? Again, does my transaction get lost in the nether? How would I get my transaction out to the miners who are willing to include my transaction? Would the nodes say "nope, fees too low on this transaction, into the round file it goes..." or would the nodes say "nope, fees too low on this transaction but maybe Bob down the street would be willing to help you out, I'll pass it on"?

[DeathAndTaxes]

What would happen if everyone did this and I send out a transaction tomorrow with no tithe to the system. Where does that transaction go? How would the user have any idea what has happened? How would they know if it still floating around out there, maybe waiting on a generous miner who has a long pro bono publico wait list, and that they don't need to resend it?

The modification I was proposing wouldn't affect relaying so tx would be relayed to every node even my node would continue to relay it.The tx is valid so rejecting it is inappropriate I just don't want to mine it.

As far are mining it into a block.  If every single miner on the planet (and all new miners coming online) had a min fee rule with no exceptions higher than your fee then it would never enter a block.  However that is obviously unrealistic.  If 10% of miners had such a rule then 90% of network would still include it in the next block so your expected confirmation time would simply be 11 minutes vs 10 minutes.  If say 99% of network excluded it from the next block (due to insufficient fee) then your expected confirmation time would be 16 hours.  Obviously both of those numbers are expected confirmation time, the actual time would be subject to variance just like any confirmaiton time is.

What if the fee requiring miners are in the minority but each of the 8 node connections that my client has just happens to have this fee rule? Again, does my transaction get lost in the nether? How would I get my transaction out to the miners who are willing to include my transaction? Would the nodes say "nope, fees too low on this transaction, into the round file it goes..." or would the nodes say "nope, fees too low on this transaction but maybe Bob down the street would be willing to help you out, I'll pass it on"?

The later.  My proposed change would only affect what tx go into the next block and only for those miners/pools which implement the patch.  Any valid tx would continued to be relayed.

[Revalin]

So miners are required to include tx that don't meet their fee requirements?  If they don't they risk having the valid blocks not forwarded?

Yes, you will be orphaned if you set a "fee > 100BTC no exceptions" include policy.  It is not reasonable to demand that the network have no right to question your policy if you institute a bad one: the whole point is to stop people from excluding all txes.  We're trying to find a minimal method that won't affect any reasonable include policy, but it does create a requirement that you might have to do something.  A relay policy with no actual mandate is pointless.  As I said:  Checks and balances.

You're becoming a broken record.  Your point has been heard and is being taken into consideration.

[DeathAndTaxes]

So miners are required to include tx that don't meet their fee requirements?  If they don't they risk having the valid blocks not forwarded?

Yes, you will be orphaned if you set a "fee > 100BTC no exceptions" include policy.  It is not reasonable to demand that the network have no right to question your policy if you institute a bad one: the whole point is to stop people from excluding all txes.  We're trying to find a minimal method that won't affect any reasonable include policy, but it does create a requirement that you might have to do something.  A relay policy with no actual mandate is pointless.  As I said:  Checks and balances.

You're becoming a broken record.  Your point has been heard and is being taken into consideration.

Please don't use strawman to support your argument.

What if fee policy was 0.01 BTC?  Less than 10% of txs in last 20 blocks had a fee of 0.01 or higher.  Hell less than 1/3rd have a fee of any kind.    Thus a requirement to include 50% of txs mandates including free txs also.  Any fee policy no matter how low (even min 1 satoshi) would face exclusion.

So at least be honest with the ramifications. 

[Revalin]

It's not a strawman.  I'm just emphasizing that some minimum standard is reasonable.  I'm sure we can come up with a method that will work.

Rather than having a minimum bar, why not make it a sliding scale?  For example:  you must include at least 20% of the top-10-percentile-fees (either since the last block or sliding-window) transactions?  As always, -5 or so to accommodate latency etc.  That would give you a lot of latitude to cherry-pick.  In many circumstances (only a couple transactions with high fees available) you would even be able to mine a null block and have it accepted.

Edit:  Or rather than allowing a null block, we could mandate that every block include at least one transaction if there are at least 5 outstanding.  Is it an undue imposition to include one free transaction if there are absolutely none that meet your include policy?  That provides a very strong incentive to include fees, and completely solves the null-miner problem:  Mystery can mine a null block if he wants to - but only in the few seconds after the a previous block that cleared out all waiting transactions, during which a null block is reasonable by any standard.

[JoelKatz]

Rather than having a minimum bar, why not make it a sliding scale?  For example:  you must include at least 20% of the top-10-percentile-fees (either since the last block or sliding-window) transactions?  As always, -5 or so to accommodate latency etc.  That would give you a lot of latitude to cherry-pick.  In many circumstances (only a couple transactions with high fees available) you would even be able to mine a null block and have it accepted.

The goal is not to force any particular policy but just to avoid miners being lazy and not gathering any transactions at all. So long as they can't take the shortcut of not including any transactions at all, their self-interest will lead them to include fee-paying transactions.

So the rule could be this simple and this modest: If there are at least four transactions that are not free and have at least the standard fee that were not included in the previous block even though they could have been, discourage any block with just a coinbase transaction.

So you are never punished for not including a free transaction. And you are never punished for not getting a new work unit when you didn't have to anyway because a new block was found. But this will solve the problem of paid transactions backing up if multiple blocks are found by miners who refuse to include any transactions.

This is, IMO, the least onerous rule that would solve the problem.

[theymos]

So the rule could be this simple and this modest: If there are at least four transactions that are not free and have at least the standard fee that were not included in the previous block even though they could have been, discourage any block with just a coinbase transaction.

You also need to discourage the block if it has only transactions you don't know about, since otherwise the miner could make up transactions.

I'm not bothered by this rule from an economic perspective, but I'm not confident in the effectiveness of discouragement (due to bad incentives), and I'm uneasy about relying on transaction propagation in any way, since propagation is imperfect and will get worse with time. I prefer gmaxwell's proposal because it only relies on the block chain.

[Killdozer]

I think if you were to do it that way, a much lower threshold would suffice. AFAIK our mystery miner only includes a single token tx in his blocks, or none. With ~80tx average per block, a 5% threshold would neatly wipe them out. At that rate, miners have little to complain about, and the likelihood of a fork is lower. At 50%, a pool might end up having an otherwise valid block excluded if they happen to charge on the high end of fees. At 5%, a pool charging enough to get excluded would not be providing anything extra to its users (if any).

Just wanted to say that this makes a lot of sense to me.
It still probably needs some detals, like only take into consideration transactions which are older than 1 minute or so (so that they have very high probability of having been globally-relayed) and perhaps not enforcing this rule if the total number of unconfirmed transactions (that are "supposed" to be in the next block) is too little. (This can happen due to a bug, or, what is more realistic, due to 2 blocks being found very short time apart (which can happen, it's all just probabilities)). So for example if there are only 5 transactions that are supposed to be included, don't enforce the rule, because in that case the outcome of the percentage calculation is biased too much by every single transaction (and thus much more prone to errors and can create unnecessary forks).

[Timo Y]

The market purists on this forum should agree that the market is working exactly as it should.

The block award is for securing the network against a double spending attack, not for confirming transactions. Transaction fees are for confirming transactions.

The null miners are essentially specialized miners that are only in the business of security, and not in the business of security+confirmation like most other miners.

I don't think this is a huge problem. Even if the null miners had 50% of hashing power, average users would hardly notice any difference. Even if they had 90%, bitcoin would still work fine. It would just take longer for transactions to confirm. 

This problem is likely to disappear in future because:

a) The block award will drop
b) As bitcoin becomes more popular, there will be more transactions per block, and thus more fees to be collected
c) FPGAs and ASICs will marginalize the hashing power of botnets.
d) Most day-to-day transactions will be between trusted parties who don't care whether a transaction isn't confirmed yet, and who don't even broadcast their transactions most of the time.

Is it worth adding complexity to the bitcoin protocol, with possible unintended consequences, in order to solve a temporary problem?

[Haplo]

The market purists on this forum should agree that the market is working exactly as it should.

The block award is for securing the network against a double spending attack, not for confirming transactions. Transaction fees are for confirming transactions.

The null miners are essentially specialized miners that are only in the business of security, and not in the business of security+confirmation like most other miners.

I don't think this is a huge problem. Even if the null miners had 50% of hashing power, average users would hardly notice any difference. Even if they had 90%, bitcoin would still work fine. It would just take longer for transactions to confirm. 

This problem is likely to disappear in future because:

a) The block award will drop
b) As bitcoin becomes more popular, there will be more transactions per block, and thus more fees to be collected
c) FPGAs and ASICs will marginalize the hashing power of botnets.
d) Most day-to-day transactions will be between trusted parties who don't care whether a transaction isn't confirmed yet, and who don't even broadcast their transactions most of the time.

Is it worth adding complexity to the bitcoin protocol, with possible unintended consequences, in order to solve a temporary problem?

Several reasons.

1: The block reward may drop in half come 2013, however if the value of 1BTC doubles, then nothing has been lost. That leaves him fully secure to continue ignoring tx until at least 2017.

2: He isn't actually contributing to security, since he's only generating blocks off the previous hash without validating anything. That means if an attacker starts throwing down invalid blocks, our friendly botnet might blindly build right on top of them and give them a nice helping hand.


Also, it depends on which method is used. Gmaxwell's proposal would not affect legitimate miners in any way. Any proposal which excludes blocks that are "too empty" would have unintended consequences, but in the event of an attempted 51% empty-block DoS attack, the ability to screen empty or cheap blocks would severely limit the damage they could do.

We've been over this like 50 times =\.

[JoelKatz]

2: He isn't actually contributing to security, since he's only generating blocks off the previous hash without validating anything. That means if an attacker starts throwing down invalid blocks, our friendly botnet might blindly build right on top of them and give them a nice helping hand.

Not true. All the building on top of an invalid block can't hide or change the fact that it's invalid and so would do no harm whatsoever. It would simply be wasted effort.

These botnets do increase the amount of hashing power an attacker would need to launch something like a 51% attack. They do increase the security of the network. And that's what the block reward is for. If the system is broken, it's because the transaction fee and volume aren't sufficient to justify the expense of including transactions in a block.

I wonder if we can detect these botnet nodes somehow, perhaps by the particular messages they send or don't send, and send them *invalid* blocks. It won't harm normal clients since they'll just ignore them. But the botnet nodes, lacking the block chain, will work on the wrong blocks more often than the right ones. This may do more harm than good in the long term though.

[benjamindees]

* Build a list of "likely to be in next block" transactions, from memory pool

you must include at least 20% of the top-10-percentile-fees (either since the last block or sliding-window) transactions

So what is the downside to this again?

This seems to solve the problem by setting a reasonable limit on miners' ability to extort the network for fees.  We really should consider a somewhat drastic solution.  Even though the mystery miner is likely a botnet, within a year or so we could easily see this same scenario play out with a large FPGA cluster that doesn't want to include transactions for whatever reason.  It isn't a problem that is going to just go away, or be solved any time soon by falling block rewards.  We're already subsidizing miners with 30% inflation;  it might as well go to those that at least process a few transactions.

[Jon]

They aren't becoming a problem. They are simply making transaction processing more valuable. When the incentive comes for them to process transactions, will be the day a higher fee is set across the board.

I see no issue. If the problem becomes bad enough, people will talk with their pocketbook.

Although, you could always coerce the miners into giving you free transactions. As for the consequences to Bitcoin as a whole in such a case, I'll leave that to you to figure out.