Miners that refuse to include transactions are becoming a problem

[Revalin]

It's a DOS vulnerability, as discussed earlier, and it makes 51% attacks easier, even if the null-miners are not directly collaborating.

D&T:  We could also make it so that you have to include ( ( n * 0.25 ) - 5 ) of the last n paid transactions, and have a lower (or no) quota for free transactions.

[DeathAndTaxes]

If those tx are free I am forced to include them or have my block rejected?

If 100% of transactions coming through are free you would be forced to include some, but realistically there are always enough paid transactions that you can meet the 25% quota easily.  If you only include 25% during an "all free transactions" hour, people will have an incentive to include a fee to get their transaction included sooner (4x sooner on average in this case).

25% is just an example; pick a value that will accommodate any reasonable miner's include/exclude policy.

Really have you looked at the last dozen or so blocks?

90%+ are free and 0% have a fee of 0.01 or higher.  So 100% are free or so worthless as to be essentially free.

So you saying "miners can set their own fees" is a blatant lie if the same miner is then forced to include free (and near free) tx.  If clients know miners are forced to include their freeloading tx what is the incentive for them to pay a realistic fee.

Of course that ignores all the timing attacks, orphaned blocks, and network latency issues any such proposal will have.  A hint: there is no unified view of the network.  Propogation delays ensure parts of the network are always out of sync.  So a miner cutting it close risks having some nodes see the block as valid but others (who just picked up new tx) see it as invalid.

Also it makes tx processing less efficient.  Most pools don't update getworks when new tx arrive however to meet the 25% treshold (which is likely 50% threshold just to be safe) they would either need to issue an LP on every tx and include even MORE free tx so that as new tx arrive they stay above the safe limit.

Essentially all that combined means miners have no pricing power (they already have very little).  All this to solve a non-problem.  Sorry no thanks.

[Raoul Duke]

I'll refrain from discussing the technical solutions because I'm not smart enough, but...

Don't you guys realise that you are doing a better job at undermining peoples trust on Bitcoin by discussing this non-issue relentlessly than "mystery jerk miner" is by not including tx's on the blocks he mines?

[Revalin]

90%+ are free and 0% have a fee of 0.01 or higher.  So 100% are free or so worthless as to be essentially free.

If miners care, why aren't they already doing something about it?  People will send free txes as long as the system allows it.

If clients know miners are forced to include their freeloading tx what is the incentive for them to pay a realistic fee.  ...  Essentially all that combined means miners have no pricing power (they already have very little).

100% chance of inclusion per block instead of 25%.  Or as I said later:  make the n based on the number of paid transactions, and let miners ignore the free ones if they want.

So a miner cutting it close risks having some nodes see the block as valid but others (who just picked up new tx) see it as invalid.

Sure, that's why the -5 exists: to give some buffer if there were a few very recent transactions.  It would only matter on very closely spaced blocks: normally you'd be including say 25%+ of transactions to meet a 10% quota and you'd have plenty of room.

Also it makes tx processing less efficient.  Most pools don't update getworks when new tx arrive...

How is that a good thing?  Miners should be encouraged to include as many recent reasonable-fee-paying transactions as possible.  Updating getworks isn't hard.

[Revalin]

Don't you guys realise that you are doing a better job at undermining peoples trust on Bitcoin by discussing this non-issue relentlessly than "mystery jerk miner" is by not including tx's on the blocks he mines?

If people's trust depends on us being hush-hush about potential problems, even small ones like this, then the whole thing is hopeless.

In the long run open discourse and letting the light fall on every corner is much more trustworthy than trying to pretend it's a perfect system that will never need improvement.

[Raoul Duke]

Don't you guys realise that you are doing a better job at undermining peoples trust on Bitcoin by discussing this non-issue relentlessly than "mystery jerk miner" is by not including tx's on the blocks he mines?

If people's trust depends on us being hush-hush about potential problems, even small ones like this, then the whole thing is hopeless.

In the long run open discourse and letting the light fall on every corner is much more trustworthy than trying to pretend it's a perfect system that will never need improvement.

You said it very well, potential problems, but whoever reads this thread and the other thread at the mining forum will read "real problem in need of a solution" and freak out.

I'm just waiting for some dumb reporter to pick this up and start writing how Bitcoin is controled by botnets who don't even process tx's and other lies...

[Revalin]

It's a real, though currently minor and not network-threatening, problem that needs a solution eventually.  It cannot currently be exploited for any real harm other than my indignation that some miners are being paid without doing their proper work.  This discussion is not about emergency damage control; we are just speculating and brainstorming ideas for how to adjust things in the future.

Better?

[Technomage]

I agree with Revalin once again. I don't claim that this is a critical issue, my problem with this is the ethical issue of leeching Bitcoin. That is what the mystery miner is essentially doing because he is only doing half of his job as a miner.

As far as solutions go I like what Revalin is proposing. It would probably be good to make it so that the portion of transactions that must be included is calculated from transactions that have a fee (any that's at least the minimum).

Again this would not remove the market from tx fees, all transactions outside that percentage (whatever percentage we chose) would still be controlled by the miners, they are not forced to add them.

In fact this could actually add incentive for users to start adding higher tx fees to make sure their transaction fits this special slot and thus would be guaranteed in the next block.

A good start would be if something like this was simply built-in to regular nodes so that we could essentially boycott malicious miners by choice. I'd love that. (referring to what MoonShadow proposed)

[DeathAndTaxes]

Once again making a distinction of paying vs non paying is dubious.

1 satoshi is paying.

If every transaction had a 1 satoshi fee the avg block would contain 0.0000008 in fees.
The entire network annually would produce ~ 0.0292 BTC in fees.

So paying vs non paying is a dubious distinction.

Going forward (once I have modified bitcoind and tested it) I will be excluding ALL transaction with < 0.01 BTC in fees.  I will also share the patch as open source and encourage other miners to adopt similar minimum fee rules (although each miner could choose a different amount).

Do you believe I should not be allowed to exclude fees I consider insufficient?  
More generally:  Do miners have a right to charge that amount they expect in an open and free marketplace for fees?  

People like to use the term boycotting (as in the above post).  If no tx had a fee of 0.01 BTC+ and as a resulted I mined a 1 tx block would that be boycotted.  What if I included a single dummy tx of my own (1 BTC from me to me with 0.01 BTC fee to me)?

[Technomage]

DeathAndTaxes is constantly approaching this issue from an irrelevant angle (from my point of view). Tx fees do not currently play a significant role in miner rewards, I have repeatedly stated that market based tx fee system is necessary eventually and I have no interest in disrupting that, on the contrary. I like that in many ways.

This issue is different and it has to do with the incentive for a potential botnet owner to simply ignore all transactions. If he has a strong incentive to do this because of a Bitcoin exploit (I consider this issue an exploit) we might have to start paying ridiculous tx fees to make him change his behaviour, which would basically destroy Bitcoin. We are nothing without cheap transactions, that is one of the main benefits of using Bitcoin.

I don't claim transactions should be free either. But they must be cheap. We have no idea what kind of benefits this mystery miner gets from not adding transactions. It could be that he simply can't run that botnet securely without having a system in place that ignores transactions. In that case he would NEVER start adding transactions, not at any tx fees, because his entire mining operation would be at risk.

The only solution I care about is one where a potential botnet miner is forced to at least work with the blockchain. I'm not interested in either keeping Bitcoin transactions free nor am I interested in forcing the miners to do something in general. I am definitely interested in any proposal that forces miners to add at least some transactions, if there are transactions to be added.

And to answer the final point, I believe dummy transactions wouldn't really help if the solution is designed properly. The nodes are aware of all transactions, if x % of them (that have at least the minimum fee) are not included in a block, they simply refuse it. It would definitely be beneficial to Bitcoin users if this could be done. I believe the tools to stop freeleech mining can be built.

[VVS dump]

I elaborated before why forced inclusion of transactions is senseless because it can be evaded without problems.

[Revalin]

I will be excluding ALL transaction with < 0.01 BTC in fees.  ...  Do you believe I should not be allowed to exclude fees I consider insufficient?

Of course you should be allowed!  That's well established as a miner's right.  Similarly it is also a relay node's right to exclude blocks they consider insufficient.  Check and slightly imperfect balance.

Relay nodes already enforce some minimum fees; those were actually reduced last year.  Right now it's probably best for free txes to be as low as possible - low friction will help growth in ways much more significant than some 0.01BTC fees.  That said I encourage you to institute this policy.  The completely homogenous one-formula-fits-all isn't good for the network.  I think we would be much better off with a gradient of minimum fees among miners instead of the very sharp instantly accepted / transaction limbo that we have now.

Even a 1 satoshi minimum would get people to at least start paying fees, and pave the road for the future where free/1-satoshi transactions are slow (just a few miners supporting them) and reasonable fees are required for fast processing.  It's probably helping your goals on the whole.

But I don't think we should make a universal mandatory minimum fee a rider on the fix for null blocks; that's really a separate issue.  In my opinion we should just try to solve the latter in a way that doesn't make the former worse or cause other collateral damage.

[Technomage]

I will be excluding ALL transaction with < 0.01 BTC in fees.  ...  Do you believe I should not be allowed to exclude fees I consider insufficient?

Of course you should be allowed!  That's well established as a miner's right.  Similarly it is also a relay node's right to exclude blocks they consider insufficient.  Check and slightly imperfect balance.

Relay nodes already enforce some minimum fees; those were actually reduced last year.  Right now it's probably best for free txes to be as low as possible - low friction will help growth in ways much more significant than some 0.01BTC fees.  That said I encourage you to institute this policy.  The completely homogenous one-formula-fits-all isn't good for the network.  I think we would be much better off with a gradient of minimum fees among miners instead of the very sharp instantly accepted / transaction limbo that we have now.

Even a 1 satoshi minimum would get people to at least start paying fees, and pave the road for the future where free/1-satoshi transactions are slow (just a few miners supporting them) and reasonable fees are required for fast processing.  It's probably helping your goals on the whole.

But I don't think we should make a universal mandatory minimum fee a rider on the fix for null blocks; that's really a separate issue.  In my opinion we should just try to solve the latter in a way that doesn't make the former worse or cause other collateral damage.

+100

[Haplo]

While it's possible that it's some minority client malfunctioning, 1.8m users and/or 16% of the total network processing power is pretty extreme. The potential advantages for running a stripped down botnet make that the most likely scenario, and whether or not that's the case 16% of the network isn't contributing to the rest while collecting all the benefits.

Also, I think people are still misunderstanding what has been proposed. While Gavin and others have proposed systems which would omit or delay "cheap" blocks, gmaxwell's proposal is simply that in order for a block to be valid they have to prove that they have access to the blockchain. It doesn't mean they have to include any tx if they don't want to, just that they have to prove that they've seen recent valid tx that have been posted to the chain.

This would be trivial for legitimate clients, official or otherwise, but for a botnet running without having the blockchain it would be impossible to fabricate, so they'd be forced to shut down or play fair. The only time "tx lists" would even be used at all is if:

1: Some other miners wanted to "lazy mine" with no blockchain, and are willing to trust full nodes to provide valid tx for them to mine.

AND

2: Miners willingly want to give up their tx to others, or have tx they don't want, and would like a facility for dumping them on lazy miners.

However, while this feature isn't necessary for booting a botnet, it would weaken security since it requires that the "dumb nodes" blindly trust full nodes, so "dumb nodes" could be used to forward an attack. For a normal, fully equipped miner, tx forwarding would be basically useless unless they have a major incentive to dump tx.

I agree that we don't need to centrally control all of the miners, and that we don't have any business telling them which/how many tx to include or not. However, I think there's plenty of good reasons for preventing someone from sitting around doing getwork without actually having the blockchain, but it's easy to prove whether this is the case or not without putting restrictions on including tx. Revalin's suggestion (or similar) would only be relevant under very different circumstances, which are much more unlikely to occur than someone freeloading with an anonymous botnet.

This could be a bug in a modified mining code's transaction verification code, or (more likely) a malicious player.  Either way, I don't think responding is necessarily the best course of action.  If it's bad code, it's not really our problem to solve, and the developers are going to notice eventually.  If it's a malicious miner, then with 15% of the network it's no small attacker; and getting us to respond in some fashion is as likely the goal as would be adding more power in the future.  Empty mining blocks were always going to be an issue, we knew this from the start, so this problem has been mentally hammered for two years.  There is nothing that we need do about it as a group (such as change the code) because by rejecting fee paying transactions they are harming themselves.  We don't wan't to alter the code to respond to this, because then we would be the reactionary group, responding to the attack vectors of an unknown malicious agent by potentially adding new attack vectors.

What we could do is publish the bitcoin addresses of the null block offenders, and both try to identify them as best we can, and (as individuals, not as a community) choose to delay transactions & blocks produced by that address.  Because the decision making process is based upon the propagation of a new block, even a short transmission delay in a majority of nodes will result in this malicious attacker's effective hashrate being reduced.  Which can function as a punishment for failing to respond to the social rules of the network, but does not require widespread code changes to deal with a particular attacker.  Users can choose whether to participate in the sanctions or not; just like they technically can already do if they have the coding skills to make the local changes.

The problem here is that the "Attacker" most likely isn't malicious in the sense of trying to take the network down. They just want free money from stolen computing power, and so far they're getting it. The real problem is that 15% of the BTC that would be going towards miners who actually maintain their rigs are going to someone who more than likely does not.

The other problem is, how the hell do you "block" 1.8 million unconnected stolen computers? If I started manually doing that now, by the time I was dead I'd still be nowhere near caught up. Requiring proof that you've seen the blockchain recently would automatically exclude anyone trying to cheat (at least via that route). Alternatively, if other people start figuring out they can make free coin by running a botnet sans the blockchain, pretty soon "mystery" will make up more like 30% of the network, and even more funds will be diverted from maintaining the legitimate side of the network. At the moment the reward for parasitic mining outweighs any potential losses from not including tx, and that's not likely to change much for at least 10 years.

Checking to see that a miner is using the blockchain is fairly trivial. If they're actually running the blockchain then there's no incentive for them to exclude tx (at the moment), and no need to babysit them to make sure they include some minimum number of tx, either (that's their choice, however arbitrary). The problem is the incentive for running without it, pure and simple.

[jetmine]

Don't you guys realise that you are doing a better job at undermining peoples trust on Bitcoin by discussing this non-issue relentlessly than "mystery jerk miner" is by not including tx's on the blocks he mines?

Very true.  +100

The same can be said about the accusation of operating a "botnet", while the facts do not support that theory.

People here speculate about how not processing transactions would benefit a botnet, and then conclude that it must be a botnet.

This lacks both facts and logic.  For the lack of facts, consider this:

The node at 88.6.216.9 did have a complete blockchain and did verify transactions, and also relay them.  It just did not include them into new blocks.  For an example of a relayed transaction, see http://www.blockchain.info/tx-index/3091519/ca264af6271cc40bf8b297e13fc2489bd9564dd18b9e60a1bd4d19046086995b

When it was still operating at 95.120.241.167, it even did include transactions into blocks.  Both free and paid.  For an example, see https://www.blockchain.info/block-height/166667 (the last transaction is a free one).

It stopped including transactions in March.

The decision to not include transactions was made after support for transactions was already in place.  It is a political decision, not a technical necessity.

For the lack of logic, just think about how your own mining client works.  Most of you readers (yes, not all, but most), are mining at a traditional pool.  You consume very little bandwidth, you dont need to store the blockchain and you dont need to know or verify any TXs.  Yet your mining effort works towards blocks with transactions included.

You work successfully under exactly the same conditions, as (some of) you claim would force a botnet to exclude transactions.  (Some of) you say, that downloading the blockchain would make botnet victims suspicious, yet you do not need to download it yourself!  (Some of) you say, the steady influx of transactions would need lots of extra CPU processing and extra network bandwidth, yet you do not need to do that yourself!  (Some of) you even say that a full bitcoind must be in place, yet you do not need it on your own miners!

Your own mining client shows how the technical "problems" can be solved, and source code is available for everyhing (both mining client and pool service).

The assumption of the technical benefit is just plain false.  The only thing that supports the idea of a botnet, is the ever repeating posts here on bitcointalk.  Those who jump to early conclusions and keep reinstating this false idea, are the ones who do damage to bitcoin.

I wonder if you stand up as quick to apologize as you do to condemn.

[MoonShadow]

It's a DOS vulnerability, as discussed earlier, and it makes 51% attacks easier, even if the null-miners are not directly collaborating.

It's not a DOS vulnerablility unless the attacker can mine nearly all blocks empty, in which case he already has 50%+ of the network and can pretty much do anything.  If he has less than a majority of the network, he can't prevent valid transaction processing every 20 minutes or less anyway.  So no, that's not an issue in this context.

And how would it make 51% attacks easier for others?  Again, I think this is an over-reaction.  If you think that this is really a problem, work on a patch that allows miners to simply not forward valid transactions that flag on a user's personal blacklist based upon address or whatever other criteria.  If enough miners agree that this is an issue, they will incorporate your patch and the null-miners will be sanctioned by the community without a community wide action need be taken.

[DeathAndTaxes]

Of course you should be allowed!  That's well established as a miner's right.  Similarly it is also a relay node's right to exclude blocks they consider insufficient.  Check and slightly imperfect balance.

To it is my right to charge what I consider fair but if I do you will exclude the block and I lose everything.  Honestly you think miner's will support this.

It would be like saying:
You can open a restaurant and charge what you want but if 5 people are waiting and you don't feed at least 25% of them you will be shutdown.  Oh an BTW some of them want to eat for free and some want to eat for a penny.  But sure you can charge what you want.

No dice.  Any system that forces me as a miner to use MY HASHPOWER purchased with MY CAPITAL and running on MY ELECTRICITY to include fee and "essentially free" transactions is a non-starter.  The "solution" is worse than the problem.

Relay nodes already enforce some minimum fees; those were actually reduced last year.  Right now it's probably best for free txes to be as low as possible - low friction will help growth in ways much more significant than some 0.01BTC fees.  That said I encourage you to institute this policy.  The completely homogenous one-formula-fits-all isn't good for the network.  I think we would be much better off with a gradient of minimum fees among miners instead of the very sharp instantly accepted / transaction limbo that we have now.

You "encourage me" but at the same time advocate a system which would orphan my blocks if I do what you are encouraging.  Thanks for your encouragement. 

Even a 1 satoshi minimum would get people to at least start paying fees, and pave the road for the future where free/1-satoshi transactions are slow (just a few miners supporting them) and reasonable fees are required for fast processing.  It's probably helping your goals on the whole.

But I don't think we should make a universal mandatory minimum fee a rider on the fix for null blocks; that's really a separate issue.  In my opinion we should just try to solve the latter in a way that doesn't make the former worse or cause other collateral damage.

I am not advocating that.  I am adovcating I set the minimum fee for my hashing power ONLY with no changes to the protocol and you are advocating orphaning my blocks if I do that.

[N12]

If those tx are free I am forced to include them or have my block rejected? 

Where do people get this idea you have the right to have your free or cheap ass (1/10th of a penny) tx forced into the next block.

Becasue we already pay you greedy miners with 33% yearly inflation.

Not enough?

[MoonShadow]

If those tx are free I am forced to include them or have my block rejected? 

Where do people get this idea you have the right to have your free or cheap ass (1/10th of a penny) tx forced into the next block.

Becasue we already pay you greedy miners with 33% yearly inflation.

Not enough?

That's a matter of opinion.  Bear in mind that including transactions into a block does add some nominial amount of additional processing load for the miner, so it's not free for him.  This null-block miner might simply be a pool that has a higher fee requirement for inclusion than the common amount.  That is his right.  And the only response for the other miners is to increase their own mining power.  This "arms race" benefits the security of the blockchain, even if a not-insignificant minority of blocks are null-set.  I'd be more productive if the miner in question published the minimum fee expected, if that is the case, but it's not necessary.

[DeathAndTaxes]

For the lack of logic, just think about how your own mining client works.  Most of you readers (yes, not all, but most), are mining at a traditional pool.  You consume very little bandwidth, you dont need to store the blockchain and you dont need to know or verify any TXs.  Yet your mining effort works towards blocks with transactions included.

You work successfully under exactly the same conditions, as (some of) you claim would force a botnet to exclude transactions.  (Some of) you say, that downloading the blockchain would make botnet victims suspicious, yet you do not need to download it yourself!  (Some of) you say, the steady influx of transactions would need lots of extra CPU processing and extra network bandwidth, yet you do not need to do that yourself!  (Some of) you even say that a full bitcoind must be in place, yet you do not need it on your own miners!

Your own mining client shows how the technical "problems" can be solved, and source code is available for everyhing (both mining client and pool service).

The assumption of the technical benefit is just plain false.  The only thing that supports the idea of a botnet, is the ever repeating posts here on bitcointalk.  Those who jump to early conclusions and keep reinstating this false idea, are the ones who do damage to bitcoin.

In a dumb miner - pool model the fact that the miner doesn't do the workload doesn't mean the workload doesn't exist.

One getwork is ~400 bytes.  Some have speculated mystery has 1.8 million nodes.  To use a traditional full pool server the bandwidth requirements alone would be 5.7Gb and 1.8 million connects every long poll.  Say he staggered those getworks to supply all nodes with work over a 10 second period (less revenue btw).  You are still talking about almost 200K simultaneous connections, 750Mbps, and enough computing power to generate 200K block headers a second.

If you want more insight ask Tycho what kind of load even small botnets do to a pool server.  Now with enough computing power (we are talking a dozen or so high end load balanced servers on a 1 Gbps connection) a 1.8 million node botnet "could" operate in a pool-miner model.  However now compare all that cost vs the incremental reward of .... ready for it ....about 2 cents per block.