Miners that refuse to include transactions are becoming a problem

[wachtwoord]

If those tx are free I am forced to include them or have my block rejected? 

Where do people get this idea you have the right to have your free or cheap ass (1/10th of a penny) tx forced into the next block.

Becasue we already pay you greedy miners with 33% yearly inflation.

Not enough?

This times 1000.

The 50 BTC block reward is for including transactions. It is money that comes from everyone that owns Bitcoins (in the form of inflation) and you should provide the required service. If we would change the protocol to force you to do so or not get paid this would not be socialistic like some propose because it is not mandated by a central government but in fact by a majority of the people.

[1713855379]

1713855379

[1713855379]

1713855379

[1713855379]

1713855379

[1713855379]

1713855379

[Revalin]

gmaxwell's proposal is simply that in order for a block to be valid they have to prove that they have access to the blockchain.  ...  Revalin's suggestion (or similar) would only be relevant under very different circumstances, which are much more unlikely to occur than someone freeloading with an anonymous botnet.

I consider gmaxwell's approach to be short-term.  It will probably solve the current problem - whoever this is will notice their blocks are being orphaned and do something to fix it - and while it's intrusive (it requires protocol changes), it has little chance of other collateral damage.

I'm looking at ways to solve it in a more general manner to improve the long-term security of the network.  My proposed change is much simpler but the consequences are complicated.  I don't think D&T's fears will actually happen in reality - I think the economic incentives will actually result in overall more fees being paid and more latitude for miners to set them - but it's not easy to model, and obviously we disagree.

I also don't consider it a foregone conclusion that it's a botnet.  They have to be on the network to receive prior blocks and transmit new ones.  Why would someone go 80% of the way when they could implement a 100% functional pool (or just use any public pool) and draw less attention?  I don't think it's a deliberate attack either because there are much better ways to leverage that much hashpower.

I have no idea what their motives are, and honestly it doesn't matter.  I just want to create the economic incentive where they must fully participate or not get paid, which will definitively, and permanently, solve the problem.

[Revalin]

If you want more insight ask Tycho what kind of load even small botnets do to a pool server.  Now with enough computing power (we are talking a dozen or so high end load balanced servers on a 1 Gbps connection) a 1.8 million node botnet "could" operate in a pool-miner model.  However now compare all that cost vs the incremental reward of .... ready for it ....about 2 cents per block.

This is an excellent argument for pools disallowing clients which aren't turning in enough results to justify their bandwidth.  If low hashrate clients (botnet or not) want to participate, they can have one guy get work and distribute it to the others.  Letting them collect 50BTC per null block to save the pool the awkward responsibility of turning away unprofitable workers is completely backward.

[Haplo]

The decision to not include transactions was made after support for transactions was already in place.  It is a political decision, not a technical necessity.

For the lack of logic, just think about how your own mining client works.  Most of you readers (yes, not all, but most), are mining at a traditional pool.  You consume very little bandwidth, you dont need to store the blockchain and you dont need to know or verify any TXs.  Yet your mining effort works towards blocks with transactions included.

You work successfully under exactly the same conditions, as (some of) you claim would force a botnet to exclude transactions.  (Some of) you say, that downloading the blockchain would make botnet victims suspicious, yet you do not need to download it yourself!  (Some of) you say, the steady influx of transactions would need lots of extra CPU processing and extra network bandwidth, yet you do not need to do that yourself!  (Some of) you even say that a full bitcoind must be in place, yet you do not need it on your own miners!

Your own mining client shows how the technical "problems" can be solved, and source code is available for everyhing (both mining client and pool service).

The assumption of the technical benefit is just plain false.  The only thing that supports the idea of a botnet, is the ever repeating posts here on bitcointalk.  Those who jump to early conclusions and keep reinstating this false idea, are the ones who do damage to bitcoin.

I wonder if you stand up as quick to apologize as you do to condemn.

If what you're saying is true, and I'm not knowledgeable enough about the various BTC miner software versions to know, then gmaxwell's proposal would not work at all, and would be a huge pain since it requires a protocol change.

It seems to me that if you're right, then it's most likely a bug, since it's inconceivable that anyone would purposefully omit tx unless as T&D stated it was to maybe to reduce server loads on an already stretched system. Unless "mystery" steps forward, assuming it's even one person, we couldn't know their intentions.

Also, if this is the case, then the method proposed by Gavin/Revalin would be necessary. If a block has far fewer tx than the average, then it gets delayed, and if it has only 1 or none it may be excluded altogether.

There are several reasons why the "I'm entitled to charge whatever I want" argument has no weight.

1: I can charge $1million USD for a single grain of sand on ebay. Nobody will buy it (I would hope), but I still have to pay for posting it at auction. In BTC if you do the same thing you still get paid, at the expense of everyone using the system.

2: Gavin/Revalin's system still gives miners a good deal of room in choosing what they want to charge. As long as the threshold for delay isn't too restrictive, and the miner is charging a reasonable enough fee that they actually get some business, they shouldn't be excluded or delayed. Unless your business model is crap, there's no penalty. Also, since it's not actually part of the protocol, there's no obligation of any party to delay or reject a block, nor any obligation to accept any block, and it could potentially be adjustable by the user. I think, however, that the sentiment of most miners is that they also are not interested in supporting cheapskates.

3: The blockchain is 2 fucking gigabytes, and climbing at an absurd rate. If I have to download a 2GB blockchain, and 15% (or more) of all the blocks are empty or nearly empty, then why have I wasted my time downloading 300MB+ of that? The only purpose of the blockchain is to be able to securely verify transactions that have occurred, and empty or nearly empty blocks add filesize without contributing to the purpose of the data. At the rate things are going it's already going to take further development just to figure out how to keep the size reasonable, and the last thing we need is a bunch of retards being paid in inflation to spam it up faster.

Finally, if it is just a bug in one of the clients rather than someone trying to take shortcuts, then implementing a screening system would make it obvious to the users of that client that something is wrong. In that case it probably wouldn't take long for it to be fixed. If nothing is done, then we have no way of knowing.

[jetmine]

In a dumb miner - pool model the fact that the miner doesn't do the workload doesn't mean the workload doesn't exist.

One getwork is ~400 bytes.  Some have speculated mystery has 1.8 million nodes.  To use a traditional full pool server the bandwidth requirements alone would be 5.7Gb and 1.8 million connects every long poll.

The getwork size is the same, whether transactions are included or not.  The quantity of getworks is the same too, whether or not transactions are included.

If your calculations prove anything, then that its not a botnet.  "Because its too difficult to manage even a moderade sized botnet (ask Tycho)" - your own words paraphrased.

The fact that transactions are not included does not add one single jota of support towards/against the botnet theory.

[Haplo]

3: The blockchain is 2 fucking gigabytes, and climbing at an absurd rate. If I have to download a 2GB blockchain, and 15% (or more) of all the blocks are empty or nearly empty, then why have I wasted my time downloading 300MB+ of that? The only purpose of the blockchain is to be able to securely verify transactions that have occurred, and empty or nearly empty blocks add filesize without contributing to the purpose of the data. At the rate things are going it's already going to take further development just to figure out how to keep the size reasonable, and the last thing we need is a bunch of retards being paid in inflation to spam it up faster.

Have you looked at the size of an empty block? If 15% of the current blocks are empty, they will account for about 5.50 megabytes.

Edit: If the entire block chain was empty blocks, it would be about 36.7 megabytes. Do you really think empty blocks being considered spam is a reasonable argument?

In a dumb miner - pool model the fact that the miner doesn't do the workload doesn't mean the workload doesn't exist.

One getwork is ~400 bytes.  Some have speculated mystery has 1.8 million nodes.  To use a traditional full pool server the bandwidth requirements alone would be 5.7Gb and 1.8 million connects every long poll.

The getwork size is the same, whether transactions are included or not.  The quantity of getworks is the same too, whether or not transactions are included.

If your calculations prove anything, then that its not a botnet.  "Because its too difficult to manage even a moderade sized botnet (ask Tycho)" - your own words paraphrased.

The fact that transactions are not included does not add one single jota of support towards/against the botnet theory.

*sigh*, well if that's true then either someone is charging a ridiculous fee, or there's a bug in some version of the miner software. If the difference in load and everything else between including tx and not including tx is basically insignificant, then there should be no reason for anyone to be purposefully excluding them.

On the other hand, if an empty block is really ~1/55th the size of a block with lots of tx, then it might really be someone trying to cheap out on bandwidth while still getting paid.

Either way, it ought to be stopped.

[DeathAndTaxes]

3: The blockchain is 2 fucking gigabytes, and climbing at an absurd rate. If I have to download a 2GB blockchain, and 15% (or more) of all the blocks are empty or nearly empty, then why have I wasted my time downloading 300MB+ of that? The only purpose of the blockchain is to be able to securely verify transactions that have occurred, and empty or nearly empty blocks add filesize without contributing to the purpose of the data. At the rate things are going it's already going to take further development just to figure out how to keep the size reasonable, and the last thing we need is a bunch of retards being paid in inflation to spam it up faster.

Nonsense.  The blockheader is tiny.  99%+ of the blockchain is transactions.  The number of blocks is based on time.  Empty or not the block chain will grow by (300 bytes) * 6 * 24 * 365 bytes per year = 15MB annually  (roughly double that if you include the only requires tx - the coinbase).  

That cost is fixed.  The only thing that would change that is a avg block time other than 10 minutes.

Transactions make up the vast majority of the blockchain.  This is why downloading blockchain is fast initially (first 50,000 blocks downloads within minutes but then it slows down).  Those blocks are mostly empty (yup even Satoshi mined empty blocks).

Finally, if it is just a bug in one of the clients rather than someone trying to take shortcuts, then implementing a screening system would make it obvious to the users of that client that something is wrong. In that case it probably wouldn't take long for it to be fixed. If nothing is done, then we have no way of knowing.

No botnet or even conventional TH/s farm would be running the standard client.  Especially not a farm (conventional or not) which is ignoring tx.

[DeathAndTaxes]

The getwork size is the same, whether transactions are included or not.  The quantity of getworks is the same too, whether or not transactions are included.

If your calculations prove anything, then that its not a botnet.  "Because its too difficult to manage even a moderade sized botnet (ask Tycho)" - your own words paraphrased.

The fact that transactions are not included does not add one single jota of support towards/against the botnet theory.

No please listen.  USING A POOL SERVER WITH A BOTNET IS NOT VIABLE DUE TO THE COST OF GETWORKS.  It was your "theory" that the nodes not being able to process tx was bogus because most miners don't.  However conventional pool miners need a pool.  I was just showing the math that current tx fees come nowhere near the cost needed to support a pool.  They don't for a botnet, they don't even for a small conventional pool.

The nodes likely are running a modified stipped down version of bitcoind.  It doesn't keep the blockchain, it doesn't need a db, it doesn't even need logs.  It simply connects to peers and looks for inv messages.  When inv message occurs the nodes internally use the last block hash, current time, no tx, the simplified coinbase, build a merkle tree consisting of 1 tx, build block header and start hashing.  Likely not all nodes are even running this.  To isolate the botnet only a "few" (as a % of total nodes) would need connections to bitcoin network.  They could rely new block notifications in p2p fashion to the rest of the swarm.

Essentially they are listen ony nodes.  They only broadcast to bitcoin network when they find a block.  Which if the avg node is 10 MH/s would only be once every 20 years (per 10 MH/s node).

Or that is how I would do it.  

[jetmine]

The problem with delaying blocks is the following.  As a simple user, you may feel personal satisfaction by not relaying the block and it may not affect you in any other way.  But as a miner, you are in a difficult situation when you receive a block that you dont "like".

For as long as you plan to dislike it, you cannot mine on top of it - because your potential next block would be waste without the "evil" block. It is chained to it and eventually you will have to publish the "evil" block along with your own next block.  To make things worse, your chances of success are higher when you relay the "evil" block as soon as possible.  Otherwise you not only depend on your own luck (difficult enough), but also on the luck of the "evil" block (which you have tried to diminish).

Therefore, if you dont like the block, you must not mine on top of it.

This is a very risky strategy.  Other miners may not be as picky as you are, and then your mining efforts may end up waste.  You are intentionally mining on an old block, knowing that a newer block exists.  This can only play out well, when you are absolutely sure that most other miners ignore the block too.

Every time an "evil" block appears, the chances of an orphan produced by (self-proclaimed) "legit" miners raises.  It can be seen as a seed for orphans.  And I mean an orphan of "legit" work, not only the "evil" work itself.

(Remember that I am talking all this under the premise of the poposed intentional relay delay)

Although the percentage of seeds is only 15%, the chance of an orphan arising from it may be much higher.  It depends on the adoption of the protocol change and also on the individual configuration that each miner chooses.  We could very well face a situation where everyones own mining effort has a 50% to be orphaned after each seed.

Effectively every miner would pay a very high price for a just slightly better service.  At 15% seeds and 50% chance to orphan, EVERY "legit" miner would loose 7% income.

A high price to pay.

To pay for what?  Well for the slightly better service quality I suppose.  "Slightly better", because a TX can make it 15% faster into the chain.  However, there is also a negative effect, which is the constant rate of orphans and reorganization of the blockchain.  This could be seen as another kind of "Dos" or attack on bitcoin.  It makes 1 or 2 confirm transactions very unreliable, and can also affect 3 confirm TX or higher.

People who rely on low-confirm transactions will now have to wait CONSIDERABLY longer, not just 15% longer for the first confirm and 0% longer for every subsequent confirms.

We have already seen runs of up to 4 or 5 consecutive blocks with no TX, all from the same source.

The higher price paid by miners (in form of extra electricity with less BTC reward) may actually make the overall service quality worse.

[theymos]

And then, miners get to choose which list to use. So why wouldn't a miner just publish his own list, then immediately use it?

Regular miners wouldn't use these lists at all. They can do whatever they want. The lists are meant to stop "dumb" miners that can't verify the chain for themselves from just blindly building onto the longest chain. This proposal makes all miners verify the chain or get someone else to do it. If this "mystery miner" is already verifying the chain properly, then there's no problem: he can set fees to whatever he wants.

[DeathAndTaxes]

And then, miners get to choose which list to use. So why wouldn't a miner just publish his own list, then immediately use it?

Regular miners wouldn't use these lists at all. They can do whatever they want. The lists are meant to stop "dumb" miners that can't verify the chain for themselves from just blindly building onto the longest chain. This proposal makes all miners verify the chain or get someone else to do it. If this "mystery miner" is already verifying the chain properly, then there's no problem: he can set fees to whatever he wants.

But he can't.  If the majority of miners include 10 free tx on the "required list" for the next block then "mystery" (or any miner or pool) is required to include them or have that block invalidated by other miners.  He can't even exclude free tx without taking a risk his block will be orphaned.

Miners already have limited pricing power due to the fact that mining is a commodity business this ensures they have no independent pricing.  If clients know the list used by majority of miners includes free tx there is little reason for them to include a fee.

[JoelKatz]

The nodes likely are running a modified stipped down version of bitcoind.  It doesn't keep the blockchain, it doesn't need a db, it doesn't even need logs.  It simply connects to peers and looks for inv messages.  When inv message occurs the nodes internally use the last block hash, current time, no tx, the simplified coinbase, build a merkle tree consisting of 1 tx, build block header and start hashing.  Likely not all nodes are even running this.  To isolate the botnet only a "few" (as a % of total nodes) would need connections to bitcoin network.  They could rely new block notifications in p2p fashion to the rest of the swarm.

If this is the issue, and there's a good chance that it is, there are three possible solutions:

1) Make it more difficult for listening nodes to get what they need to process a block with no transactions.

2) Make it easier for listening nodes to get what they need to process a block with transactions.

3) Raise the transaction fees so that there's enough incentive for botnet operators to get transactions into their blocks.

[finway]

It's all about incentives.

Mining & killing bitcoin? 
Make no sense.

[theymos]

But he can't.  If the majority of miners include 10 free tx on the required list then "mystery" (or any miner or pool) is required to include them or have that block invalidated by other miners.  He can't even exclude free tx without taking a risk his block will be orphaned.

You're talking about the "discouragement" proposal. I am not.

Discouraging blocks should be a last-resort tool. The incentives are wrong: building onto a discouraged block has no extra cost for miners, but discouraging a block has significant extra costs.

[jetmine]

Or that is how I would do it.  

Interesting theory, I didnt think of this.

However, it does not address the fact that the blocks all come from the same IP (or small group of IPs).  In your scenario, it would be a random one out of 1.8M IPs.

[DeathAndTaxes]

True.  If the botnet operator is paranoid he may wish to minimize the connections to the rest of the bitcoin network (to keep the ip of nodes unknown).

In that case take my theory above and modify it so there is a single bitcoind active at one time.  When it detects a new block it notifies the rest of the swarm.  Using a mesh p2p like protocol the "gateway" nodes notifies 50 nodes who notify 50 each, who notify 50 each.  Each node is aware of the current gateway.  When it finds a block it relays it to the gateway who broadcasts it to the bitcoin network.

Really that would be sub optimal but very isolated.  If one takes the gateway down the operator promotes another node to the gateway.  By keeping the bulk of the swam unknown and with no direct communication with bitcoin network it is opaque and dark.

If the botnet has 1.8 million nodes and only one is active as the gateway at a time shutting it down would be tough.  Even if the ISP did shut it down.  Well there are 1,799,999 more nodes to go.

Likely the next question is "If D&T is right, why doesnt' the gateway simply include txs like any other node would?".  A little thinking I think will reveal a very obvious reason.

[Haplo]

Transactions make up the vast majority of the blockchain.  This is why downloading blockchain is fast initially (first 50,000 blocks downloads within minutes but then it slows down).  Those blocks are mostly empty (yup even Satoshi mined empty blocks).

The problem with delaying blocks is the following.  As a simple user, you may feel personal satisfaction by not relaying the block and it may not affect you in any other way.  But as a miner, you are in a difficult situation when you receive a block that you dont "like".

For as long as you plan to dislike it, you cannot mine on top of it - because your potential next block would be waste without the "evil" block. It is chained to it and eventually you will have to publish the "evil" block along with your own next block.  To make things worse, your chances of success are higher when you relay the "evil" block as soon as possible.  Otherwise you not only depend on your own luck (difficult enough), but also on the luck of the "evil" block (which you have tried to diminish).

D'oh.

If this is the issue, and there's a good chance that it is, there are three possible solutions:

1) Make it more difficult for listening nodes to get what they need to process a block with no transactions.

2) Make it easier for listening nodes to get what they need to process a block with transactions.

3) Raise the transaction fees so that there's enough incentive for botnet operators to get transactions into their blocks.

gmaxwell's solution might provide #1. If D&T is correct, then since gmaxwell's solution requires a miner to use the previous block's tx inputs (rather than just the hash) as verification, this would force mystery to either run more complete versions or bow out. The disadvantages are that it requires a change in protocol, and would increase the size of each block by an unknown amount.

For #2 it's been proposed to replace the blockchain with an acyclic directed graph, but I don't know enough about that to compare it in any way. Apparently this would make tx inclusion cheaper, although I have no idea how that would work, or how the filesize compares to the blockchain.

#3 unfortunately is probably not going to be viable for several years, maybe a decade. When the average number of tx increases by a large amount (maybe to 10k+) then even a small tx fee would generate substantial income, but as of right now, and for at least a few years yet, the block reward is much more significant, and has currency appreciation supporting it.

[JoelKatz]

For #2 it's been proposed to replace the blockchain with an acyclic directed graph, but I don't know enough about that to compare it in any way. Apparently this would make tx inclusion cheaper, although I have no idea how that would work, or how the filesize compares to the blockchain.

A possibly simpler solution would be a server-to-server version of 'getwork'. This would allow standalone mining clients to connect to any Bitcoin node and ask them which transactions they would include in a block they were mining. Right now, anyone can trivially design a standalone miner that mines blocks with no transactions. This would allow them to trivially design a standalone miner that mines blocks with transactions.

There's a "poisoning the well" problem though. If this is used primarily by botnets, it would be very tempting to include invalid transactions to cause them to mine invalid blocks. (The clients would have no way to validate them.) That would just cause them to go back to mining blocks with no transactions though.

[westkybitcoins]

But he can't.  If the majority of miners include 10 free tx on the required list then "mystery" (or any miner or pool) is required to include them or have that block invalidated by other miners.  He can't even exclude free tx without taking a risk his block will be orphaned.

You're talking about the "discouragement" proposal. I am not.

Discouraging blocks should be a last-resort tool. The incentives are wrong: building onto a discouraged block has no extra cost for miners, but discouraging a block has significant extra costs.

Then Haplo's clarification is correct? (Thank you for that, Haplo.) So it means instead of forcing miners to include transactions, they would instead be forced to access recent blocks and extract data from them.

What's still a little confusing about that on a conceptual level is this: mining ALREADY requires access to recent blockchain data, in the form of a hash of the prior block. Even if it required new code, couldn't the method already being used to distribute that prior-block-hash to all the nodes on a botnet be copied/modified to also distribute the newly-required "proof of verification?"

[MoonShadow]

The problem with delaying blocks is the following.  As a simple user, you may feel personal satisfaction by not relaying the block and it may not affect you in any other way.  But as a miner, you are in a difficult situation when you receive a block that you dont "like".

For as long as you plan to dislike it, you cannot mine on top of it - because your potential next block would be waste without the "evil" block. It is chained to it and eventually you will have to publish the "evil" block along with your own next block.  To make things worse, your chances of success are higher when you relay the "evil" block as soon as possible.  Otherwise you not only depend on your own luck (difficult enough), but also on the luck of the "evil" block (which you have tried to diminish).

Therefore, if you dont like the block, you must not mine on top of it.

Refusing to relay a valid transaction is different than refusing to recognize it as valid.  In theory, a talented coder can set relay rules for his client already.  There is nothing preventing nodes with additional relay rules from rising any more than there is nothing preventing a miner from setting additional block inclusion rules, but those only affect that miner and those who choose to make use of his code & ruleset.  A single miner who refuses to relay valid blocks produced by a competing mining pool amounts to little.  But if a significant number of nodes have a similar relay rule that delays the transmission of a valid block without transactions, then that rule becomes a defacto sanction against the miner who produces null-set blocks.