Miners that refuse to include transactions are becoming a problem

[DeathAndTaxes]

Currently, it is possible to get a tx pushed with no fee, but the commit times are horrible and there's no guarantee that it will even work. Nothing really changes here, and miners can still do whatever they want, however stupid it may be.

Haplo please learn the protocol before you try to "fix it".

Unless your tx is considered spam (under the 1 day & 1 BTC threshold)  there is no need for a fee and confirmation times are routinely the next block.

[1715037296]

1715037296

[1715037296]

1715037296

[1715037296]

1715037296

[1715037296]

1715037296

[theymos]

Whatever list is used by 51% of miners will orphan out any other chain. 

Why?

[Haplo]

Currently, it is possible to get a tx pushed with no fee, but the commit times are horrible and there's no guarantee that it will even work. Nothing really changes here, and miners can still do whatever they want, however stupid it may be.

Haplo please learn the protocol before you try to "fix it".

Unless your tx is considered spam (under the 1 day & 1 BTC threshold)  there is no need for a fee and confirmation times are routinely the next block.

True, although it probably won't be next year. Currently what I said does hold true for "spam" tx, and next year the same will probably hold true for all tx. However, if the 0 fee txs are passed to bots, then the penalty for not including a fee ends up falling on the legitimate miners, who could otherwise extract a higher fee and from a larger number of tx.

[DeathAndTaxes]

Whatever list is used by 51% of miners will orphan out any other chain.  

Why?

51% of miners follow a list which says tx A must be included.
49% of miners follow a list which don't.

It is no different than a 51% "attack".

The 49% may solve a block BUT that block will be ignored by the miners in camp "A".  Eventually the A camp will pull ahead and orphan all the blocks of the 49%.

So say current block height #123
Both camp A & camp B work on block #124.
A miner in camp B solves block #124 but it is missing tx from "A" list.
Miners in camp A invalidate block #124 and continue to build off of #123.

Due to variance "camp b" may pull ahead temporarily but just like a 51% attack eventually camp A will pull ahead and orphan the chain supported by the "B" miners.

This means whatever list is used by 51% must be used by all miners.  That gives massive power to those who control a lot of hashing power like the Big 3 pools.  If the Big 3 pools make "list A" they even miner is defacto forced to use it.  To not use it is economic suicide as any block the mine which is not in compliance with the "list A" risks being orphaned.  No miners is going to take that chance.  All miners will be forced to follow the rules set by the Big 3.

The Big 3 then could decide not to share the list with all miners (permanent mining exclusion unless the excluded gain 51% of hashing power), charge fees for inclusion (think of it as a pool fee but they don't even have to provide a pool), or share "hidden tx" between themselves to orphan out miners not paying to mine.  If only those paying for access to the list know of the list and hidden tx then outside miners can't even hope to compete by including all tx (even every free one) blindly hoping to make a compliant block.  Even if they include all tx they know of they will be missing the hidden ones.

[theymos]

The 49% may solve a block BUT that block will be ignored by the miners in camp "A".

That's not the current behavior, and that behavior is not part of gmaxwell's proposal. Blocks won't get rejected/discouraged for not including transactions. All blocks just need to include an extra "proof of validation".

[westkybitcoins]

The 49% may solve a block BUT that block will be ignored by the miners in camp "A".

That's not the current behavior, and that behavior is not part of gmaxwell's proposal. Blocks won't get rejected/discouraged for not including transactions. All blocks just need to include an extra "proof of validation".

Hmmm.

Under gmaxwell's proposal, the miner would need to include exactly the transactions specified by the person who verified that the transactions and the previous block were valid. So if blockchain.info published a set of 10 verified transactions, the miner would have to include all 10 of them.

Specified by whom?  The person who solved the last block?  If so why aren't the transactions in their block?

Anyone could publish lists of transactions that should be included in the next block along with the necessary "verification proof". Bitcoin nodes themselves could publish lists using a new network message.

When there are conflicting lists?

The miner could choose the list to use. Someone could even publish valid lists that are always empty, though rational miners won't use these because including a transaction is almost free.

You're misunderstanding the proposal. Anyone who is verifying the chain can publish a list. Lists can use any fee rules. For example, Bitcoin Block Explorer might publish lists at /q/getValidationProof/minFee, which would only list transactions with a fee above minFee. Under this system, dumb miners would be able to choose essentially arbitrary fee policies; they'd just be forced into mining only properly-verified transactions and blocks.

I'm not getting this proposal at all.

First... these tx-lists can be published by anyone verifying the chain? Bitcoin nodes can publish lists? Just any ol' node can do that? Considering the prospect of malicious nodes trying to crack this new feature, isn't that tempting fate, unless such lists can be ignored? And if they can, what's the point of having them?

And then, miners get to choose which list to use. So why wouldn't a miner just publish his own list, then immediately use it?

I'm failing to see exactly how all this is supposed to realistically handicap a miner and "encourage" them to put transactions into their block.

And if it DOESN'T do that, then what's the point? What is this "proof of validation" supposed to do if it doesn't result in transactions being forced into a block?

And if it DOES force transactions into a block, but I produce a block without those transactions, how could my block NOT be rejected?

[istar]

This MUST be solved or Bitcoin could become useless and just creating a higher transaction fee is not good enough.

We do not know what the goal of this miner is.
It might not be to earn Bitcoins but to devalue Bitcoin.
This could be the work of someone who has or are about to create a alternate cryptocurrency.

The solution/s has to be designed to prevent or increase the cost for a worst case scenario.
We cannot design a solution for "the best case" which is, he cares about the value of Bitcoin.

If you do not realize this, your opinion is not of much use.

Another part of a solution could be to add a some real proof of stake for miners, but it would not be enough.

[VVS dump]

This MUST be solved or Bitcoin could become useless and just creating a higher transaction fee is not good enough.

We do not know what the goal of this miner is.
It might not be to earn Bitcoins but to devalue Bitcoin.
This could be the work of someone who has or are about to create a alternate cryptocurrency.

The solution/s has to be designed to prevent or increase the cost for a worst case scenario.
We cannot design a solution for "the best case" which is, he cares about the value of Bitcoin.

If you do not realize this, your opinion is not of much use.

Another part of a solution could be to add a some real proof of stake for miners, but it would not be enough.

we dont even know this "miner" has malicious intent. it could have been just someone building some mining asic or autonomous fpga controller that has constraints in size. I mean it's not too hard to write your own mining controller.

[Technomage]

The reason I put this thread to the dev & tech forum was that I want to explore solutions to this, not start another thread where we speculate who it is. There is a massive thread on that at the mining forum. I don't really think that it matters who he is, we should always work on the worst case assumption and I honestly don't think there are any good cases here.

Personally I think that it's very likely to be a botnet, simply because adding transactions is the standard way to mine and also very cheap under any regular or even semi-regular mining configuration. One thing is for sure, our mystery miner gets major benefits from not adding any transactions. This is clear. The only good explanation to why he could get benefits from doing this is a botnet.

Satoshi saw this scenario as unlikely but he did have plans in mind. I say we follow on his footsteps and make it so that a portion of announced transactions must be in the next block or it's refused by the network. This portion can be small, we just need to make it big enough that our mystery miner is forced to at least work with the blockchain. This would be done in a way that a portion must be included even if all of the announced transactions are without tx fees.

This doesn't invalidate the tx fee market, the miners are free to do with the rest of the transactions as they wish. The forced portion can be fairly small and then the rest of the transactions will follow regular rules that miners can set however they wish.

I have no clue how to make this happen from a technical perspective, I'm simply throwing it out there based on what Satoshi said. If you guys have ideas on how to do this or if you want to discard it altogether, tell us.

[VVS dump]

btw, you're trying to fix a problem that isnt there. me outlining possible causes for that miner to sign empty blocks was just a way of showing that this might not be desired behaviour, or that it might simply be due to limits, but not with malicious intent. But as stated before, bitcoin is free, and therefore impounding measures prohibiting people from chosing what they'd like to integrate is bs IMHO.

besides that, your idea of having miners forcefully integrate a certain ratio of the set of unconfirmed transactions is a bad bad idea. consider this:
I hate my pool because the op is greedy and charging ridiculous fees. So whenever I find a real_target share I simply wait 10 seconds before submitting it, and use those ten seconds to send out many microtransactions. some bitcoind's will get that quite in time, and consider the new block invalid because it did not meet the quota. because I have a pool that pays out orphans, I'll still get paid(unless those 10 seconds make the difference and someone else finds the block, but that'd impair my profits only minimally), the transaction costs for spamming are minimal(or even none at all if I choose to use >1d >1btc inputs), and the pool admin gets ripped of another 50 bitcoins. With so much greed, bad behaviour and willing malevolence in the bitcoin community, this is a likely scenario.

btw, I think forcing the miners to include a minimum of tx is a kind of socialism.

[VVS dump]

The reason I put this thread to the dev & tech forum was that I want to explore solutions to this, not start another thread where we speculate who it is. There is a massive thread on that at the mining forum. I don't really think that it matters who he is, we should always work on the worst case assumption and I honestly don't think there are any good cases here.

Personally I think that it's very likely to be a botnet, simply because adding transactions is the standard way to mine and also very cheap under any regular or even semi-regular mining configuration. One thing is for sure, our mystery miner gets major benefits from not adding any transactions. This is clear. The only good explanation to why he could get benefits from doing this is a botnet.

Satoshi saw this scenario as unlikely but he did have plans in mind. I say we follow on his footsteps and make it so that a portion of announced transactions must be in the next block or it's refused by the network. This portion can be small, we just need to make it big enough that our mystery miner is forced to at least work with the blockchain. This would be done in a way that a portion must be included even if all of the announced transactions are without tx fees.

This doesn't invalidate the tx fee market, the miners are free to do with the rest of the transactions as they wish. The forced portion can be fairly small and then the rest of the transactions will follow regular rules that miners can set however they wish.

I have no clue how to make this happen from a technical perspective, I'm simply throwing it out there based on what Satoshi said. If you guys have ideas on how to do this or if you want to discard it altogether, tell us.

i'm totally against 'fixing' this non-issue, but out of curiosity, what portion would you suggest?

if it's a low amount like 2%, doesn't that only solve 2% of your 'problem'?

that wouldnt fix the problem at all. patching bitcoind to not relay transactions is an exactly two line patch changing 4 bytes. the bad or notso bad miner could simply apply that patch and have a hundred transactions present all the time. since he doesnt relay them, he can be 100% sure that he will be the one including them into a block, while recollecting the fees needed to pay for those tx. Result: we animate someone to do blockchain spam with senseless tx.

[MoonShadow]

This could be a bug in a modified mining code's transaction verification code, or (more likely) a malicious player.  Either way, I don't think responding is necessarily the best course of action.  If it's bad code, it's not really our problem to solve, and the developers are going to notice eventually.  If it's a malicious miner, then with 15% of the network it's no small attacker; and getting us to respond in some fashion is as likely the goal as would be adding more power in the future.  Empty mining blocks were always going to be an issue, we knew this from the start, so this problem has been mentally hammered for two years.  There is nothing that we need do about it as a group (such as change the code) because by rejecting fee paying transactions they are harming themselves.  We don't wan't to alter the code to respond to this, because then we would be the reactionary group, responding to the attack vectors of an unknown malicious agent by potentially adding new attack vectors.

What we could do is publish the bitcoin addresses of the null block offenders, and both try to identify them as best we can, and (as individuals, not as a community) choose to delay transactions & blocks produced by that address.  Because the decision making process is based upon the propagation of a new block, even a short transmission delay in a majority of nodes will result in this malicious attacker's effective hashrate being reduced.  Which can function as a punishment for failing to respond to the social rules of the network, but does not require widespread code changes to deal with a particular attacker.  Users can choose whether to participate in the sanctions or not; just like they technically can already do if they have the coding skills to make the local changes.

[Revalin]

You can't beat this by playing whack-a-mole.  Right now it's only a couple IPs but the problem should be solved generally.

I do think it needs to be solved.  While it's not significantly hurting the network now, it's a vulnerability and it should be fixed.  And ethically, I object to people making money when they're skipping out on their responsibilities.

I don't see the need for mandatory tx fees.  What's wrong with simply refusing to relay blocks that don't include at least ( ( n * 0.25 ) - 5 ) transactions, where n is the number of valid transactions broadcast since the last block?  Empty blocks would be allowed during slow times or when a block was recently sent (the -5), and the 0.25 would let you be reasonably selective about what you mine.

[MoonShadow]

What's wrong with simply refusing to relay blocks that don't include at least ( ( n * 0.25 ) - 5 ) transactions, where n is the number of valid transactions broadcast since the last block?  Empty blocks would be allowed during slow times or when a block was recently sent (the -5), and the 0.25 would let you be reasonably selective about what you mine.

Nothing, so long as this rule remains a choice of the user.  It can't be a change that breaks the current system.  For example, you can add a rule that prevents your node from forwarding an otherwise valid block or transaction by whatever criteria you choose; so long as you don't reject the block if it gets into the blockchain.  The rules for validity must remain the same, but you should be able to add whatever rules to further limit transmission that you like.

[sturle]

Why would a rational miner want to allow free tx forever?

Forever is unlikely, but for the next 100 years it is very rational.  If we want Bitcoin usage to grow, we need to facilitate free or very cheap transactions.  Otherwise people will just use PayPal or whatever, which is bad for everyone who mine coins.  As long as the block reward is more than one can reasonably expect from fees, it is completely rational to keep fees as low as possible without encouraging spam.  Discouraging spam and laundering is the only reason to have fees now, and for a long time into the future.

The mystery miner's goal is clearly out to devaluate Bitcoin by making it less useful due to slower transactions.  I doubt fees would change anything.  Has any of the mystery miner's block rewards been spent yet, btw?

[MoonShadow]

I'll be not shocked at all, if this turns out to be a bug in a minority client; such as in combined mining.

[DeathAndTaxes]

I don't see the need for mandatory tx fees.  What's wrong with simply refusing to relay blocks that don't include at least ( ( n * 0.25 ) - 5 ) transactions, where n is the number of valid transactions broadcast since the last block?  Empty blocks would be allowed during slow times or when a block was recently sent (the -5), and the 0.25 would let you be reasonably selective about what you mine.

If those tx are free I am forced to include them or have my block rejected? 

Where do people get this idea you have the right to have your free or cheap ass (1/10th of a penny) tx forced into the next block.

[MoonShadow]

Why would a rational miner want to allow free tx forever?

1) A charity accepting donations, mining for their own benefit.

2) a bank accepting free transactions of their own account holders, as an off network benefit.

3) Wal-Mart accepting free transactions for in-store purchases.

[Revalin]

If those tx are free I am forced to include them or have my block rejected?

If 100% of transactions coming through are free you would be forced to include some, but realistically there are always enough paid transactions that you can meet the 25% quota easily.  If you only include 25% during an "all free transactions" hour, people will have an incentive to include a fee to get their transaction included sooner (4x sooner on average in this case).

25% is just an example; pick a value that will accommodate any reasonable miner's include/exclude policy.

[MoonShadow]

You can't beat this by playing whack-a-mole.  Right now it's only a couple IPs but the problem should be solved generally.

I do think it needs to be solved.  While it's not significantly hurting the network now, it's a vulnerability and it should be fixed. 

It's not a vulnerability.