Bitcoin Forum

Currently we have a situation where a major solo miner, most likely a botnet, is mining Bitcoin blocks without adding any transactions to them. Approximately 15% of the network hashing power is currently creating blocks that do not have any transactions in them. There is a large thread about this in the mining forum: https://bitcointalk.org/index.php?topic=67634.0

So far it seems to be unclear why this is done, with most setups it shouldn't matter from a bandwidth/cpu point of view if he adds transactions or not. Clearly he has a reason for this and it could be either that he has found a way to mine less noticeable with infected PC's by not adding transactions to blocks OR he is doing this with malicious intent, perhaps trying to hurt Bitcoin. That could prove profitable to him if he short sells BTC at the same time, especially if he is planning to add more hashing power.

At the moment he has only 15% which is not a real threat to the network but what if it rises to 30%? Then one third of all blocks refuse to include transactions. Developers should take this seriously, it's in my opinion a high priority. We should start solving this before it becomes a bigger problem, and it definitely could become a major issue.

I'm not interested in speculation of why he is doing this. All of the possibilities are bad, some are just less bad than others. I'm interested in finding a solution, some way that we can fight this. Perhaps by making it less profitable to just mine blocks without adding transactions.

At first I was thinking we could treat empty blocks as invalid, but the attacker might just add a few pre-made offline transactions to his botnet code and still keep it more secretive than a full bitcoind.

The best  solution I can think of is to slowly lower the subsidy over many years, while developing specialized hardware to outperform botnets. :) I remember hearing somewhere that Satoshi wanted to upgrade the transaction fee system before he left, does anyone have any further reading?

I hadn't realized this problem had gotten so bad so quickly.  Clearly including transactions for fees isn't a great enough incentive.  I agree it should be addressed.

I've always thought relay nodes should take a greater role in security.  If they refuse to relay blocks that don't include at least 80% (measured by fees to prevent cheap transaction spam) of transactions they've seen since the last block it might solve it - no-transaction blocks would have very poor propagation and therefore would be highly-orphaned, giving a big economic incentive for them to play nice.

On the other hand I'm not sure how well this would behave in circumstances where there is a major network split with only a few nodes holding it together.  If the two halves of the net aren't seeing most of the other side's transactions and the couple relay nodes between are dropping both side's blocks (due to failures to include), could that worsen the problem?  The couple bridge-nodes would of course be relaying transactions across to both sides, but I'm concerned if it was compounded by other problems causing poor propagation.

Not relaying "smelly" blocks is a very interesting idea.

It doesn't have to be a binary relay/don't relay... you could immediately relay good blocks but wait a while before relaying bad blocks, and make the length of time you wait to relay based on how bad you think they are (maximum of maybe 10 minutes before you relay).

But... not relaying new blocks immediately might just encourage the bad guys to try to connect directly to as many nodes as possible, and that could be bad for network health.

What are the side effects of 15% of "extra" empty blocks?

1)  The difficulty increases.  This is good for the network, but bad for miners
2)  It takes longer for transactions to make it into blocks.

Am I missing something?  Neither one seems like THAT big of a deal.

You are missing something. The problem is that 15% is now, it's not necessarily that low forever. To me it seems like not adding any transactions is advantageous to a botnet miner for some reason (which is unknown) and this needs to be countered. Imagine if in a couple of months this percentage is 30, then our network is basically one third slower than it should be. It's a big deal that will make people lose confidence in Bitcoin's reliability.

We need to find a way to either give those who add transactions more carrot for doing that or give those that don't do that more stick. Whatever the solution is, we definitely need one.

People will lose confidence if the avg 1-confirmation time is 13 minutes instead of 10?  Really?  Especially given the 90% confidence interval is 3 to 31 minutes anyways.

I wouldn't consider it a problem for even 75% of miners to exclude all transactions. This would only slow transactions down by 10-20 minutes, and eventually the fee mechanism would sort it out.

I am a little concerned that they're not actually verifying transactions like they should be. This reduces the percentage of network power that an attacker needs in order to execute a "50%" attack. Applying gmaxwell's proposal (https://bitcointalk.org/index.php?topic=68396.0) might be worthwhile.

So how does that work then? If they let an invalid transaction into the block then that block will be rejected by the network and they have wasted their hashing effort.

ByteCoin

If these miners are building off of the longest chain without looking at any of the transactions (which may not be the case), then any attacker capable of getting a few blocks in a row can get these dumb miners to work for him. Full clients will reject these invalid chains, of course, but lightweight clients won't.

Now that I think about it more, it's probably more likely that the miners are getting the latest hash from Bitcoin nodes or websites, which might not be quite as bad as using the longest chain. Still weakens the network a lot, though.

It was addressed in the original Bitcoin design. Over time, the block reward will drop to zero, and transaction fees will make up the majority of the value of mining blocks. Nothing more needs to be done.

If it was perfection from the start then why do we need developers at all now?

Design =/= implemented code.

I simply find it unacceptable that my transactions and everyone elses transactions with Bitcoin can become slower thanks to a leeching botnet that simply profits from the blocks but doesn't do the one thing mining is actually useful for which is adding transactions to the blockchain.

Do we really want to let this leeching continue without doing anything about it? This is not just about inconvenience, if there is a solution we should put a stop to this. I'm quite confident that I'm not the only one who doesn't like this.

Maybe it isn't such a big threat but it does bother me a lot for many reasons. It's the kind of thing that would make Bitcoin users leave Bitcoin and start using a competing cryptocurrency. Of course there are no real cryptocurrency competitors yet but in 5 years Bitcoin might not have this luxury. Issues like this do matter.

Mining isn't just about adding tx to the block chain.  The hashing power still adds to the network security.  The economic value from that hashing power provides the incentive to not be disruptive.  There is no economic incentive to including txs.  People being cheap isn't a flaw.

If it becomes a big enough hassle, people will stop waiting for confirmations. Once that convenience problem turns into a security problem, the devs will probably start thinking hard about fixes.

Good points by both DeathAndTaxes and Kluge. I agree but still not happy about it. Perhaps the hash power of these no-tx botnets need to become a bit larger for people to really notice this issue. It could be that it doesn't even happen and we're left with this small issue that actually goes away eventually just as Syke said. But the thing is we don't really know.

This could explain why transactions are not included.  If this is a manufacturer doing a burn-in on an fpga cluster (http://www.sevensols.com/index.php?seccion=1410&subseccion=1428), omitting transactions would be a good way to avoid being accused of terrorism or whatever.

Then again the possibility of one manufacturer (and likely gov't contractor) having 15% of the hashing power is not good to think about.

As I understand it (and I probably understand poorly), the miners redo the transaction calculations only every few minutes. So, even if a miner is good intentioned and creating blocks with valid transactions, a transaction in the last part of the 10 minutes might be missed and will have to wait untill the next block. Assuming I have all that correct, could we not have a validation system where the node keeps track of the transactions and if a block is announced by a different node, it would check to see if transactions of a certain age are present and if not, reject the block? For instance, time 0 is solve time of last block and time 10 is solve time for current block. I keep track of transactions and if I see a new block at time 10 that doesn't include a transaction from time 5 (or before), I reject it. Is that doable or would we get into timestamp issues with not everybody agreeing on the time?

discussed a short while ago, i think it was shot down because it can easily cause forks.

It might be worth somebody contacting the ISP providing bandwidth to the errant node and asking them to investigate.

I don't really see the problem - as mining is planned to be a free market, it took actually longer than I expected until someone found out that the added complexity and electricity used pool-server wise with getworks and whatnot is probably not even matched by the income from these laughable transaction fees.

If you want to battle this, set the default transaction fee to 1 BTC for example - then it would be VERY worthwile to include transactions as well to collect the fees, even for Botnets and not including transactions would hurt a lot.

The only hurtful thing that empty blocks do, is increasing the difficulty by their hashrate used. On the other hand probably one day these coins will be spent and given back to the network - especially the "proof of stake" fans should be happy about someone getting 15% of all new coins.

tl,dr: higher transaction fees = more incentive to include transactions in blocks, as always.

It increases the time-to-first-confirm - if the "null blocks" rate grows to 90%, the average time to first confirmation will be nearly two hours!

It may also make double spends easier.

I'm quite surprised that all invested, relatively honest miners are not taking issue with botnets.  I've read a number of miner's opinions and many seem to be dangerously cavalier on the subject of botnet participation in Bitcoin's core infrastructure.

IMO.  A few bullets to address some posted opinions.

- "Cheap" botnet operators do not equate to legitimate participants in a free market. 
- Free loading botnets do not equal "Cheap," business operators.
- A botnet is not a business doing "good business."
- An absense of botnets does not equate to block chain insecurity.

Botnets are an unhealthy rationalization of Bitcoin security.  IMO, the discussion on the matter of botnets should not be limited to technical impact on Bitcoin.  Trading for security of one aspect (IMO, an unnecessary contribution to security) while trading away reputation of Bitcoin's core infrastructure as substantially supported by criminal entities.  Consumers, by and large, are particularly sensitive to the perceived honesty of an organization they do business with.  For the fledgling Bitcoin botnets are not something we want newcomers to worry about.  Whether or not the consumer's/user's perception of botnet risk is accurate, they will judge Bitcoin based on the knowledge that Bitcoin's existence partially relies on criminal activities.
Do you think the average consumer, let alone the coffee house, will like using Bitcoin to pay for a latte if they are aware of the above?

Botnets do not equate to business conducting good business as some have mentioned. 
Profit being the single interest of business necessitates law and regulation.  The no TX botnet is a prime example of a business where it's only consideration is profit that, currently, is not required to operate as a legitimate business thus giving advantage above all other parties who have decided to operate within legitimate parameters.  At worst botnet inclusion could recruit further exploitation.
On the flip side if botnets are not considered a threat in any respect, e.g. total network hashing, reputation, and instead are considered a positive contribution to block chain security they are indeed an insult to those who support Bitcoin through legitimate means, have expenses and follow the Bitcoin protocol in an honest manner.

Any entity that profits with all expenses being externalized, i.e. not obligated to pay for expenses, while typically great for the business is systemically unhealthy to the populous and the economy.  This is well understood economic principle.  A monopoly of a market in specific terms of having almost all expenses externalized is not a good example of a good free market.

Botnets, as the 15% total hashrate no TX botnet, is an exploitation of the temporary hashing subsidies of Bitcoin.  It's the truth even though there are multiple ways to rationalize botnet participation in Bitcoin.

A rebutal on "botnets"
1) We don't know it IS a botnet.
2) If it is there is absolutely nothing you can do to prevent them.  If botnets could be stopped they would.
3) If this is a botnet and they included tx you wouldn't even know.  The entire rationale can be boiled down to lots of hashpower with no txs = botnet which is dubious at best.
4) There is no need to "rationalize" anything.  Botnets exist and they will be attracted to potential profit.  Mining is profitable and thus botnets will participate. 

Some of us just don't feel like joining in the gnashing of teeth and assorted other nonsense.

Lets say you have 100% support of the Bitcoin community.  All miners, all pools, all users.  How do you stop botnets?
If you can't then I think you can see your entire post was pointless.

My concern with botnets isn't that they're botnets.

I'm concerned about two things:  they are both a) making money without doing the job they were paid to do (improve the network), and b) actually degrading the network while they do it.

I object to (a) not because it's a botnet (as you mention we don't even know if that's the case), but because they're not doing their job.  I object to (b) for obvious reasons.

We should fix the economic incentives so that miners have to do the job we're paying them to do instead of just collecting money for wasted work.

So what was all this nonsense ...


So botnets which play by your "rules" are good?
Also what if this isn't a botnet?  What if it is just a large hashing entity?  So botnets playing by your rules > someone operating under the protocol but not following yoru rules?

The entity (botnet or not) IS improving security of the network.  Network isn't degraded and economic incentive will increase as transaction fees become more important.  To call 1 tx blocks "wasted work" should a deep misunderstanding of how Bitcoin works.

+100

I don't always agree with you but on this particular issue I agree one hundred percent. This is not a "critical problem" but it is still an outrage. The current 15% of "freeleech" is still not that much but I'm confident that people will start voicing their opinions on this a lot more loudly if this percentage increases.

DeathAndTaxes is partially right, it's not complete freeleech. But other than that I don't agree with him.

Pay high transaction fees, such as 25% of transaction. If fees are higher than block rewards, the transactions will be excluded only by fool. Problem solved!

Why not put in a requirement of at least 1 valid transaction for a block to be valid?  If there are no pending transactions then there is a bigger problem happening.  You would just have to be very clear on what is a valid transaction to prevent penny spamming and such.

Thanks MM :)

It's not an issue now, but better to think ahead.
Miners and pools need to start publishing a schedule of minimum fees for transactions they will accept, and I guess their relay IPs.

Similar to the problems discussed in
https://bitcointalk.org/index.php?topic=51712.0;all

i don't think anything needs to be done or changed... we're already heading towards a partial fix in december, then continual adjustments to 'economic incentives' after that.

I think whoever is doing this is brilliant.

There's been concern for a long time that the bitcoin network will eventually weaken due to pitiful transaction fees providing no incentive for new/continued mining.

Not only has this zero-tx miner forced this issue to the fore, he has also provided an incentive for all of us to do, over time, what needs to be done to alleviate that concern.

Begin bumping up our transaction fees.

It's doubtful that small bumps in our fees will quickly end zero-tx mining. But it's also doubtful zero-tx mining is going to cause serious problems in the near future. But over time, it begins to matter.

As our transaction fees go up, not only will we be incentivizing new miners (strengthening the network and lowering the % of hashing power belonging to the zero-tx miners) but, eventually, the zero-tx miners (surely a handful of miners already did this) will begin to start processing transaction fees just because the lost profit would make it stupid not too.

And yes, with a big enough transaction fee reward per block, even a zero-tx mining botnet (which we don't know it is) will switch. And the halving of the block generation reward will only hasten that.

Looks like the simplest, easiest, and ideal solution is for us to do what needed to be done anyway.

Well played.

This is an interesting proposal. This is how a merchant would decide where to send transactions in other current clearing systems. If it can be done, it'd lead to competition and diversity among pools, and wouldn't that add to the overall network security?

It might create some competition in wallet/PoS systems as well - pools or their resellers might put out transaction software to directly push TXs through their pool (or a network of syndicated pools, etc) - having wallets that only relay to specific clearinghouses (pools) could open up additional fraud detection/protection services, etc.

If the confirmation times ever become a problem, people will just start paying more in transaction fees, and the whole thing will balance itself out.
It's in the design.

It depends how much profit there is to be had vs the effort needed for this guy to upgrade his architecture and do things properly. It may be that now he has whatever-it-is set up and running, he doesn't care any more and it'd take a lot of missed profit for him to start caring about it. If Bitcoin reaches a point where it's incredibly expensive to use because of misaligned mining incentives it'll just end up useless as a currency.

I really only see three possibilities here:

1) Botnet and for scalability/whatever reasons they are not including transactions. Solution: investigate and get the owner arrested, if possible.

2) Custom hardware buildout. Friendly miner. Might not realize or care that he's harming the network by producing empty blocks, or not understand how to integrate transactions properly. Talking to them might help.

3) Custom hardware buildout. Knows what he is doing, does not care about overall network harm, does not care about incremental profit unless it's really large. Purely mercenary/profit motivated. Might be encouraged to upgrade his software if MtGox refused to do business with them until fixed. Otherwise just has to be tolerated, modulo something like Gregs protocol-level change proposal.

At any rate, there's little point discussing it further until somebody has contacted the ISP of the IPs in question and asked them to look into it.

Transactions fees are completely ridiculous right now. I mean, the objective of fees is to cover the cost of a service. I can't talk for other people, but right now, fees are surely not covering my electricity bills.

Currently, a ton of people are shooting transactions left and right for free. I paid from my pocket those computers with this electricity, I took the time and the risk to provide this service, and I should give you a free ride on my service? Bitcoin generation is a temporary measure for the birth of the network, and we should not depend on it. I find it insulting when people act like transactions should be free, and that we, miners, should do the good(aka get screwed) thing and paying from our pocket so you can move your Bitcoins without any fees.

Those who are not including fees are the ones harming the network right now. I hope more miners does the same thing and force people to add more fees to their transactions.

Some short-sighted miners here it seems...

The real Bitcoin economy would likely not handle a huge increase in transaction fees very well.  Enforcing the minimum would probably be fine.  But doubling or tripling it would likely cause pain.  And even that would get nowhere near the current block reward.

It's not even clear that this miner would include transactions if the fees were higher.  Right now he isn't including the transactions with fees.  He's including zero transactions.  No one knows the reason for this.

Your electric bill is being covered by the block reward.  If you want fees to go up in order to cover this cost, the reward would likely have to come down significantly in order to meet halfway.  Otherwise, the (miniscule) real economy would be hugely affected, and price could plummet.

Be careful what you ask for.

There is a fixed cost (in terms of software, complexity, bugs, troubleshooting, etc), for including ANY transaction.  The aggregate transaction fees are so beyond worthless (a few pennies per block) that it doesn't make any economical sense to include any transactions.  Including only paying transactions doesn't improve that dynamic.  Including only paying transactions may lead to more paying transactions but that is an unknown and indirect benefit.

If transaction fees were higher then there may be interest.  At a minimum the miner may need to consider it either now or in the future.  Today he is pulling down ~$250 per block.  However say difficulty rises, the reward falls, and there is a crash on the market.  That might make a block only worth $100.   Now $100 is great unless you have gotten use to months and months of $250 per block.  While realistic transaction fees wouldn't bring him back to $250 maybe they boost it to $100 vs $105.  If $5 enough?  I am not sure but it might be.

This miner has some cost.  Even as a botnet (if he is a botnet) he has opportunity cost.  Fees create an economic incentive to include transactions.  No economic incentive exists now.  Having a few fees with a token amount doesn't change that.

The avg block has about 80 tx.  Even at 0.02 BTC each that is 1.6 BTC.  When block reward falls to 25 and extra 1.6 BTC is significant.  If transaction volume increases 50% over the next year if avg fee was 0.02 BTC that would be ~ 10% less income by excluding fees.

Is 10% enough to change his method of mining?  Who knows but 0.3% certainly isn't.

I'd also say that more fees also would encourage more honest miners to ramp up their hashing power, diminishing this guy's %.

Incorrect.  Including transactions creates incentives for bitcoin users to use the network (faster transactions).  Slower transactions discourage users from using the network.  This is why deepbit includes as many transactions as it can, and doesn't really worry about fees too much.  Deepbit wants to create incentives for bitcoin use.

Too many no-tx blocks, and users will drift away, causing the value of each bitcoin to drop.

Thus, in the short term, no-tx blocks may make money for the botnetter, but in the long run it will make their (and yours) bitcoins worth less.

The value of bitcoin lies in the strength of the network, and the number of users.

or those users could simply include fees and increase the economic incentive for miners to process them promptly. :)

If fees were high and the miner was still excluding them I would be inclined to agree.  Still I don't think "mystery" actions are affecting Bitcoin users in any meaningful way.

1 confirm time has increased from 10 minutes to 11.5 minutes but most users look at 6 confirm time.  That has increased from 60 minutes to 61.5 minutes.  A 2.5% slowdown and well within the normal variance of confirmations so that an uninformed users wouldn't even notice tx being "slower".

So I agree there is a tragedy of commons effect but as long as the # of miners excluding all tx remains a minority it is unlikely to have any meaningful effect.

Personally I actually prefer mining at a pool that includes transactions (even if they don't pay the fees out to miners).

Including transactions is important for the Bitcoin network to be utilized by merchants and users, and that usage is important for the future of Bitcoin.
Right now with 50BTC block reward (or even with 25BTC) transfer fees would need to be immensely high to compete with block rewards. Too high to ensure the use of the BTC network in its early stage. As the number of transfers increase and the block rewards decrease, transfer fees will become more viable. But that requires a high network utilization with a large volume of transactions.

While I don't advocate free transactions (any fee right now however small is better than zero, as it habituates the concept of charging TX fees), including transactions in mining operations is a really easy and cheap way for a miner to show support for Bitcoin. And if you want to see Bitcoin as a future currency that support is certainly worthwhile.

Not right now, I'm not asking for an increase tomorrow. But in around 9 months, miners will have their payday cut by half. I'm not going to do the "good thing for Bitcoin" if I'm stuck paying from my pocket because of low fees. And it will be the same for a bunch of miners there, if they can't profit, they close the rigs. What happen if they close their rigs? Well, solominer is going to have a bigger % of the network and because of the drop of miners, blocks will take a longer time to solve, slowing even more the flow of transactions.

I don't see this situation as a problem, I see this as a wake-up call. It's going to take that long before we see software that manage included transactions in a block, or "optimal" pools that include only transactions with fees.

you forgot maybe solo-miner also closes his rigs at this point.

Yeah, but other people with the same idea could do the same thing.

...and people without the same idea could also jump back in.

nice balancing effect, huh? :D

Yeah, but right now, 95% of the miners include all the transactions, it's the 5% who sort which transactions to include that seems to disturb a couple of people around here.

I don't think solo-miner is a problem, it's more that people think that miners managing the flow of transactions is a problem. We'll see next year, but I'm pretty sure some people will start sorting transactions (like me, if I have the means of doing it, I'll do it, no problems).

Indeed, making the definitely wrong assumption that everyone has the same costs would mean that even small tx fees would exclude tx excluders from profiting. I would think the 10% number mentioned as a possible near future amount would do a lot of good.

Instead of increasing TX fees, we better see this as an attack, and should look for another solution.  The more popular Bitcoin becomes, the more enemies it will have.

Who said excluding? I said lower %. Easy concept really.

There is no way you can exclude dishonest miners from mining on free computing power.

I disagree, I think it should be fairly easy. In terms of fees, for the moment, the gain from mining coins is larger than the gains from any reasonable transaction fees. After looking over the numbers, the drop in coin production is likely to be met with a corresponding rise in FX rate, which will probably cover most of the cost of mining, and is already taken care of within the BTC framework.

However, I second Theymos in supporting gmaxwell's proposal. I think Gavin's idea would probably lead to a threshold war with the botnet(s). However, in contrast to gmaxwell's proposal:


This seems to me to be basically pointless. As miners become more reliant on fees and less reliant on coin generation, they will not likely want to donate their tx fees to some non-contributing bots. The programming effort implied would be wasted towards placating non-contributors without actually providing any benefit to legitimate operators. BTC would be better off just giving them the stick.

Elsewhere it has been proposed to extend bicoind to allow for remote one:many serving of the blockchain, and if that were implemented it might allow the botnet to contribute legitimately for very similar costs. This addition might also lower the cost of current and new mining rigs since it eliminates the need to store the whole 2GB block chain on every rig, and it would allow users to serve their devices from a BTCBank or from a machine they own. Of course, I don't know enough specifics about BTC to know if this is actually feasible, but it sounds good anyway :P.

That still doesn't exclude them. You diminish their gain some %. Exactly what I was pointing out since the beginning.

Exactly.  If tx fees are significant enough a botnet will include them. 

Today excluding all tx costs a miner roughly 0.06% of potential revenue.  He likely is thinking OH NOES I only get 99.94% of potential revenue.  Whatever will I do.

If hypothetically tx volume doubled over the next year and fees increased to just 1 bitcent per tx on average after the subsidy cut excluding tx fees would cost a miner 6% of potential revenue. 

Is that enough?  Who knows but I would imagine 6% gross loss vs 0.06% gross loss at least makes a miner think what could I change to collect some of that uncollected revenue.

Every miner (even "mystery") has a tipping point. 

Gavin's proposal would reduce their % by delaying the empty blocks such that most of them expire.

gmaxwell's proposal would require them to maintain the list of tx in order to have their blocks accepted at all. That is, if they don't have access to the primary blockchain/tx data, they cannot even construct valid blocks. That would be very hard (although not necessarily impossible) for a botnet to get around.

In the case that they did somehow get around it, Gavin's proposal could be used on top of gmaxwell's to delay "cheap" blocks and reduce the total impact on the network. I don't think that would even be necessary, and the more miners depend on tx fees, the less of a difference it will make.

However, in order to produce a similar income to coin mining, at 80tx per block, it would require a fee of around 0.5BTC, or US $2.50, which is pretty exorbitant even for an ATM withdrawal. While that would certainly give miners an incentive to boost their capacity, it doesn't actually solve the freeloader problem and it would create a disincentive for BTC users. Such a rate hike would be far above the design of the currency.


Again, that behavior is part of the design of the currency, and should not be a concern. It's also not an appropriate solution to the problem at this stage of the currency's development.

This is a strawman.  Fees don't need to be EQUAL to the block reward they just nee to be relevent.

If you make $100K a year and I say "hey man for a couple hours of setup work I can show you how you can make a WHOLE PENNY more next year" you likely won't be interested.  However if I said "hey man for a couple hours of setup work I can show you how you can make $10K more a year you might be.

$10K < $100K but it is at least relevant fraction.
$0.01 < $100K and is so tiny as to be immaterial.

To influence behavior fees don't need to exceed block subsidy they just need to be a relevant FRACTION of block subsidy.

As I showed above after the subsidy drop if avg fees were a mere 1 bitcent per transaction it would be a relevent fraction (~6%) of potential revenue and that would be hard for any miner (botnet or not) to ignore.




So you say, I disagree.  If 1 cent per tx kills Bitcoin then it was going to die anyways.   I am not saying it needs to be part of the protocol we simple need to get enough miners to exclude non-paying transactions to force freeoaders to invest in the success of Bitcoin.  If you think 1 cent is too much to be able to transfer an unlimited amount of value anywhere in the world instantly then you simply have unrealistic expectations and we may need to start working on that. :)

Ok, in the example you gave, you're saying around 10%. Each block yields 50BTC, equiv $250 USD, at 10%, that would be $25 USD, or 5 BTC. 5BTC/80tx = .0625BTC = $0.31 per tx. Still pretty high for a tx, especially if it's for small amounts.

If $0.31 causes there to be less than 80tx per block on average, that means even higher fees will be required to maintain 10%, which is a further disincentive to making tx. Until the number of coins generated drops significantly, and until the number of average tx increases significantly, this is still not enough of a disincentive for the botnets to stop mining empty blocks, nor is the incentive high enough for them to take on the whole 2GB blockchain on every one of their nodes.

While this isn't a huge problem right now, if nothing is done to deny their entry into the network, it could eventually become a 51% attack, or could be used as a stepping stone for one. Artificially beefing up the miner's pocketbooks would be a far more expensive and less effective solution than simply changing the code to exclude freeloaders. Depending on how much BTC appreciates, the opportunity cost of freeloading might continue to be justified even after 2017. It also depends on the cost of running dummy miners vs miners w/ full blockchain. If the savings is significant enough (apparently it is for the moment), then direct disincentive will be required to prevent them from competing with legit miners or else the performance and the security of BTC will continue to be compromised.

He was (clearly) talking about a time coming soon when total tx fees may make up 10% of the reward miners get for mining. Not 10% tx fees.


Go for it. Or maybe you could just read the discussion, that might be more your level.

No I said with TRANSACTION VOLUME DOUBLED
WITH SUBSIDY CUT TO 25 BTC
AND A 0.01 BTC FEE.

and that was only an example

25 BTC subsidy
160 tx * 0.01 = 1.60 BTC
1.60 / (25 + 1.60 )  =  6%

A 1 bitcent tx fee (avg) on 160 tx per block after the subsidy cut would make fees contributing 6% of miner's total reward.


I disagree.

First 51% and 1tx block have nothing to do with each other.  51% attack can happen even if botnet includes all txs.

So lets keep it on the topic of 1tx blocks.  As shown in that example above even a modest fee after the subsidy cut results in fees contributing a SMALL BUT SIGNIFICANT PORTION OF REVENUE.

Hashing power follows price.  More correctly hashing power follows block compensation.  Fees adding 6% to block means 6% rise in hashing power.  That makes "mystery" share smaller.  Mystery sees revenue decline due to subsidy cut and then sees it decline even further due to tx fees (which he is excluding) making up greater share of total rewards. 

Unless mystery is an idiot he is going to start saying ... "I would like 6% more revenue lets see how I can get a cut".  If he doesn't then maybe 8%, 10%, 15% eventually drives him to the tipping point.

Someday fees will need to support 100% of the network.  We can start the ball rolling at > 0.03%.


I haven't seen a single proposal that would either work or wouldn't have significant side effects and open the network up to "gaming" and malicious attacks.  Protocol changes should be the last choice.  Miners requiring higher fees is a simple strategy to try first.

It does, depending on how the botnet is handling tx. If the botnet is running off a few servers providing the blockchain, then that botnet would increase the amount of computing power needed to hit 51%. As it stands, since the botnet is blind, it can be incorporated by an intelligent attacker to make up 51%. At the current botnet size, it gives about 15% leverage either way, although this may increase or decrease in the future.


Well, in 2013, when that becomes relevant, then we can talk about what a reasonable fee is. 1 bitcent isn't bad, but it can't be predicted how the value of BTC will respond to the drop in subsidy, or what kind of losses that will mean to the miners (or not). If the value of 1 BTC goes to $10, then an even smaller fee would suffice, since technically the miners wouldn't be losing anything.

Let's just say mystery saves, say, 25% of costs by not running the blockchain, and the value of BTC doubles, then he will still have no incentive to play nice, and may even decide to take a larger chunk. The only way that would not be the case is if bitcoin's value appreciated more slowly than the subsidy decrease, or if fees became fairly exorbitant.

The main problem is that, for the time being, the BTC community is fairly vulnerable, and it is precisely during this time when that 15% of total processing on the network would be the biggest liability to BTC's survival. If someone with the will and the resources figures that out, we could have a big problem. Even if not, that's still 15% of all BTC produced going towards someone who isn't investing in the actual function of the network. Ideally, the means should be provided so that tx's can be mined properly without the cost of a full blockchain, then mystery would be adding 17.5% or so to the total processing power of the network rather than wasting 15% running in place.

Also, I don't see how gmaxwell's proposal, with my revisions, would open up the BTC network to "gaming". Every block must be cryptographically valid as well as providing proof that it's running on correct blockchain information. It's pretty hard to make an attack within those constraints.

I'm showing ignorance here, but do all miners need to accept/require the same fees? Or could one pool take lower fees than another? (thus upping it's own TX volume, and potentially earning more in fees)

Miners can decide however they want, what tx's to include.
The client software on the other hand can decide what tx's are broadcast through the network.

To me it's clear that a solution that makes it impossible, unpractical or at the very least clearly unprofitable to refuse to add any transactions is VERY MUCH preferable to a "solution" that simply hopes it becomes unprofitable enough for the mystery miner to some day start adding transactions.

This whole thing is an absolute outrage. Miners get their 50 BTC reward because they maintain the network and maintaining the Bitcoin network is not just about providing security (which the extra hashing power surely does) but making sure that Bitcoin transactions are actually included in the blocks. Miners who don't do this are partially leeching on Bitcoin and that should NOT BE TOLERATED.

I would accept a solution based on nothing but higher tx fees ONLY if the stick based solutions have severe side-effects. Other than that I would basically accept any stick-solution, whatever the hell it is. The main problem here is less the practical delays that 15% of no-tx blocks cause (which are undoubtedly small), than simply the fact that it's an outrage that it's even possible for someone to mine like this when transactions are clearly such an important part of Bitcoin.

This whole thing should not be confused with the issue of creating a tx fee market in the future. As block rewards go lower, that will certainly be necessary if we continue to use proof of work. Tx fees will start playing a bigger part in miner rewards, that much is obvious. And the tx fee system needs to change from what it is now. It is however a separate issue.

I don't see the outrage.

As far as forcing miners to include all tx does that include free ones? 
Does that include ones with a token fee (say 1 satoshi)?
Why as a miner should I be FORCED to include free loader transactions?

The solution I proposed doesn't force anyone to accept any given tx, however it does require that they include valid tx data, rather than just ignoring the blockchain. If a mining rig is already accounting for the blockchain, then it would be nonsensical for them to exclude every tx, especially if some of the tx going through their network include substantial fees. There's a big difference between a miner getting few tx because of high fees and one producing token 1tx blocks just trying to look valid. The solution I proposed would make it dead obvious if any miner wasn't using the blockchain, so those blocks would be thrown out completely and the coin they generated would be invalidated. No more carrot, no more problem.

Of course, it would be nice if mystery was actually contributing, but it might be more difficult to implement the means for that to happen, and there's no guarantee that he/they will adopt it even then. Kicking them out would definitely work, although with less potential upside.

So basically, unless validating the tx hashes of blocks is ungodly more complicated than rigging remote serving of the blockchain, then the stick is the more viable and assured option.

A I understand it (possibly wrong) your solution is that miner must include in the block a hash of the tx of the prior block.  The issue is that doesn't accomplish anything.  A few distributed servers (or nodes, or calls to blockchain.info) could provide that data via remote call.  A miner could easily then have nodes process blank blocks with valid signatures.

Of course you aren't the only one with proposed solutions.  At least two people have proposed systems which invalidate blocks that don't contain a certain amount of tx.  Satoshi clearly believed miners should be able to set whatever fee requirement they want.

I'm afraid I don't know that much specifics about how much of the blockchain is actually required in order to validly mine tx. However, gmaxwell's system uses the input data from the previous block's tx, the input data from the tx currently being mined, and the output data, which is added to the current coinbase in order to create a sort of "signature". If the rig isn't validly mining tx, then it wouldn't be able to include valid tx data from the tx it's mining, nor a valid signature.

If they can get around that by using a few servers for distribution, then it seems to me they should also be able to validly mine tx. Problem solved.

However, after re-reading this a few times it seems to me that the primary reason mystery does not do this (or use a mining pool) is because his computing power is probably stolen, and anything that could be used to trace him would put him up shit creek. In that case, the only way to deal with it is to give him the stick. If he tries to get around it, he makes himself traceable, game over.

How would he be traceable.  He can use a few of the nodes to provide validation hashes for the rest of the nodes.  Obviously every block requires the prior blocks hash so he must already be distributing that anyways.  He has to have at least one copy of the bitcoind running somewhere as he needs to be aware of other blocks generated and the current dominant chain.  Requiring "current" tx is nonsensical.  If there are no tx in the block then that adds nothing to what miner needs.  Unless one intends to force miners to include all tx that is.


But he IS validly mining.  Imagine I wanted to run a pool which requires a 1 BTC tx fee.  Currently in last 7 days guess how many tx (other than the coinbase) would be in my pool's blocks.  ..... 0.  

So would I be "invalid mining" (or whatever nonsense term you want to make up)?  The reality is no tx meet my fee requirements.  It doesn't matter than you or others might think my fee conditions are stupid.  Satoshi intended an open fee market.    If I feel my computing time is worth 1 BTC per fee that is my right as a miner.

Under gmaxwell's proposal, the miner would need to include exactly the transactions specified by the person who verified that the transactions and the previous block were valid. So if blockchain.info published a set of 10 verified transactions, the miner would have to include all 10 of them.

well, just to add my 2 cents. If it's a botnet, then it'd need a lot of bandwith and controlling processing power. Someone in the thread about  this in the mining section gave 1.8m clients as an average number of estimated clients. If I had a botnet of this size, I'd mod bitcoind to just have the coinbase for getwork, and the clients listen for block changes(could be done via IRC). that way, bots would only "getwork" when a new block comes out, which reduces the amount of resources needed to serve the botnet amssively.

Well, you wouldn't be doing very good business if you actually had to depend on tx fees, which is mostly the point. Admittedly it's a difficult problem to define an acceptable threshold, but if push comes to shove it could be done. However, unless a lot of miners start purposefully killing their tx throughput that shouldn't be an issue at all. If mystery really is running sans a blockchain, which seems to be the case, then only he would be affected since the stingy miner can still produce the minimum verification, whereas mystery could not.

If mystery finds a way to run with the blockchain with his stolen computing power, then he could continue to do business, but would be running like a normal miner, problem solved. Given his current cushy 16% market share, however, nothing short of chasing him out would motivate him to go to that kind of trouble. Downloading a 2GB file onto thousands millions of computers you don't own without being noticed would be nothing short of a miracle, and not our problem, for that matter.


That's a lot of freaking clients. And also, that's probably exactly what's going on, or something very close. I'm glad someone knows how this crap works :P.

But it is my choice right?  Or is the choice on what is an acceptable fee decided by the cental agency of bitcoin fees? I have a right to charge whatever amount I want for my hashing power.  Do you agree?  As the subsidies decline the market will be more competitive but it is still my choice.

If tx MUST be included then does that include all free tx.  Am I now forced to include all free tx?  What incentive is there for a user to pay any fee then?

Does it include all tx w/ a fee?  If so then why every pay more than 1 satoshi?
Does it include all tx w/ a "minimum defined" fee?  Say 0.0005 BTC.  If so then why would any user pay more?

If someone says you must include 25% of all pending transactions... well what if those transactions are below my processing cost (because the users are cheap bastards)?  I am forced to include them or lose the block?  Really?

Specified by whom?  The person who solved the last block?  If so why aren't the transactions in their block?

Anyone could publish lists of transactions that should be included in the next block along with the necessary "verification proof". Bitcoin nodes themselves could publish lists using a new network message.

When there are conflicting lists? :(
When the list contains free tx and I as a miner don't want to include them? :(

Wouldn't it be in the best interest of free-loader nodes to publish and follow lists which put all free tx into the required list thus ensure free tx on the backs of miners?

It comes down to a fundamental question:
Do miners have a right to set the price for their services and thus indirectly choose which tx are included in the block they are working on?

The miner could choose the list to use. Someone could even publish valid lists that are always empty, though rational miners won't use these because including a transaction is almost free.

Why would a rational miner want to allow free tx forever?

Thanks for explaining it theymos but I like it less now with the explanation.  

For example in the last block:

http://blockchain.info/block-index/197728/0000000000000628467e112dd0aca3b6e5abbdd12cbd1b79074c446e85a2689d

there was 0 tx with a fee of 0.01 or higher?

So if majority of miners include a list with these tx I am forced:
a) to mine for nothing
b) to risk my block being considered invalid

Horrible idea.  Mining should be an open free market and if I don't want to include any tx without a sufficient fee I should be able to.  

More centralization for nothing.

If DeepBit and Slush built a shared list they could
a) determine pricing of entire bitcoin economy
b) force other pools to pay for access to the list or risk them building invalid blocks
c) to aid in b they could witholding tx from the list and publish them only to subscribers (if you don't pay you can't mine)

You're misunderstanding the proposal. Anyone who is verifying the chain can publish a list. Lists can use any fee rules. For example, Bitcoin Block Explorer might publish lists at /q/getValidationProof/minFee, which would only list transactions with a fee above minFee. Under this system, dumb miners would be able to choose essentially arbitrary fee policies; they'd just be forced into mining only properly-verified transactions and blocks.

I think there's some confusion here.

Currently, it is possible to get a tx pushed with no fee, but the commit times are horrible and there's no guarantee that it will even work. Nothing really changes here, and miners can still do whatever they want, however stupid it may be.

Also, while gmaxwell did propose a mechanism for forwarding lists of tx, this was primarily intended to give the bots some pre-verified tx they could use to mine into their own blocks. In other words, only real miners can produce such a list (since only they can verify the previous tx of the blockchain) and the only way a bot's block could be included at all is if it voluntarily included some tx's sent to it. In other words, you've got it backwards; if a miner doesn't want a 0 fee tx, it can dump it on the bots.

However, I suggest against that simply because it might give people an incentive not to include a fee, and still be able to get their tx committed, thus robbing potential fees from miners who actually own their equipment. A stolen mining rig costs nothing, so even if they only processed 0tx fees they could still compete and drive down prices.

(basically what theymos said, but with a bit more detail)

Of course they would.  Whatever list is used by 51% of miners will orphan out any other chain. 

Haplo please learn the protocol before you try to "fix it".

Unless your tx is considered spam (under the 1 day & 1 BTC threshold)  there is no need for a fee and confirmation times are routinely the next block.

Why?

True, although it probably won't be next year. Currently what I said does hold true for "spam" tx, and next year the same will probably hold true for all tx. However, if the 0 fee txs are passed to bots, then the penalty for not including a fee ends up falling on the legitimate miners, who could otherwise extract a higher fee and from a larger number of tx.

51% of miners follow a list which says tx A must be included.
49% of miners follow a list which don't.

It is no different than a 51% "attack".

The 49% may solve a block BUT that block will be ignored by the miners in camp "A".  Eventually the A camp will pull ahead and orphan all the blocks of the 49%.

So say current block height #123
Both camp A & camp B work on block #124.
A miner in camp B solves block #124 but it is missing tx from "A" list.
Miners in camp A invalidate block #124 and continue to build off of #123.

Due to variance "camp b" may pull ahead temporarily but just like a 51% attack eventually camp A will pull ahead and orphan the chain supported by the "B" miners.

This means whatever list is used by 51% must be used by all miners.  That gives massive power to those who control a lot of hashing power like the Big 3 pools.  If the Big 3 pools make "list A" they even miner is defacto forced to use it.  To not use it is economic suicide as any block the mine which is not in compliance with the "list A" risks being orphaned.  No miners is going to take that chance.  All miners will be forced to follow the rules set by the Big 3.

The Big 3 then could decide not to share the list with all miners (permanent mining exclusion unless the excluded gain 51% of hashing power), charge fees for inclusion (think of it as a pool fee but they don't even have to provide a pool), or share "hidden tx" between themselves to orphan out miners not paying to mine.  If only those paying for access to the list know of the list and hidden tx then outside miners can't even hope to compete by including all tx (even every free one) blindly hoping to make a compliant block.  Even if they include all tx they know of they will be missing the hidden ones.

That's not the current behavior, and that behavior is not part of gmaxwell's proposal. Blocks won't get rejected/discouraged for not including transactions. All blocks just need to include an extra "proof of validation".

Hmmm.






I'm not getting this proposal at all.

First... these tx-lists can be published by anyone verifying the chain? Bitcoin nodes can publish lists? Just any ol' node can do that? Considering the prospect of malicious nodes trying to crack this new feature, isn't that tempting fate, unless such lists can be ignored? And if they can, what's the point of having them?

And then, miners get to choose which list to use. So why wouldn't a miner just publish his own list, then immediately use it?

I'm failing to see exactly how all this is supposed to realistically handicap a miner and "encourage" them to put transactions into their block.

And if it DOESN'T do that, then what's the point? What is this "proof of validation" supposed to do if it doesn't result in transactions being forced into a block?

And if it DOES force transactions into a block, but I produce a block without those transactions, how could my block NOT be rejected?

This MUST be solved or Bitcoin could become useless and just creating a higher transaction fee is not good enough.

We do not know what the goal of this miner is.
It might not be to earn Bitcoins but to devalue Bitcoin.
This could be the work of someone who has or are about to create a alternate cryptocurrency.

The solution/s has to be designed to prevent or increase the cost for a worst case scenario.
We cannot design a solution for "the best case" which is, he cares about the value of Bitcoin.

If you do not realize this, your opinion is not of much use.

Another part of a solution could be to add a some real proof of stake for miners, but it would not be enough.

we dont even know this "miner" has malicious intent. it could have been just someone building some mining asic or autonomous fpga controller that has constraints in size. I mean it's not too hard to write your own mining controller.

The reason I put this thread to the dev & tech forum was that I want to explore solutions to this, not start another thread where we speculate who it is. There is a massive thread on that at the mining forum. I don't really think that it matters who he is, we should always work on the worst case assumption and I honestly don't think there are any good cases here.

Personally I think that it's very likely to be a botnet, simply because adding transactions is the standard way to mine and also very cheap under any regular or even semi-regular mining configuration. One thing is for sure, our mystery miner gets major benefits from not adding any transactions. This is clear. The only good explanation to why he could get benefits from doing this is a botnet.

Satoshi saw this scenario as unlikely but he did have plans in mind. I say we follow on his footsteps and make it so that a portion of announced transactions must be in the next block or it's refused by the network. This portion can be small, we just need to make it big enough that our mystery miner is forced to at least work with the blockchain. This would be done in a way that a portion must be included even if all of the announced transactions are without tx fees.

This doesn't invalidate the tx fee market, the miners are free to do with the rest of the transactions as they wish. The forced portion can be fairly small and then the rest of the transactions will follow regular rules that miners can set however they wish.

I have no clue how to make this happen from a technical perspective, I'm simply throwing it out there based on what Satoshi said. If you guys have ideas on how to do this or if you want to discard it altogether, tell us.

btw, you're trying to fix a problem that isnt there. me outlining possible causes for that miner to sign empty blocks was just a way of showing that this might not be desired behaviour, or that it might simply be due to limits, but not with malicious intent. But as stated before, bitcoin is free, and therefore impounding measures prohibiting people from chosing what they'd like to integrate is bs IMHO.

besides that, your idea of having miners forcefully integrate a certain ratio of the set of unconfirmed transactions is a bad bad idea. consider this:
I hate my pool because the op is greedy and charging ridiculous fees. So whenever I find a real_target share I simply wait 10 seconds before submitting it, and use those ten seconds to send out many microtransactions. some bitcoind's will get that quite in time, and consider the new block invalid because it did not meet the quota. because I have a pool that pays out orphans, I'll still get paid(unless those 10 seconds make the difference and someone else finds the block, but that'd impair my profits only minimally), the transaction costs for spamming are minimal(or even none at all if I choose to use >1d >1btc inputs), and the pool admin gets ripped of another 50 bitcoins. With so much greed, bad behaviour and willing malevolence in the bitcoin community, this is a likely scenario.

btw, I think forcing the miners to include a minimum of tx is a kind of socialism.

that wouldnt fix the problem at all. patching bitcoind to not relay transactions is an exactly two line patch changing 4 bytes. the bad or notso bad miner could simply apply that patch and have a hundred transactions present all the time. since he doesnt relay them, he can be 100% sure that he will be the one including them into a block, while recollecting the fees needed to pay for those tx. Result: we animate someone to do blockchain spam with senseless tx.

This could be a bug in a modified mining code's transaction verification code, or (more likely) a malicious player.  Either way, I don't think responding is necessarily the best course of action.  If it's bad code, it's not really our problem to solve, and the developers are going to notice eventually.  If it's a malicious miner, then with 15% of the network it's no small attacker; and getting us to respond in some fashion is as likely the goal as would be adding more power in the future.  Empty mining blocks were always going to be an issue, we knew this from the start, so this problem has been mentally hammered for two years.  There is nothing that we need do about it as a group (such as change the code) because by rejecting fee paying transactions they are harming themselves.  We don't wan't to alter the code to respond to this, because then we would be the reactionary group, responding to the attack vectors of an unknown malicious agent by potentially adding new attack vectors.

What we could do is publish the bitcoin addresses of the null block offenders, and both try to identify them as best we can, and (as individuals, not as a community) choose to delay transactions & blocks produced by that address.  Because the decision making process is based upon the propagation of a new block, even a short transmission delay in a majority of nodes will result in this malicious attacker's effective hashrate being reduced.  Which can function as a punishment for failing to respond to the social rules of the network, but does not require widespread code changes to deal with a particular attacker.  Users can choose whether to participate in the sanctions or not; just like they technically can already do if they have the coding skills to make the local changes.

You can't beat this by playing whack-a-mole.  Right now it's only a couple IPs but the problem should be solved generally.

I do think it needs to be solved.  While it's not significantly hurting the network now, it's a vulnerability and it should be fixed.  And ethically, I object to people making money when they're skipping out on their responsibilities.

I don't see the need for mandatory tx fees.  What's wrong with simply refusing to relay blocks that don't include at least ( ( n * 0.25 ) - 5 ) transactions, where n is the number of valid transactions broadcast since the last block?  Empty blocks would be allowed during slow times or when a block was recently sent (the -5), and the 0.25 would let you be reasonably selective about what you mine.

Nothing, so long as this rule remains a choice of the user.  It can't be a change that breaks the current system.  For example, you can add a rule that prevents your node from forwarding an otherwise valid block or transaction by whatever criteria you choose; so long as you don't reject the block if it gets into the blockchain.  The rules for validity must remain the same, but you should be able to add whatever rules to further limit transmission that you like.

Forever is unlikely, but for the next 100 years it is very rational.  If we want Bitcoin usage to grow, we need to facilitate free or very cheap transactions.  Otherwise people will just use PayPal or whatever, which is bad for everyone who mine coins.  As long as the block reward is more than one can reasonably expect from fees, it is completely rational to keep fees as low as possible without encouraging spam.  Discouraging spam and laundering is the only reason to have fees now, and for a long time into the future.

The mystery miner's goal is clearly out to devaluate Bitcoin by making it less useful due to slower transactions.  I doubt fees would change anything.  Has any of the mystery miner's block rewards been spent yet, btw?

I'll be not shocked at all, if this turns out to be a bug in a minority client; such as in combined mining.

If those tx are free I am forced to include them or have my block rejected? 

Where do people get this idea you have the right to have your free or cheap ass (1/10th of a penny) tx forced into the next block.

1) A charity accepting donations, mining for their own benefit.

2) a bank accepting free transactions of their own account holders, as an off network benefit.

3) Wal-Mart accepting free transactions for in-store purchases.

If 100% of transactions coming through are free you would be forced to include some, but realistically there are always enough paid transactions that you can meet the 25% quota easily.  If you only include 25% during an "all free transactions" hour, people will have an incentive to include a fee to get their transaction included sooner (4x sooner on average in this case).

25% is just an example; pick a value that will accommodate any reasonable miner's include/exclude policy.

It's not a vulnerability.

It's a DOS vulnerability, as discussed earlier, and it makes 51% attacks easier, even if the null-miners are not directly collaborating.

D&T:  We could also make it so that you have to include ( ( n * 0.25 ) - 5 ) of the last n paid transactions, and have a lower (or no) quota for free transactions.

Really have you looked at the last dozen or so blocks?

90%+ are free and 0% have a fee of 0.01 or higher.  So 100% are free or so worthless as to be essentially free.

So you saying "miners can set their own fees" is a blatant lie if the same miner is then forced to include free (and near free) tx.  If clients know miners are forced to include their freeloading tx what is the incentive for them to pay a realistic fee.

Of course that ignores all the timing attacks, orphaned blocks, and network latency issues any such proposal will have.  A hint: there is no unified view of the network.  Propogation delays ensure parts of the network are always out of sync.  So a miner cutting it close risks having some nodes see the block as valid but others (who just picked up new tx) see it as invalid.

Also it makes tx processing less efficient.  Most pools don't update getworks when new tx arrive however to meet the 25% treshold (which is likely 50% threshold just to be safe) they would either need to issue an LP on every tx and include even MORE free tx so that as new tx arrive they stay above the safe limit.

Essentially all that combined means miners have no pricing power (they already have very little).  All this to solve a non-problem.  Sorry no thanks.

I'll refrain from discussing the technical solutions because I'm not smart enough, but...

Don't you guys realise that you are doing a better job at undermining peoples trust on Bitcoin by discussing this non-issue relentlessly than "mystery jerk miner" is by not including tx's on the blocks he mines?

If miners care, why aren't they already doing something about it?  People will send free txes as long as the system allows it.



100% chance of inclusion per block instead of 25%.  Or as I said later:  make the n based on the number of paid transactions, and let miners ignore the free ones if they want.



Sure, that's why the -5 exists: to give some buffer if there were a few very recent transactions.  It would only matter on very closely spaced blocks: normally you'd be including say 25%+ of transactions to meet a 10% quota and you'd have plenty of room.



How is that a good thing?  Miners should be encouraged to include as many recent reasonable-fee-paying transactions as possible.  Updating getworks isn't hard.

If people's trust depends on us being hush-hush about potential problems, even small ones like this, then the whole thing is hopeless.

In the long run open discourse and letting the light fall on every corner is much more trustworthy than trying to pretend it's a perfect system that will never need improvement.

You said it very well, potential problems, but whoever reads this thread and the other thread at the mining forum will read "real problem in need of a solution" and freak out.

I'm just waiting for some dumb reporter to pick this up and start writing how Bitcoin is controled by botnets who don't even process tx's and other lies...

It's a real, though currently minor and not network-threatening, problem that needs a solution eventually.  It cannot currently be exploited for any real harm other than my indignation that some miners are being paid without doing their proper work.  This discussion is not about emergency damage control; we are just speculating and brainstorming ideas for how to adjust things in the future.

Better?

I agree with Revalin once again. I don't claim that this is a critical issue, my problem with this is the ethical issue of leeching Bitcoin. That is what the mystery miner is essentially doing because he is only doing half of his job as a miner.

As far as solutions go I like what Revalin is proposing. It would probably be good to make it so that the portion of transactions that must be included is calculated from transactions that have a fee (any that's at least the minimum).

Again this would not remove the market from tx fees, all transactions outside that percentage (whatever percentage we chose) would still be controlled by the miners, they are not forced to add them.

In fact this could actually add incentive for users to start adding higher tx fees to make sure their transaction fits this special slot and thus would be guaranteed in the next block.

A good start would be if something like this was simply built-in to regular nodes so that we could essentially boycott malicious miners by choice. I'd love that. (referring to what MoonShadow proposed)

Once again making a distinction of paying vs non paying is dubious.

1 satoshi is paying.

If every transaction had a 1 satoshi fee the avg block would contain 0.0000008 in fees.
The entire network annually would produce ~ 0.0292 BTC in fees.

So paying vs non paying is a dubious distinction.

Going forward (once I have modified bitcoind and tested it) I will be excluding ALL transaction with < 0.01 BTC in fees.  I will also share the patch as open source and encourage other miners to adopt similar minimum fee rules (although each miner could choose a different amount).

Do you believe I should not be allowed to exclude fees I consider insufficient?  
More generally:  Do miners have a right to charge that amount they expect in an open and free marketplace for fees?  

People like to use the term boycotting (as in the above post).  If no tx had a fee of 0.01 BTC+ and as a resulted I mined a 1 tx block would that be boycotted.  What if I included a single dummy tx of my own (1 BTC from me to me with 0.01 BTC fee to me)?

DeathAndTaxes is constantly approaching this issue from an irrelevant angle (from my point of view). Tx fees do not currently play a significant role in miner rewards, I have repeatedly stated that market based tx fee system is necessary eventually and I have no interest in disrupting that, on the contrary. I like that in many ways.

This issue is different and it has to do with the incentive for a potential botnet owner to simply ignore all transactions. If he has a strong incentive to do this because of a Bitcoin exploit (I consider this issue an exploit) we might have to start paying ridiculous tx fees to make him change his behaviour, which would basically destroy Bitcoin. We are nothing without cheap transactions, that is one of the main benefits of using Bitcoin.

I don't claim transactions should be free either. But they must be cheap. We have no idea what kind of benefits this mystery miner gets from not adding transactions. It could be that he simply can't run that botnet securely without having a system in place that ignores transactions. In that case he would NEVER start adding transactions, not at any tx fees, because his entire mining operation would be at risk.

The only solution I care about is one where a potential botnet miner is forced to at least work with the blockchain. I'm not interested in either keeping Bitcoin transactions free nor am I interested in forcing the miners to do something in general. I am definitely interested in any proposal that forces miners to add at least some transactions, if there are transactions to be added.

And to answer the final point, I believe dummy transactions wouldn't really help if the solution is designed properly. The nodes are aware of all transactions, if x % of them (that have at least the minimum fee) are not included in a block, they simply refuse it. It would definitely be beneficial to Bitcoin users if this could be done. I believe the tools to stop freeleech mining can be built.

I elaborated before why forced inclusion of transactions is senseless because it can be evaded without problems.

Of course you should be allowed!  That's well established as a miner's right.  Similarly it is also a relay node's right to exclude blocks they consider insufficient.  Check and slightly imperfect balance.

Relay nodes already enforce some minimum fees; those were actually reduced last year.  Right now it's probably best for free txes to be as low as possible - low friction will help growth in ways much more significant than some 0.01BTC fees.  That said I encourage you to institute this policy.  The completely homogenous one-formula-fits-all isn't good for the network.  I think we would be much better off with a gradient of minimum fees among miners instead of the very sharp instantly accepted / transaction limbo that we have now.

Even a 1 satoshi minimum would get people to at least start paying fees, and pave the road for the future where free/1-satoshi transactions are slow (just a few miners supporting them) and reasonable fees are required for fast processing.  It's probably helping your goals on the whole.

But I don't think we should make a universal mandatory minimum fee a rider on the fix for null blocks; that's really a separate issue.  In my opinion we should just try to solve the latter in a way that doesn't make the former worse or cause other collateral damage.

+100

While it's possible that it's some minority client malfunctioning, 1.8m users and/or 16% of the total network processing power is pretty extreme. The potential advantages for running a stripped down botnet make that the most likely scenario, and whether or not that's the case 16% of the network isn't contributing to the rest while collecting all the benefits.

Also, I think people are still misunderstanding what has been proposed. While Gavin and others have proposed systems which would omit or delay "cheap" blocks, gmaxwell's proposal is simply that in order for a block to be valid they have to prove that they have access to the blockchain. It doesn't mean they have to include any tx if they don't want to, just that they have to prove that they've seen recent valid tx that have been posted to the chain.

This would be trivial for legitimate clients, official or otherwise, but for a botnet running without having the blockchain it would be impossible to fabricate, so they'd be forced to shut down or play fair. The only time "tx lists" would even be used at all is if:

1: Some other miners wanted to "lazy mine" with no blockchain, and are willing to trust full nodes to provide valid tx for them to mine.

AND

2: Miners willingly want to give up their tx to others, or have tx they don't want, and would like a facility for dumping them on lazy miners.

However, while this feature isn't necessary for booting a botnet, it would weaken security since it requires that the "dumb nodes" blindly trust full nodes, so "dumb nodes" could be used to forward an attack. For a normal, fully equipped miner, tx forwarding would be basically useless unless they have a major incentive to dump tx.

I agree that we don't need to centrally control all of the miners, and that we don't have any business telling them which/how many tx to include or not. However, I think there's plenty of good reasons for preventing someone from sitting around doing getwork without actually having the blockchain, but it's easy to prove whether this is the case or not without putting restrictions on including tx. Revalin's suggestion (or similar) would only be relevant under very different circumstances, which are much more unlikely to occur than someone freeloading with an anonymous botnet.


The problem here is that the "Attacker" most likely isn't malicious in the sense of trying to take the network down. They just want free money from stolen computing power, and so far they're getting it. The real problem is that 15% of the BTC that would be going towards miners who actually maintain their rigs are going to someone who more than likely does not.

The other problem is, how the hell do you "block" 1.8 million unconnected stolen computers? If I started manually doing that now, by the time I was dead I'd still be nowhere near caught up. Requiring proof that you've seen the blockchain recently would automatically exclude anyone trying to cheat (at least via that route). Alternatively, if other people start figuring out they can make free coin by running a botnet sans the blockchain, pretty soon "mystery" will make up more like 30% of the network, and even more funds will be diverted from maintaining the legitimate side of the network. At the moment the reward for parasitic mining outweighs any potential losses from not including tx, and that's not likely to change much for at least 10 years.

Checking to see that a miner is using the blockchain is fairly trivial. If they're actually running the blockchain then there's no incentive for them to exclude tx (at the moment), and no need to babysit them to make sure they include some minimum number of tx, either (that's their choice, however arbitrary). The problem is the incentive for running without it, pure and simple.

Very true.  +100

The same can be said about the accusation of operating a "botnet", while the facts do not support that theory.

People here speculate about how not processing transactions would benefit a botnet, and then conclude that it must be a botnet.

This lacks both facts and logic.  For the lack of facts, consider this:

The node at 88.6.216.9 did have a complete blockchain and did verify transactions, and also relay them.  It just did not include them into new blocks.  For an example of a relayed transaction, see http://www.blockchain.info/tx-index/3091519/ca264af6271cc40bf8b297e13fc2489bd9564dd18b9e60a1bd4d19046086995b

When it was still operating at 95.120.241.167, it even did include transactions into blocks.  Both free and paid.  For an example, see https://www.blockchain.info/block-height/166667 (the last transaction is a free one).

It stopped including transactions in March.

The decision to not include transactions was made after support for transactions was already in place.  It is a political decision, not a technical necessity.

For the lack of logic, just think about how your own mining client works.  Most of you readers (yes, not all, but most), are mining at a traditional pool.  You consume very little bandwidth, you dont need to store the blockchain and you dont need to know or verify any TXs.  Yet your mining effort works towards blocks with transactions included.

You work successfully under exactly the same conditions, as (some of) you claim would force a botnet to exclude transactions.  (Some of) you say, that downloading the blockchain would make botnet victims suspicious, yet you do not need to download it yourself!  (Some of) you say, the steady influx of transactions would need lots of extra CPU processing and extra network bandwidth, yet you do not need to do that yourself!  (Some of) you even say that a full bitcoind must be in place, yet you do not need it on your own miners!

Your own mining client shows how the technical "problems" can be solved, and source code is available for everyhing (both mining client and pool service).

The assumption of the technical benefit is just plain false.  The only thing that supports the idea of a botnet, is the ever repeating posts here on bitcointalk.  Those who jump to early conclusions and keep reinstating this false idea, are the ones who do damage to bitcoin.

I wonder if you stand up as quick to apologize as you do to condemn.

It's not a DOS vulnerablility unless the attacker can mine nearly all blocks empty, in which case he already has 50%+ of the network and can pretty much do anything.  If he has less than a majority of the network, he can't prevent valid transaction processing every 20 minutes or less anyway.  So no, that's not an issue in this context.

And how would it make 51% attacks easier for others?  Again, I think this is an over-reaction.  If you think that this is really a problem, work on a patch that allows miners to simply not forward valid transactions that flag on a user's personal blacklist based upon address or whatever other criteria.  If enough miners agree that this is an issue, they will incorporate your patch and the null-miners will be sanctioned by the community without a community wide action need be taken.

To it is my right to charge what I consider fair but if I do you will exclude the block and I lose everything.  Honestly you think miner's will support this.

It would be like saying:
You can open a restaurant and charge what you want but if 5 people are waiting and you don't feed at least 25% of them you will be shutdown.  Oh an BTW some of them want to eat for free and some want to eat for a penny.  But sure you can charge what you want.

No dice.  Any system that forces me as a miner to use MY HASHPOWER purchased with MY CAPITAL and running on MY ELECTRICITY to include fee and "essentially free" transactions is a non-starter.  The "solution" is worse than the problem.


You "encourage me" but at the same time advocate a system which would orphan my blocks if I do what you are encouraging.  Thanks for your encouragement. 

Even a 1 satoshi minimum would get people to at least start paying fees, and pave the road for the future where free/1-satoshi transactions are slow (just a few miners supporting them) and reasonable fees are required for fast processing.  It's probably helping your goals on the whole.

I am not advocating that.  I am adovcating I set the minimum fee for my hashing power ONLY with no changes to the protocol and you are advocating orphaning my blocks if I do that.

Becasue we already pay you greedy miners with 33% yearly inflation.

Not enough?

That's a matter of opinion.  Bear in mind that including transactions into a block does add some nominial amount of additional processing load for the miner, so it's not free for him.  This null-block miner might simply be a pool that has a higher fee requirement for inclusion than the common amount.  That is his right.  And the only response for the other miners is to increase their own mining power.  This "arms race" benefits the security of the blockchain, even if a not-insignificant minority of blocks are null-set.  I'd be more productive if the miner in question published the minimum fee expected, if that is the case, but it's not necessary.

In a dumb miner - pool model the fact that the miner doesn't do the workload doesn't mean the workload doesn't exist.

One getwork is ~400 bytes.  Some have speculated mystery has 1.8 million nodes.  To use a traditional full pool server the bandwidth requirements alone would be 5.7Gb and 1.8 million connects every long poll.  Say he staggered those getworks to supply all nodes with work over a 10 second period (less revenue btw).  You are still talking about almost 200K simultaneous connections, 750Mbps, and enough computing power to generate 200K block headers a second.

If you want more insight ask Tycho what kind of load even small botnets do to a pool server.  Now with enough computing power (we are talking a dozen or so high end load balanced servers on a 1 Gbps connection) a 1.8 million node botnet "could" operate in a pool-miner model.  However now compare all that cost vs the incremental reward of .... ready for it ....about 2 cents per block.

This times 1000.

The 50 BTC block reward is for including transactions. It is money that comes from everyone that owns Bitcoins (in the form of inflation) and you should provide the required service. If we would change the protocol to force you to do so or not get paid this would not be socialistic like some propose because it is not mandated by a central government but in fact by a majority of the people.

I consider gmaxwell's approach to be short-term.  It will probably solve the current problem - whoever this is will notice their blocks are being orphaned and do something to fix it - and while it's intrusive (it requires protocol changes), it has little chance of other collateral damage.

I'm looking at ways to solve it in a more general manner to improve the long-term security of the network.  My proposed change is much simpler but the consequences are complicated.  I don't think D&T's fears will actually happen in reality - I think the economic incentives will actually result in overall more fees being paid and more latitude for miners to set them - but it's not easy to model, and obviously we disagree.

I also don't consider it a foregone conclusion that it's a botnet.  They have to be on the network to receive prior blocks and transmit new ones.  Why would someone go 80% of the way when they could implement a 100% functional pool (or just use any public pool) and draw less attention?  I don't think it's a deliberate attack either because there are much better ways to leverage that much hashpower.

I have no idea what their motives are, and honestly it doesn't matter.  I just want to create the economic incentive where they must fully participate or not get paid, which will definitively, and permanently, solve the problem.

This is an excellent argument for pools disallowing clients which aren't turning in enough results to justify their bandwidth.  If low hashrate clients (botnet or not) want to participate, they can have one guy get work and distribute it to the others.  Letting them collect 50BTC per null block to save the pool the awkward responsibility of turning away unprofitable workers is completely backward.

If what you're saying is true, and I'm not knowledgeable enough about the various BTC miner software versions to know, then gmaxwell's proposal would not work at all, and would be a huge pain since it requires a protocol change.

It seems to me that if you're right, then it's most likely a bug, since it's inconceivable that anyone would purposefully omit tx unless as T&D stated it was to maybe to reduce server loads on an already stretched system. Unless "mystery" steps forward, assuming it's even one person, we couldn't know their intentions.

Also, if this is the case, then the method proposed by Gavin/Revalin would be necessary. If a block has far fewer tx than the average, then it gets delayed, and if it has only 1 or none it may be excluded altogether.

There are several reasons why the "I'm entitled to charge whatever I want" argument has no weight.

1: I can charge $1million USD for a single grain of sand on ebay. Nobody will buy it (I would hope), but I still have to pay for posting it at auction. In BTC if you do the same thing you still get paid, at the expense of everyone using the system.

2: Gavin/Revalin's system still gives miners a good deal of room in choosing what they want to charge. As long as the threshold for delay isn't too restrictive, and the miner is charging a reasonable enough fee that they actually get some business, they shouldn't be excluded or delayed. Unless your business model is crap, there's no penalty. Also, since it's not actually part of the protocol, there's no obligation of any party to delay or reject a block, nor any obligation to accept any block, and it could potentially be adjustable by the user. I think, however, that the sentiment of most miners is that they also are not interested in supporting cheapskates.

3: The blockchain is 2 fucking gigabytes, and climbing at an absurd rate. If I have to download a 2GB blockchain, and 15% (or more) of all the blocks are empty or nearly empty, then why have I wasted my time downloading 300MB+ of that? The only purpose of the blockchain is to be able to securely verify transactions that have occurred, and empty or nearly empty blocks add filesize without contributing to the purpose of the data. At the rate things are going it's already going to take further development just to figure out how to keep the size reasonable, and the last thing we need is a bunch of retards being paid in inflation to spam it up faster.

Finally, if it is just a bug in one of the clients rather than someone trying to take shortcuts, then implementing a screening system would make it obvious to the users of that client that something is wrong. In that case it probably wouldn't take long for it to be fixed. If nothing is done, then we have no way of knowing.

The getwork size is the same, whether transactions are included or not.  The quantity of getworks is the same too, whether or not transactions are included.

If your calculations prove anything, then that its not a botnet.  "Because its too difficult to manage even a moderade sized botnet (ask Tycho)" - your own words paraphrased.

The fact that transactions are not included does not add one single jota of support towards/against the botnet theory.

*sigh*, well if that's true then either someone is charging a ridiculous fee, or there's a bug in some version of the miner software. If the difference in load and everything else between including tx and not including tx is basically insignificant, then there should be no reason for anyone to be purposefully excluding them.

On the other hand, if an empty block is really ~1/55th the size of a block with lots of tx, then it might really be someone trying to cheap out on bandwidth while still getting paid.

Either way, it ought to be stopped.

Nonsense.  The blockheader is tiny.  99%+ of the blockchain is transactions.  The number of blocks is based on time.  Empty or not the block chain will grow by (300 bytes) * 6 * 24 * 365 bytes per year = 15MB annually  (roughly double that if you include the only requires tx - the coinbase).  

That cost is fixed.  The only thing that would change that is a avg block time other than 10 minutes.

Transactions make up the vast majority of the blockchain.  This is why downloading blockchain is fast initially (first 50,000 blocks downloads within minutes but then it slows down).  Those blocks are mostly empty (yup even Satoshi mined empty blocks).


No botnet or even conventional TH/s farm would be running the standard client.  Especially not a farm (conventional or not) which is ignoring tx.

No please listen.  USING A POOL SERVER WITH A BOTNET IS NOT VIABLE DUE TO THE COST OF GETWORKS.  It was your "theory" that the nodes not being able to process tx was bogus because most miners don't.  However conventional pool miners need a pool.  I was just showing the math that current tx fees come nowhere near the cost needed to support a pool.  They don't for a botnet, they don't even for a small conventional pool.

The nodes likely are running a modified stipped down version of bitcoind.  It doesn't keep the blockchain, it doesn't need a db, it doesn't even need logs.  It simply connects to peers and looks for inv messages.  When inv message occurs the nodes internally use the last block hash, current time, no tx, the simplified coinbase, build a merkle tree consisting of 1 tx, build block header and start hashing.  Likely not all nodes are even running this.  To isolate the botnet only a "few" (as a % of total nodes) would need connections to bitcoin network.  They could rely new block notifications in p2p fashion to the rest of the swarm.

Essentially they are listen ony nodes.  They only broadcast to bitcoin network when they find a block.  Which if the avg node is 10 MH/s would only be once every 20 years (per 10 MH/s node).

Or that is how I would do it.  

The problem with delaying blocks is the following.  As a simple user, you may feel personal satisfaction by not relaying the block and it may not affect you in any other way.  But as a miner, you are in a difficult situation when you receive a block that you dont "like".

For as long as you plan to dislike it, you cannot mine on top of it - because your potential next block would be waste without the "evil" block. It is chained to it and eventually you will have to publish the "evil" block along with your own next block.  To make things worse, your chances of success are higher when you relay the "evil" block as soon as possible.  Otherwise you not only depend on your own luck (difficult enough), but also on the luck of the "evil" block (which you have tried to diminish).

Therefore, if you dont like the block, you must not mine on top of it.

This is a very risky strategy.  Other miners may not be as picky as you are, and then your mining efforts may end up waste.  You are intentionally mining on an old block, knowing that a newer block exists.  This can only play out well, when you are absolutely sure that most other miners ignore the block too.

Every time an "evil" block appears, the chances of an orphan produced by (self-proclaimed) "legit" miners raises.  It can be seen as a seed for orphans.  And I mean an orphan of "legit" work, not only the "evil" work itself.

(Remember that I am talking all this under the premise of the poposed intentional relay delay)

Although the percentage of seeds is only 15%, the chance of an orphan arising from it may be much higher.  It depends on the adoption of the protocol change and also on the individual configuration that each miner chooses.  We could very well face a situation where everyones own mining effort has a 50% to be orphaned after each seed.

Effectively every miner would pay a very high price for a just slightly better service.  At 15% seeds and 50% chance to orphan, EVERY "legit" miner would loose 7% income.

A high price to pay.

To pay for what?  Well for the slightly better service quality I suppose.  "Slightly better", because a TX can make it 15% faster into the chain.  However, there is also a negative effect, which is the constant rate of orphans and reorganization of the blockchain.  This could be seen as another kind of "Dos" or attack on bitcoin.  It makes 1 or 2 confirm transactions very unreliable, and can also affect 3 confirm TX or higher.

People who rely on low-confirm transactions will now have to wait CONSIDERABLY longer, not just 15% longer for the first confirm and 0% longer for every subsequent confirms.

We have already seen runs of up to 4 or 5 consecutive blocks with no TX, all from the same source.

The higher price paid by miners (in form of extra electricity with less BTC reward) may actually make the overall service quality worse.

Regular miners wouldn't use these lists at all. They can do whatever they want. The lists are meant to stop "dumb" miners that can't verify the chain for themselves from just blindly building onto the longest chain. This proposal makes all miners verify the chain or get someone else to do it. If this "mystery miner" is already verifying the chain properly, then there's no problem: he can set fees to whatever he wants.

But he can't.  If the majority of miners include 10 free tx on the "required list" for the next block then "mystery" (or any miner or pool) is required to include them or have that block invalidated by other miners.  He can't even exclude free tx without taking a risk his block will be orphaned.

Miners already have limited pricing power due to the fact that mining is a commodity business this ensures they have no independent pricing.  If clients know the list used by majority of miners includes free tx there is little reason for them to include a fee.

If this is the issue, and there's a good chance that it is, there are three possible solutions:

1) Make it more difficult for listening nodes to get what they need to process a block with no transactions.

2) Make it easier for listening nodes to get what they need to process a block with transactions.

3) Raise the transaction fees so that there's enough incentive for botnet operators to get transactions into their blocks.

It's all about incentives.

Mining & killing bitcoin? 
Make no sense.

You're talking about the "discouragement" proposal. I am not.

Discouraging blocks should be a last-resort tool. The incentives are wrong: building onto a discouraged block has no extra cost for miners, but discouraging a block has significant extra costs.

Interesting theory, I didnt think of this.

However, it does not address the fact that the blocks all come from the same IP (or small group of IPs).  In your scenario, it would be a random one out of 1.8M IPs.

True.  If the botnet operator is paranoid he may wish to minimize the connections to the rest of the bitcoin network (to keep the ip of nodes unknown).

In that case take my theory above and modify it so there is a single bitcoind active at one time.  When it detects a new block it notifies the rest of the swarm.  Using a mesh p2p like protocol the "gateway" nodes notifies 50 nodes who notify 50 each, who notify 50 each.  Each node is aware of the current gateway.  When it finds a block it relays it to the gateway who broadcasts it to the bitcoin network.

Really that would be sub optimal but very isolated.  If one takes the gateway down the operator promotes another node to the gateway.  By keeping the bulk of the swam unknown and with no direct communication with bitcoin network it is opaque and dark.

If the botnet has 1.8 million nodes and only one is active as the gateway at a time shutting it down would be tough.  Even if the ISP did shut it down.  Well there are 1,799,999 more nodes to go. :(

Likely the next question is "If D&T is right, why doesnt' the gateway simply include txs like any other node would?".  A little thinking I think will reveal a very obvious reason.

D'oh.


gmaxwell's solution might provide #1. If D&T is correct, then since gmaxwell's solution requires a miner to use the previous block's tx inputs (rather than just the hash) as verification, this would force mystery to either run more complete versions or bow out. The disadvantages are that it requires a change in protocol, and would increase the size of each block by an unknown amount.

For #2 it's been proposed to replace the blockchain with an acyclic directed graph, but I don't know enough about that to compare it in any way. Apparently this would make tx inclusion cheaper, although I have no idea how that would work, or how the filesize compares to the blockchain.

#3 unfortunately is probably not going to be viable for several years, maybe a decade. When the average number of tx increases by a large amount (maybe to 10k+) then even a small tx fee would generate substantial income, but as of right now, and for at least a few years yet, the block reward is much more significant, and has currency appreciation supporting it.

A possibly simpler solution would be a server-to-server version of 'getwork'. This would allow standalone mining clients to connect to any Bitcoin node and ask them which transactions they would include in a block they were mining. Right now, anyone can trivially design a standalone miner that mines blocks with no transactions. This would allow them to trivially design a standalone miner that mines blocks with transactions.

There's a "poisoning the well" problem though. If this is used primarily by botnets, it would be very tempting to include invalid transactions to cause them to mine invalid blocks. (The clients would have no way to validate them.) That would just cause them to go back to mining blocks with no transactions though.

Then Haplo's clarification is correct? (Thank you for that, Haplo.) So it means instead of forcing miners to include transactions, they would instead be forced to access recent blocks and extract data from them.

What's still a little confusing about that on a conceptual level is this: mining ALREADY requires access to recent blockchain data, in the form of a hash of the prior block. Even if it required new code, couldn't the method already being used to distribute that prior-block-hash to all the nodes on a botnet be copied/modified to also distribute the newly-required "proof of verification?"

Refusing to relay a valid transaction is different than refusing to recognize it as valid.  In theory, a talented coder can set relay rules for his client already.  There is nothing preventing nodes with additional relay rules from rising any more than there is nothing preventing a miner from setting additional block inclusion rules, but those only affect that miner and those who choose to make use of his code & ruleset.  A single miner who refuses to relay valid blocks produced by a competing mining pool amounts to little.  But if a significant number of nodes have a similar relay rule that delays the transmission of a valid block without transactions, then that rule becomes a defacto sanction against the miner who produces null-set blocks. 

And to that end, this is what I dug up researching this today:


Some food for thought.

I think so.


It's not just recent blocks: they'd need to be able to access random unspent transactions in the chain.

The proof of verification could be distributed, but then at least you know that someone is doing the proper verification and miners aren't blindly building onto invalid blocks.

To encourage users to use the network.

A network's value increases as its userbase increases (and vice versa).

reread what you quoted.

Personally, I favor a drastic rule such as

* Build a list of "likely to be in next block" transactions, from memory pool
* Do not relay blocks unless they contain at least 50% of the transactions in the "likely to be" list

This is "drastic" in my estimation because such a rule has notable downsides,

* Increases possibility of short term fork
* Creates de facto requirement that at least 50% of each block are standard transactions (as defined by isStandard)
* and some other minor fallout

However, it is a strong rule that does address the issue at hand, while permitting valid, no-tx-activity zero-transaction blocks to exist.

So miners are required to include tx that don't meet their fee requirements?  If they don't they risk having the valid blocks not forwarded?

That's similar to what Gavin and Revalin proposed, the downsides of which have been discussed at length :P.

Gmaxwell's proposal should achieve the same effect without the downsides of a threshold-based approach. The main advantage of an approach like yours is that if the bot owner finds a way to include verification without including current tx, he still gets shut out for non-cooperation.

I think if you were to do it that way, a much lower threshold would suffice. AFAIK our mystery miner only includes a single token tx in his blocks, or none. With ~80tx average per block, a 5% threshold would neatly wipe them out. At that rate, miners have little to complain about, and the likelihood of a fork is lower. At 50%, a pool might end up having an otherwise valid block excluded if they happen to charge on the high end of fees. At 5%, a pool charging enough to get excluded would not be providing anything extra to its users (if any).

What would happen if everyone did this and I send out a transaction tomorrow with no tithe to the system. Where does that transaction go? How would the user have any idea what has happened? How would they know if it still floating around out there, maybe waiting on a generous miner who has a long pro bono publico wait list, and that they don't need to resend it?

What if the fee requiring miners are in the minority but each of the 8 node connections that my client has just happens to have this fee rule? Again, does my transaction get lost in the nether? How would I get my transaction out to the miners who are willing to include my transaction? Would the nodes say "nope, fees too low on this transaction, into the round file it goes..." or would the nodes say "nope, fees too low on this transaction but maybe Bob down the street would be willing to help you out, I'll pass it on"?

The modification I was proposing wouldn't affect relaying so tx would be relayed to every node even my node would continue to relay it.The tx is valid so rejecting it is inappropriate I just don't want to mine it.

As far are mining it into a block.  If every single miner on the planet (and all new miners coming online) had a min fee rule with no exceptions higher than your fee then it would never enter a block.  However that is obviously unrealistic.  If 10% of miners had such a rule then 90% of network would still include it in the next block so your expected confirmation time would simply be 11 minutes vs 10 minutes.  If say 99% of network excluded it from the next block (due to insufficient fee) then your expected confirmation time would be 16 hours.  Obviously both of those numbers are expected confirmation time, the actual time would be subject to variance just like any confirmaiton time is.


The later.  My proposed change would only affect what tx go into the next block and only for those miners/pools which implement the patch.  Any valid tx would continued to be relayed.

Yes, you will be orphaned if you set a "fee > 100BTC no exceptions" include policy.  It is not reasonable to demand that the network have no right to question your policy if you institute a bad one: the whole point is to stop people from excluding all txes.  We're trying to find a minimal method that won't affect any reasonable include policy, but it does create a requirement that you might have to do something.  A relay policy with no actual mandate is pointless.  As I said:  Checks and balances.

You're becoming a broken record.  Your point has been heard and is being taken into consideration.

Please don't use strawman to support your argument.

What if fee policy was 0.01 BTC?  Less than 10% of txs in last 20 blocks had a fee of 0.01 or higher.  Hell less than 1/3rd have a fee of any kind.    Thus a requirement to include 50% of txs mandates including free txs also.  Any fee policy no matter how low (even min 1 satoshi) would face exclusion.

So at least be honest with the ramifications. 

It's not a strawman.  I'm just emphasizing that some minimum standard is reasonable.  I'm sure we can come up with a method that will work.

Rather than having a minimum bar, why not make it a sliding scale?  For example:  you must include at least 20% of the top-10-percentile-fees (either since the last block or sliding-window) transactions?  As always, -5 or so to accommodate latency etc.  That would give you a lot of latitude to cherry-pick.  In many circumstances (only a couple transactions with high fees available) you would even be able to mine a null block and have it accepted.

Edit:  Or rather than allowing a null block, we could mandate that every block include at least one transaction if there are at least 5 outstanding.  Is it an undue imposition to include one free transaction if there are absolutely none that meet your include policy?  That provides a very strong incentive to include fees, and completely solves the null-miner problem:  Mystery can mine a null block if he wants to - but only in the few seconds after the a previous block that cleared out all waiting transactions, during which a null block is reasonable by any standard.

The goal is not to force any particular policy but just to avoid miners being lazy and not gathering any transactions at all. So long as they can't take the shortcut of not including any transactions at all, their self-interest will lead them to include fee-paying transactions.

So the rule could be this simple and this modest: If there are at least four transactions that are not free and have at least the standard fee that were not included in the previous block even though they could have been, discourage any block with just a coinbase transaction.

So you are never punished for not including a free transaction. And you are never punished for not getting a new work unit when you didn't have to anyway because a new block was found. But this will solve the problem of paid transactions backing up if multiple blocks are found by miners who refuse to include any transactions.

This is, IMO, the least onerous rule that would solve the problem.

You also need to discourage the block if it has only transactions you don't know about, since otherwise the miner could make up transactions.

I'm not bothered by this rule from an economic perspective, but I'm not confident in the effectiveness of discouragement (due to bad incentives), and I'm uneasy about relying on transaction propagation in any way, since propagation is imperfect and will get worse with time. I prefer gmaxwell's proposal because it only relies on the block chain.

Just wanted to say that this makes a lot of sense to me.
It still probably needs some detals, like only take into consideration transactions which are older than 1 minute or so (so that they have very high probability of having been globally-relayed) and perhaps not enforcing this rule if the total number of unconfirmed transactions (that are "supposed" to be in the next block) is too little. (This can happen due to a bug, or, what is more realistic, due to 2 blocks being found very short time apart (which can happen, it's all just probabilities)). So for example if there are only 5 transactions that are supposed to be included, don't enforce the rule, because in that case the outcome of the percentage calculation is biased too much by every single transaction (and thus much more prone to errors and can create unnecessary forks).

The market purists on this forum should agree that the market is working exactly as it should.

The block award is for securing the network against a double spending attack, not for confirming transactions. Transaction fees are for confirming transactions.

The null miners are essentially specialized miners that are only in the business of security, and not in the business of security+confirmation like most other miners.

I don't think this is a huge problem. Even if the null miners had 50% of hashing power, average users would hardly notice any difference. Even if they had 90%, bitcoin would still work fine. It would just take longer for transactions to confirm. 

This problem is likely to disappear in future because:

a) The block award will drop
b) As bitcoin becomes more popular, there will be more transactions per block, and thus more fees to be collected
c) FPGAs and ASICs will marginalize the hashing power of botnets.
d) Most day-to-day transactions will be between trusted parties who don't care whether a transaction isn't confirmed yet, and who don't even broadcast their transactions most of the time.

Is it worth adding complexity to the bitcoin protocol, with possible unintended consequences, in order to solve a temporary problem?

Several reasons.

1: The block reward may drop in half come 2013, however if the value of 1BTC doubles, then nothing has been lost. That leaves him fully secure to continue ignoring tx until at least 2017.

2: He isn't actually contributing to security, since he's only generating blocks off the previous hash without validating anything. That means if an attacker starts throwing down invalid blocks, our friendly botnet might blindly build right on top of them and give them a nice helping hand.


Also, it depends on which method is used. Gmaxwell's proposal would not affect legitimate miners in any way. Any proposal which excludes blocks that are "too empty" would have unintended consequences, but in the event of an attempted 51% empty-block DoS attack, the ability to screen empty or cheap blocks would severely limit the damage they could do.

We've been over this like 50 times =\.

Not true. All the building on top of an invalid block can't hide or change the fact that it's invalid and so would do no harm whatsoever. It would simply be wasted effort.

These botnets do increase the amount of hashing power an attacker would need to launch something like a 51% attack. They do increase the security of the network. And that's what the block reward is for. If the system is broken, it's because the transaction fee and volume aren't sufficient to justify the expense of including transactions in a block.

I wonder if we can detect these botnet nodes somehow, perhaps by the particular messages they send or don't send, and send them *invalid* blocks. It won't harm normal clients since they'll just ignore them. But the botnet nodes, lacking the block chain, will work on the wrong blocks more often than the right ones. This may do more harm than good in the long term though.

So what is the downside to this again?

This seems to solve the problem by setting a reasonable limit on miners' ability to extort the network for fees.  We really should consider a somewhat drastic solution.  Even though the mystery miner is likely a botnet, within a year or so we could easily see this same scenario play out with a large FPGA cluster that doesn't want to include transactions for whatever reason.  It isn't a problem that is going to just go away, or be solved any time soon by falling block rewards.  We're already subsidizing miners with 30% inflation;  it might as well go to those that at least process a few transactions.

They aren't becoming a problem. They are simply making transaction processing more valuable. When the incentive comes for them to process transactions, will be the day a higher fee is set across the board.

I see no issue. If the problem becomes bad enough, people will talk with their pocketbook.

Although, you could always coerce the miners into giving you free transactions. As for the consequences to Bitcoin as a whole in such a case, I'll leave that to you to figure out.

If the top 10 transactions are 1 satoshi then that forces miners to include all tx with at least a 1 satoshi fee.  It creates a race to the bottom.  Smart users will keep fees as low as possible and encourage others users to do so.  "Look as long as nobody pays too much the miners are forced to include are freeloader txs".

Your insulting language is counterproductive.  You are aware that passing any breaking change on block validity rules will require support of a majority of us "greedy miners".  Also money supply growth isn't 30% anymore and after the subsidy drop it will be closer to 10%.  By the next subsidy break it will be ~4%.  Users have enjoyed price deflation due to the fact that the demand for currency has grown faster than the money supply.

Okay I was only suspicious before, but now I'm nearly convinced that this issue has done much more to harm Bitcoin by bringing to the fore the absolute lunacy of some miners.

Some of you seem completely oblivious to the fact that nearly every Bitcoin you "earn" comes at the expense of the scant few actual producers and investors and participants in the real economy.  Half of you could fall off the face of the earth and the Bitcoin network would be no worse off.  In fact, the only reason this hasn't happened is because you are being subsidized by the non-negotiable inflation built into the protocol.  The abundance of zero-fee transactions should give you a hint as to the marginal market demand for mining services.

"Price deflation," give me a break.  That is entirely paid for by the few businesses willing to accept Bitcoins as payment for real goods and services.  And for the amount of time that Bitcoin has been in widespread public view, that number is dangerously low.  Accepting the premise that miners can earn rewards without processing a single transaction not only doesn't encourage new participants, it screams "stay away."  You are shooting yourselves in the feet by doing so.

For the record, I didn't call you "greedy".  I called you "short-sighted" and "extortionist".  If you were greedy, you would at least act in your own best interests.  Many of you can't even seem to do that.  Take a step back, do a little math and think about how many transaction fees you will be earning a year from now, if the average block still has only 50 transactions in it.  Then ask yourself why you are here arguing for higher transaction fees, which are ultimately insignificant compared to the block reward, and which will only serve to discourage growth in transactions between now and when the block reward goes away.

Did it never occur to you that the uncertainty caused by having 15% of the network not processing transactions is already hurting your profitability more than accepting a solution ever would?

Come on, get real.  What is the possible market for international transfers in an illiquid digital currency nobody uses?  Do you seriously think Bitcoin does anything that Western Union can't?  Do you think anyone will pay a premium to use a payment service that isn't even denominated in their local currency?  How long do you think Bitcoin will last if the only people using it are criminals and drug dealers willing to pay exorbitant transaction fees?

You haven't monopolized the secret directions to El Dorado like you seem to think you have.  You have simply drank the Kool-Aid, and are going to be completely blindsided when this desperate grasping for every last BTC ends up backfiring spectacularly...

Actually... I think we should re-think this. Why not just let them make up transactions? It seems that if they include any transactions (maybe needing to add up to a minimum amount) that this specific problem is solved.

To make up a transaction, the following must be true:
1) You hold the private keys for the transaction or have a method of getting a transaction signed. (this would be death to 100% decentralized botnets (centralized botnets would just make a mining pool))
2) You know that the transaction isn't a double-spend. (so a decentralized botnet can't give the same private keys/pre-signed transactions out more than once without each computer holding a blockchain)

Honestly, I can't believe that this didn't occur to anyone else, and that we've just kept saying "the miner could make up transactions" to dismiss such an effective idea. By requiring at least one (reasonably sized) transaction that isn't confined to the block itself, we finally have a link to previous transactions that MUST be verified. If it isn't verified, your chances of creating invalid blocks rises significantly.

Another way to look at that: the reduction in the mandatory fee for large transactions was reduced too much, too quickly within the main client. While I personally pay those fees often enough, it hardly amounts to much. At the very least, the mandatory fee shouldn't be reduced again anytime soon (a $100 price for bitcoins would still make the 0.0005btc fee be only worth $0.05. At current prices I pay one-quarter of one cent per fee.)

Similarly, what is the default optional fee that the latest version of the standard client is set to upon install? If it's less than 0.01btc (currently $0.05 per tx,) perhaps raising this default optional fee to 0.01btc in the next release would be a good start at addressing the long-term low-fee problem, since it wouldn't change any aspect of the code or protocol, and could be reduced by users as needed.

Even if it defaults to 0.01btc already, maybe an additional notice could be displayed whenever the user reduces the optional fee, including text such as "Lowering this fee, while possible, is strongly discouraged.")

I can't see any practical objections to at least starting with that.

Where do you get this garbage about exorbitant transaction fees?  A bit penny is exorbitant?  In what universe?

If you think the value and utility of Bitcoin is so useless why are you even here?


Srawman aside I never claimed I had.   You seem to have a very deep seated hatred of miners.

You seriously think, at current prices, that a miner charging a 0.01btc per-tx fee is exorbitant? And in the same context as comparing Bitcoin to Western Union, no less?

While I can see how many would think that's a bit much (as I do,) I hardly find that to be anything close to "lunacy." In fact, such a strong reaction to the idea has me wondering who exactly it is that's desperately grasping for every last BTC they can get at.

That'd make things a little more difficult for botnet operators, but it can be bypassed. The botnet software could listen for blocks and transactions on the Bitcoin network and include them without checking them or storing them. The resulting blocks will usually be valid, since legitimate nodes don't relay bad transactions. The botnet could even "check" each transaction by seeing if its other peers either already have the new transaction or will accept it.

I just realized that this attack also applies to other proposals requiring some amount of transactions from the memory pool to be in the next block.

Spell out the contradiction, friend.

I've found myself asking this more than I would have expected lately.

Although to be fair, when you have an entire mining pool willing to use it's resources to attack any new alt-coin that springs up (in similar fashion to this, I note,) then it's not as if people are very free to just create their own alternatives and go run with them... they're pretty much stuck with Bitcoin if they like the idea of a cryptocurrency but want changes to it.

The law of unintended consequences rears its ugly head again.

An alt-coin which uses a different PoW (or PoS) would be immune to that particular attack.

So would any alt-coin that wasn't a dump and pump piece of junk.  You know one that actually submitted a proposal, took in feedback, had a launch date announced well in advance, had a working testnet, got support of multiple pools and miners to have sufficient strength on day one.

Of course no alt-coin has shown any innovation, any progress.  They are wothless pump and dump scams. 

My point still holds. Alt-coins being attacked by legitimate bitcoin pools that take it upon themselves to police the world of cryptocurrencies has a chilling effect on dissatisfied bitcoin users who might otherwise have tried to create a non-pump-and-dump alt-coin.

I personally don't need to be saved from the scammy alt-coins. I'd rather be saved from all those trying to change bitcoin into their vision of an alt-coin.

Even though benjamindees seems to be on a warpath against the miners, he has some good points. This issue, even though it's not critical, is one more thing that makes Bitcoin suspicious overall. I'm one of the biggest Bitcoin-enthusiasts out there and even I don't like this at all. I can't explain something away to other people if I'm bothered by it as well. In addition I'd like to point out some important aspects to this.

First of all, some of you claim that it's just a matter of finding a tx fee high enough to incentivize this mystery miner to start adding transactions. I think this is entirely false. From what some people have speculated, the benefits he could potentially gain from not adding any transactions, or not verifying transactions at all, can be sort of huge for him. There is a possibility that he simply doesn't care what the tx fees are in the transactions because he doesn't even look at them.

This means that we could have a tx fee of gazillion and a gazillion people paying those tx fees, but he still wouldn't care, because it would possibly lead to many of his miners to be discovered and thus his whole mining operation would be threatened. It could certainly be that Bitcoin needs to become the most expensive money transfer method in the world for him to care, and then it wouldn't even matter anymore. Who would care about Bitcoin if it was that expensive?

So not doing anything is basically ignoring the issue and hoping it goes away or doesn't become too bad. My other point is somewhat related to this, I'm starting to feel that there is an ideological conflict going on right here. We have people who want to desperately hold on to a pure market approach even when we have a clear exploit that allows miners to leech on the network. Then we have people who think that some restrictions are appropriate to stop the leeching.

Then there is this notion that the fixed block rewards are for creating blocks and tx fees are for adding transactions, this has to be the most ridiculous idea I've seen in this thread so far. Who came up with this? I'd like to see Satoshi saying that, I have a hard time imagining it though. Bitcoin is a cryptocurrency and for a currency being able to transact is critical. Ín the case of Bitcoin it's critical to be able to transfer money cheaply, that is one of Bitcoin's major benefits.

Again, I'm not saying that transactions should be free nor that tx fees should be dictated in a socialistic way, but being able to intentionally not work with transactions at all and still get a 50 BTC reward is one hell of an exploit that is likely not going away with the block reward drop.

So we will at least have to put up with this for 5 years, possibly much longer. This is because if Bitcoin continues to grow, bitcoins will certainly be more valuable in the future. That will make sure that nothing really changes as far as mining is concerned, our network just gets a bit less secure relative to the market cap. From the perspective of our mystery miner, nothing changes.

Finally, I think that this issue is not big enough to put a solution in place that either heavily restricts a future tx fee market or has other major side-effects. For example I feel that jgarzik's proposal was too drastic and now it seems, at least according to theymos, that basically all of the recent proposals are problematic. I don't support doing anything hasty about this issue so let's just continue to think it through.

No. The path forwards is to discover how this is being done, so that people whose computers are compromised can get rid of the malware.

Meanwhile, empty blocks are a minor irritation but do not threaten the foundations of Bitcoin.

Here are Satoshi's thoughts on the matter (http://www.mail-archive.com/cryptography@metzdowd.com/msg09967.html) from the early days of Bitcoin:

I really don't understand how some of you simply ignore the fact that transaction fees are irrelevant in comparison to block reward, and will be for the forseeable future.  Even if they weren't, there are plenty of people out there with the incentive to disrupt the Bitcoin network by mining without including transactions, regardless of what the fees are.

Nothing you have suggested so far changes that.

Not sure what your definition for "forseable future" is.  As long as tx fees are essentially 0 so no matter what the block reward is ~0/anything is still essentially 0%.

Even a modest rise in fees can provide meaningful change to that dynamic.  Hypothetical scenario.  Over next 12 months tx volume doubles to ~100 per block and block reward falls to 25 BTC.  If miners start enforcing fees and avg fee ends up being just 0.01 BTC per tx that is 1 BTC in fees and 25 BTC reward.  Fees are a small but hardly insignificant 4% of total revenue in just 12 months.

Hopefully on a longer time frame say 3 or 4 years tx volume will quintuple (or more).  At say 250 tx per block and 12.5 BTC subsidy.  An avg fee of just 0.01 BTC per tx results in fees making up 16% of total revenue.

If USD:BTC remains the same then mystery faces falling revenue and will need to adapt (or make less).  If USD:BTC prices then global hashing power will also rise as miners chase larger block rewards (including fees) and "mystery" makes up a smaller portion of the network.

Going forward bitcoin will need to be supported more and more by fees anyways.  Any "problem" myster presents is temporary at best.  Breaking changes to protocol to "fix" a temporary problem seem dubious.

It's a simple fact that any extra incentive to mining including transactions will make mining more profitable to those who do, and will push difficulty up, directly reducing the power of ANY individual entity.

Simple logic really: more reward to mining including transactions => more proper mining, more difficulty, less baseline reward for everybody.

Whatever else is done, this step sounds trivial and it's also fair. It also reduces uncertainty about the reactions and instability that block reward halving might cause.

This is off topic, but I honestly don't think that the first block reward drop will change things that much. It will make tx fees slightly more relevant but still not very relevant. If you presume that Bitcoin usage is going to grow in the coming years, then Bitcoin price will most likely more than double as well. Also the smaller inflation will probably increase appreciation in Bitcoin's value. That will basically compensate rewards in a way that the total miner reward in fiat is likely to actually increase significantly per block, which will mean more mining investments, more competition and a more secure network in absolute terms (although weaker in relative terms).

In relative terms the network is bound to get much weaker than it is now but that is actually fine, at least for a long while. The security level in relative terms is overkill right now and in absolute terms it can only get better if Bitcoin continues to grow. Eventually we'll reach a point where tx fees have a significant role and then we have an interesting situation where we basically switch to a tx fee market system that works or if that fails, the only other solution to providing a safe enough network is a PoS/PoW hybrid.

Now back to the topic. I have to agree that it's possible that this problem could get less significant in the future and simply go away eventually but we can't know for sure. It could also get worse if more botnets start to apply similar methods. I think it's healthy to think about possible solutions if this gets worse. It's bound to get better eventually just like you guys are saying but the issue is how bad does it have to go in the short term to be unacceptable.

My problem with this hasn't been the practical issues of it, it doesn't increase average tx confirmation time by much and that angle won't change significantly even if the percentage of no-tx hash power increases. The problem is that the mystery miner entity is partially leeching on Bitcoin and I don't like it, and apparently there are a lot of people like me. But it could be that trying to apply a quick solution to this could be problematic in the long term. The best solution to this would be if we could somehow expose the botnet (if it indeed is a botnet) and stop it that way. That is very difficult as well though.

This is the broken window fallacy, by the way, Atlas.

Agree.

Perhaps the mystery miner includes transactions which meets his fee requirements as well?  He's just set a high fee requirement.  Try a transaction with 50 BTC fee, and check if he includes it.

The point of the proposed change is to reward responsible miners who actually contribute to the network with more blocks.  Satoshi suggested a part of each block to be free, and miners who believe in Bitcoin as a currency include those free transactions.  If Bitcoin transactions get more expensive than PayPal transaction (which you actually suggest for small transactions later in this thread), then people will use PayPal instead, making Bitcoin less valuable for everyone.  It is not in the interest of most miners.

Paypal costs $0.30 plus 2.9%. 

Current fees are roughly 0.001131751 BTC per tx. 
Even if tx fees on avg were 0.01 that would be an 883% increase but still a tiny fraction of Paypal.

However compared to Paypal (using current BTC:USD exchange rate)
$1 USD transferred  Paypal fee: 0.07 BTC (7x higher)
$10 USD transferred Paypal fee: 0.13 BTC (14x higher)
$100 USD transferred Paypal fee: 0.70 BTC (70x higher)
$1,000 USD transferred Paypal: 6.50 BTC (650x higher)

MtGox for example never includes tx fees.  One of the most sucessful bitcoin ventures and it provides nothing in compensation to the network.  The Bitcoin network will eventually need to be run on fees.  Nobody is saying massive fees overnight or that even 1 bitcent is going to make up the majority of block rewards but it starts the ball rolling.

Currently: ~0% fees, 100% subsidy
Someday: 100% fees, 0% subsidy

Even IF average fees were increased 800% (<1 bitcent per tx on average) and tx volume doubles after the block reward declines to 25 BTC fees would only make up 4%.  Still I consider 4% more healthy than 0%.


One Bitcent tx fee would increase fees

There are two issues at hand, which are separate:

1) what TX fees should be
2) how to deal with mines not fully participating in the network

A flat fee of any significance is incompatible with micropayments. 1bitcent seems a very steep fee for something like a NYT or WSJ article I wouldn't pay more than 1bitcent to read :)

I've read the discussion about this from when satoshi was still around - I'm aware he didn't think Bitcoin to be well suited for micropayments yet, but I don't think that setting a flat fee is necessarily the right thing to do. People want to pay other people micropayments, and we should let 'em if we can. I've actually wanted to send such payments myself, and didn't because of the fee.

I'm not sure I understand why the fee is always discussed to be flat or per KB.

Maybe the client could recommend a Percentage based fee + 1 satoshi per output?

- Personally I like recommendations. Having the option to send for free for now seems important. It's not affecting the mining bottom line to allow free TX,  why not have a distinguishing factor in the market to use bitcoin? Some time in the future it may be determined that free TXs do not have enough of a chance due to blocks being full of paid TXs. It is only at this time that we MUST change the fee structure (on the network side)

I'm not an expert on #2, but it didn't seem to concern satoshi, and it doesn't look like anyone has raised serious technical concerns in the near to mid-term with MM not relaying TXs. It's not a serious security issue at this time. Technomage suggested taking time to consider the long term approach. This is correct. Making changes in haste will only create new problems down the road - problems which may not be so simple to correct then. If this system is to last 20 or more years, we can't think about this just in terms of what miners and users want today, but what will be compatible with what they'll want 20yrs from now.

Because the network has no idea how much the tx is for.  It only knows the inputs and outputs which are greater than the amount being "spent".


It could but miners have no effective way to determine what that % is. 

I lay out some reasons why % based fee rules are not viable.
https://bitcointalk.org/index.php?topic=73941.msg819583#msg819583


There are alternative methods of handling micro payments.  Long, only read if interested in alternative payment scenarios. One would be bulk contracts.  Say WSJ wants to to charge 0.01 BTC per page.  WSJ could pregenerate a list of payment addresses to provide users.  It would provide a copy of the public addresses to a mining pool.  Since micropayments are unlikely to be a source of fraud and likely need to be 0-confirm anyways WSJ can wait for longer than normal confirmations.

The pool could create a contract with WSJ for a flat fee or a nominal fee per tx. So on a per tx basis maybe the pool has a fee requirement of 0.5 cents (USD nominal) however by signing a bulk contract with prepayment the WSJ effective cost is (1/10th of cent per tx).  WSJ pays $500 and pre-purchases 500K low priority tx.  The pool fills blocks as normal w/ "priority tx" and then fills up the blocks w/ these lower priority tx as space is available.  Sometimes WSJ gets 1 block confirms but often it is 4 blocks, 8 blocks, 20 blocks.

The analogy would be low cost bulk mailing from postal service vs Fedex overnight.  junk mail doesn't work at Fedex prices it doesn't mean everything needs to be one size fits all. 

Now hypothetically if enough micro tx demand existed a broker will acts as middle man.  It would contract w/ multiple pools for their "junk mail rates".  The broker would sell the services to multiple micro tx receivers.  The brokers cut would be based on the difference between what they contract mining pools for and what they sell that service to service providers which use micro txs.

Got it.

I really like the suggestion on how to handle micropayments. What i like most about it is it provides miners a means of competition and distinction, and it also gets them into more business than just mining, which, long term, is probably  thankless, low margin work.

I don't see micropayments being a problem. Even if the average tx-fee paid rises, there will likely always be free transaction processing, even if it's just the company accepting the micropayments that mines those blocks. It will just take longer on average for such transactions to be processed and receive their first confirmation. So unless you're wanting that $0.05 you sent to your friend to be available for a resend in 30 minutes, micropayments should be handled just as well later on as they are now.

Similar system would provide non-51% double spend protection.  Broker gets contracts for 51%+ of hashing power.  Merchant sends tx to broker who sends it to contracted pools.  Pools guarantee they will include tx in the next block and will not replace it will a double spend under any conditions.  A merchant (like say Walmart) would have a high level of confidence they couldn't be double spent economically.  Walmart would process tx instantly using the guarantee provided by a broker.  Likely that 0-confirm guarantee would be worth a lot more than say normal tx processing so it would provide an additional revenue stream for pools/miners.

Cross border personal transfers cost 1%, which will make Bitcoin more expensive for any amount < current price at a fee of 0.01 BTC per transaction.  Double that for the transfer to your spending wallet (free in PayPal).  A quick check of my main spending wallet shows that 88 of 320 transactions are 2 BTC or less.  With your fee policy, PayPal would be cheaper for more than 1/4 of the transactions from my personal spending wallet!  Take away one of Bitcoin's most important advantages, and see what happens to the value of your mined coins.  Please try this experiment on one of the alternative chains first, and see how successful it is.
Currently all normal transactions (with the exception of spam and fast laundering) are free.  For transactions resembling spam there is a fee of 0.0005 BTC.  In very rare cases, like when a block is mined by a malicious miner, a free transaction have to wait a block.
This is an increase of 0.01 BTC for almost all normal transactions.  Currently 0.05 USD.  Less than a year ago it would be 0.31 USD per transaction, which is higher than even the normal payment PayPal fees.  And it comes in addition to the collective fee implied by inflation.
If MtGox had to pay fees, guess who would pay the fees.  Their customers.  This goes for all commerce.  Nothing comes for free.  If you want to pay 5 BTC for some service, you will have to buy 5.01 BTC.  Or 5.02 if you pass through your own wallet.  Only 10 US cents by the current value, but you know as well as everyone else that the current value is quite low compared to what it was.
Substitute "subsidy" with "inflation", and you have something there.  A tax on everyone who own bitcoins.  And "someday" is not in our lifetime.
Yes, and 25 BTC is still a very high block reward.  4% fees in addition to that is irrelevant.  Free transactions is a major advantage, and it will solve itself if it becomes a problem because free transactions will no longer run smoothly if transaction volume increase enough.  If you want those 25 BTC to be worth something, you need to mine a fair share of free transactions.  And follow the standard rules to make fee polices predictable for users.  Most of us don't do any adjustments to the fees.

Out of curiosity: Do you pay 0.01 BTC fee for your transactions?

sturle I don't think you get the idea of a free market.  

You think 0.01 is too high..  Fine don't pay it.  some miners set fee requirement at 0, some at a single satoshi, some at 0.001 BTC.  Some at 0.01.  The market determines the price.  The intersection of supply and demand.

You make it sound like if I (me little ole D&T) set min fee of 0.01 for my 0.15% of network suddently no tx which a fee less than 0.01 will ever be processed.  Take all your arguments and replace 0.01 with 0.001 instead.  Problem solved.


Extrapolating the cost when BTC is worth $30 or $100, or $20,000 is just intellectually bankrupt.  Obviously no miner's fees would be fixed in stone for the life of the GPU.  I think a tx should be worth a couple cents.  If BTC price doubles the tx fee can be cut in half and miner generates the same.  As BTC rises competitive pressures will ensure prices are held in check.  That is also a part of a dynamic fee market. Even the spam fee eventually will need to be adjusted.  OH NOES BTC has a 0.005 hardcoded fee on tx smaller/younger than 1 BTC/1day.  When BTC is $100,000 each that means all tx will cost $1,000.  It will cost $1,000 to send $1.00.  OH NOES OH NOES SELL YOUR GPUS NOW BTC IS HORRIBLY HORRIBLY FLAWED.

Ironically you just pointed out something the total cost to users (in terms of fees + inflation) is falling.  Even with a 0.01 fee on every tx without exception (as if I could control 100% of all hashing power on the planet) the implicit cost is lower today than even a year ago and will be significantly lower after the subsidy drop.


Of course not.  Why would someone pay more fees than necessary?  That is the point.  Charging a higher fee gives those who pay more a higher level of service.  

Except then you've killed de-centralization, and added more costs between customers and merchants, and as has been mentioned, between customers and their own private addresses. Also, if the network as a whole doesn't guarantee against double spends, then the security isn't high enough for a merchant to be confident using it in the first place.

The miners don't deserve to be throwing that much weight around, as it just loads costs onto the coin without providing any value that isn't available by cheaper and more reliable means.

I think per-byte or per-kb fees are reasonable, but there has to be some way for the market to regulate fees, or else a single large miner charging 100 times as much as everyone else could cause tx to become so unreliable that the currency is choked out of existence. For example, if "mystery" included txs, but charged 1BTC to do it, just getting over 85% confidence rate that your tx will commit would cost 1BTC.

Certainly 100% commit should be more expensive than 50%, but the blockchain is a tragedy of the commons type problem for one, and for two you don't need tx fees to operate, and you probably won't need them until 2025 or later, if BTC even survives that long.

I would say, every block should have the fee rate of the producing miner published with it, and if the fee is > 2 orders of magnitude greater than 50% confidence, or if the miner doesn't include the tx expected for the fee they charge, then the block should be excluded. The only problem with doing it that way is that, since currently most tx are free (which I think was a bad decision on satoshi's part), there's no comparable fee at the 50% confidence mark. Using max(MinFeeCharged, 50% confidence) instead could probably get around that. With that system, if 50% confidence or MinFeeCharged is 0.0001 BTC, then you could charge up to 0.0999 BTC and still get published, with no penalty for charging less or nothing.


You get paid a subsidy, idiot. You aren't subject to the laws of supply and demand, any more than mystery is dependent on including tx for his income. Also, there is little/no room for competition of service in this market; there's only one quality of service and the only variation is how much time it takes to verify a tx. The only thing a greedy miner can accomplish by charging an excessive fee is to slow everyone's tx down so that BTC is less competitive as a currency.

Based on current interest rates and the usual amounts of currency that are exchanged, yes .01 BTC is exorbitant, and would kill the market if everyone used it. The fees you're suggesting are in fact higher than much of the business that is done, and are multiplicative since any business transaction will generally involve several re-organizations of coin by each party. That's exactly the sort of thing that needs to be prevented.

Compare your fee of US$0.05 with the usual tx fees charged by a stock exchange router: US$0.0005 or less, which is even less than the current spam fee and they don't even get subsidies.

I can't believe how freedom scares so much people. Miners are free to choose what transactions are going to add to the mined block the same way we, the users, are free to select a fee according to our needs.

Aren't you agree? No problem. Mount a mining rig and start mining allowing free transactions.

The same is doing D&T. He's paying for a patch wich allows him to select transactions to add in his blocks. He's not forcing you to follow him. But if a lot of miners agree with him.... well... that's the power of democracy and you must accept it or start your own mining rig.

In the future I want a free mining market where miners can pick the tx with the fees they want AND, very important for fighting MtGoxtatorship, some big companies offering laundering tainted coins included in fake high transaction fees .

As long as they are including valid transactions, who cares if they have the entire block chain, or by what method they are selecting transactions to include in the block?

Any invalid transaction inclusion will result in their hard work being wasted, as that block would be disregarded by all.  Miners only get paid if the blocks are relayed, accepted, built upon.  Picking random transactions seen since last block, without any validity checking, is a valid strategy, if inefficient and potentially costly.

Hmm.
A) Isn't this nearly the same (if the broker is powerful enough) as actually having 51%+ of the network? What if the broker made a deal to get miners to not process other brokers' TXs?

B) even if the above is no, how can the broker guarantee to the customer 51%+? He could be unaware that his competitor has seen his brokered control over the network, and contracted to increase the total network size to remove the possibility of the first broker executing any 51% attacks. I don't see how this can be offered.

I do like the general idea of the broker though.

Hrm, it occurs to me that any miner charging a lower fee would be able to collect higher fees in addition to lower fees, so between two miners with equal hashing power, the one with the lower fees will make more money.

The main problem arises directly because of the subsidy. If a merchant is doing a significant transaction and wishes to ensure a fast verification, then the customer must pay for it. When a miner gets a block reward, both the customer and the merchant are paying for it.

1: Because of the subsidy, a miner with significant hashing power could charge impossible fees and still get paid virtually 100% the same amount. The advantage of doing this is limited, but not zero, and generally reduces the use-value of BTC as a whole. Mystery has already demonstrated a good reason for doing this.

2: If the miner in question was a 51% attacker, they could legitimately charge impossible fees on purpose in order to DoS the network. If someone had the resources and the intention, perhaps a government, then virtually nothing could be done about it.

3: If 50%+ of a miner's income is coming from tx fees, then only the situation of the 51% attacker would really be relevant, since miners that did include txs could easily out-earn and outhash a miner smaller than that. If, on the other hand, the exchange value of BTC increases at a rate faster than the subsidy decreases, then charging more than a token fee is stealing. Unless, of course, the costs of running a mining rig increase faster than the value of the subsidy, which seems fairly unlikely since the cost per unit of power or storage decreases exponentially.

I would assume that your average merchant, from time to time, might want a transaction confidence rate of at least 75%, and 80-90% or higher would be ideal for sensitive, high value transactions. If a large miner is charging more than 100 times the amount than for a 50% confidence tx, then 100% confidence, or only 2:1 the value, is costing 100 times as much, or more. The cost to value for using bitcoin as a currency then becomes something like 1/50 as much or worse in that case. In the competitive case, this wouldn't matter much, but in the current case there really is no competition since any reasonable tx fee will be an insignificant source of income to any miner for the foreseeable future. A miner who includes tx doesn't, and can't, make enough money to employ more hashing power than one who charges enough that no tx are included, so a lower cost service has no advantages at all.

The only advantage of lower fees at the moment is that, if all miners charge a 1 bitcent tx fee, bitcoin dies tomorrow and all their earnings from mining become worthless. However, for someone who only trades their BTC for fiat, this risk is almost totally externalized and can't be corrected by the market.

I think the disadvantage of floating price control is far outweighed by the advantages of keeping high-confidence verification costs at least half-reasonable, and by the resistance to a 51% who tried to DoS by excluding every tx by using an unpayable fee. Also, if you publish the fee along with the block, then it becomes very easy to verify that the expected tx for that fee have actually been included, so working the system a la mystery becomes virtually impossible. If the network becomes flooded with idiots like D&T who demand both hefty taxes AND a welfare check, then the network will fail even if their claim is "legit", so the disadvantage is really moot.

The TL;DR of it is: No changes to protocol necessary thanks. (Too many updates as it is - "patches to my money" makes me nervous!)


Too much money is being spent on mining compared to the transaction volume I think.

I get it increases security, but surely a "few supercomputers" are enough capacity if not too much for now.

When the reward is lowered alot of these rigs will shut off.

Even botnets have costs:
1. Risk of being arrested.
2. Programming/hacking time.
3. Updating after anti-virus updates (having to do major virus software updates regularly isn't exactly free)

Fees will rise to some set level; because the cost of processing each transaction is always similar.

If it approaches the actual hardware cost then it will be quite low since the BTC network could theoretically run on a few racks and some well connected nodes.

Lets say its 0.01 btc or in that area, you can still do nano-payments:
1. Transfer 1-5$ to your used site.
2. Use this to pay smaller fees they throw at you.
3. BTC network cost is only and needs only to be paid once.


So there is a small chance the network will trend to become very small and insecure though widely used.
I can only guess, but there are at least a few facts that may prevent this from happening:
 
1. Hardware sitting unused and depreciating in value can earn back some of the loss by mining for transactions.
A few idle corporate super computers doing this and the network is pretty safe.

2. BTC banks may ramp up idle hashing power immediately after big important transactions, then throttle down once its done. This would secure the network and their transactions against double spends.

If they detect an attack later all the banks and others will online their power to defend their interests (a double spend attack messes with every single transaction even if only one party is getting directly screwed).

This would be as simple as not working on the longest chain, but the longest chain YOUR transaction is included in.

The 51% attack would only look successful for a few hours.

3. BTC businesses might run a few nodes at a "loss" to further their user base.

4. Humans are not machines; its possible that people will NOT trend towards paying the minimum hardware cost, but will regard 0.01BTC (at current value) as arbitrary and just pay it.
Doing so you get an oversized and secure network like the one today.

So it is not a problem to you then if people like me, who actually want Bitcoin to succeed, stop relaying blocks with no free transactions?  And when we become a majority, you are basically shut out of the network until you start behaving.  Interesting idea.  Because I don't want to support greedy miners who stop perfectly legitimate transactions just because they can run a denial of service attack against transaction processing and demand a ransom from users.  Initially I thought you opposed Revalin's idea, but I can see you actually support it wholeheartedly.  You just don't want to take part, but that's up to you.

Haha, now you take the value in USD into account.  Earlier your arguments have been about percentages of block reward.  Now you are basically admitting intellectual bankruptcy.  Well done.
If you want to adjust it for the price of Bitcoin, be prepared to increase it every week.  Transaction processing does not cost you a single cent.  You will notice the downward spiral in both price and the number of transactions as people stop trusting Bitcoin, because Bitcoin transactions are slow or expensive.  Users dependence on miners seems to be clear to you, but you don't grasp the fact that miners are dependant on users as well.  Short term profit sounds good to you, but you don't understand how bad your idea of unpredictable service is in the long term.
If you think this is a problem, please find a better algorithm.  Adjust it dynamically according to total network hashrate perhaps?

By artificially decreasing service to everyone, making transactions unpredictable and Bitcoin less useful and less valuable.  It's just a malicious activity, and I will support every means to stop it.  Transaction processing is essentially free, and the price should reflect that.  If block size becomes a bottleneck, or when the block reward approaches 0, fees may be necessary.  In a hundred years or so.

One thing that might change over these tx fee discussions is the network topology:

One large network has pros and cons, and suppose the fee on the 'large'  global BTC network is comparably large, and the tx processing becoming slow (as mining rewards will continue to be much more profitable than transactions for a while). A large global enterprise with a sufficient number of customers (like Facebook, Google, Amazon) decided to use BTC at some point, they could make transactions between users fast and without fees. Only going outside of their network to the open P2P BTC network would require high fees to get processed in time. That's comparable to international wire transfers vs. PayPal's internal payments. I see no reason why this shouldn't happen with bitcoin. I see the ad for BitPal: "Sign up to be a member of our xyz BTC community, that has thousands of members and transfer bitcoins quickly without fee". While the global BTC backbone mines primarily for coins and only concerns itself with large, expensive transactions. It really feels so contrary to the open concept of bitcoin, but I see that as a possible option at some point.

I think you are right, except that it isn't contrary to the spirit of freedom. It costs what it costs to use bitcoin, very low now, likely more later. But anyone can get access by paying only the legitimate costs, no licenses, permissions etc.

In addition to BitPalish companies any WOT of any kind that springs up will be able to save fees too. Be it as small as my family or a large Bitcoin OTC style thing or Hawala.

The "free market" really is about freedom. The market may indicate a price where supply meets demand, but each individual is free to trade at whatever price they like.

People often trade away from the "market price". In my city, there is a product which can be bought at the market stall on Saturdays for £1, but which costs £1.99 at the department store and £2.99 at a specialty store. The free market doesn't "dictate" that everyone sells for the "market price".

This says all that I need to know about your worldview.  The free market isn't theoretical at all, it's the default emergent order in the absence of coercion.  And capitalism isn't remotely the same thing.

The "absence of coercion", see that is what I really doubt can or will be possible anytime soon or maybe anytime at all, which makes its true implementation a theoretical exercise for me. You don't really know much about my worldview, but you are right, I don't buy into that "absence of coercion" scenario. That is an ideal.

Meanwhile ... some pools send an LP with no txn to speed up the fact that they need to send an LP to every miner in the pool as quickly as possible when a block is found.

e.g. it's not hard to find a quick DeepBit block with no txns.

So yeah your not gonna get too far with trying to enforce rules like this silly one.

The only purpose of miners in BTC is to include and record TX.

1: Any miner not including tx for technical reasons is a communist parasite collecting a welfare check and contributing nothing. These people should be excluded 100%. One day there will be 1000tx+ per block, and miners trying to take shortcuts to just win their block subsidy will fuck the network if they dump 1000tx per block on everyone else. This is especially true once people are actually paying a fee to get their tx into a block faster, since they will still end up paying a fee but will get nothing in return.

2: The blockchain is a contentious shared resource, the only purpose of which is to record and verify tx. Any miner who excludes all tx for fee reasons should be considered a monopolist and be excluded.

3: Because miners are a communist entitlement-sucking union, they should be forced to meet a quota in order to regulate how much control they have over prices. If miners are forced to include 10% of tx, then getting into that 10% spot means bidding on the price of those slots. This doesn't stop miners from choosing the 10% they decide to include, or from excluding everyone who doesn't pay their price assuming enough people are willing to do so.

Case in point: D&T ran out as fast as possible and commissioned fee software, calculating how much he has to charge people in order to regain 50BTC per block after the subsidy is cut down, as though miners "deserve" to be paid 50BTC per block in fees, regardless of the market value of 1BTC, regardless of the value of the service they provide, and regardless of what users are willing to pay (100% exclusion for those paying less than his fee). Because the blockchain is a time-limited, shared resource, if all miners adopt similar policies, either because of what they think they're entitled to or because they want to save bandwidth so they can find a block a few seconds before the next pool, then the whole blockchain might soon be empty of tx.

The same type of problem occurred in the whaling industry, and because individual whalers had no personal incentive to limit their hunting, collectively whalers nearly hunted all species to extinction. By the same token, a subsidy with no regulation is likely to lead to all miners destroying the currency, because it is in their stupid, short term self-interest to cut corners and extort high fees regardless of whether or not BTC collapses as a currency.

Haplo I demand you back up your LIES with a cite.

Please provide a specific cite where I stated:
a) fees should always remain the same regardless of the market value of BTC
b) miners "deserve" 50 BTC per block in fees
c) how much fees are necessary to regain 50 BTC per block.

You will learn reputation is earned here and with only 49 posts you are starting out as a blatant liar.  I expect that when you can't find evidence to back up your lies you will correct your blatantly false statements about me.  We can disagree on the facts but I won't stand by when some fraking noob start lying to my face.    Regardless of your actions the quote will remain so others can see how little your word is worth.


Quoted for prosperity how little value Haplo's word has:

This is wrong from the start.

Okay, you are entitled to your own opinion, even though reality doesn't care what my or your opinion may be.

There is at least one more obvious and just as important purpose to mining ... securing the block-chain by generating new blocks.
You need both to happen since mining blocks with a million txn's will not help anything if you are not actually generating required difficulty blocks.

OK I guess I need to explain my previous post?

When a block is generated, every standard pool on the internet needs to send a message to every pool miner to start work on the new block and include the details of that work
This needs to get to the miners as quickly as possible to stop them wasting their precious mining resources on work that doesn't secure the block-chain, but rather switch to work that does help secure the block-chain.
Thus if the pool simply creates the smallest possible block and thus can process it as quickly as possible so it can send out these change work details as quickly as possible, then it makes sense to do that (this processing needs to be done repeatedly, for each miner)
As soon as that work is processed and if it is found to not provide a block, each miner will next work on a block header for the same block but that contains the transactions that the pool decided need to be processed.

So there is even a rather simple technical reason for there being blocks that only contain the coinbase txn and no other txn.

So, yeah, being god and saying everyone must do this coz you think it is right, doesn't really work that well in the real world where "science" decides :)

What processing do you think the pool has to do per-miner that has anything to do with the size of the block?

It must process the full block to produce the 80 byte header and the Midstate for each miner.
This processing is directly related to the number of txn's in the block - i.e. generation of the merkle root for the blocks merkle tree.

The tree root (part of the 80 bytes) is simply the transaction number of the coinbase for a block with no extra txns - thus requires no merkle tree processing

If you don't understand the merkle tree processing read here for starters:
https://en.bitcoin.it/wiki/Dump_format

No, it doesn't.

Generating the merkle tress, less the coinbase transaction, need only be done once per block. Adding the coinbase transaction at the tip has constant expense regardless of what else is in the tree.

The processing would be done only once anyway, unless you wanted to include newer transactions.

Once your merkle tree of transactions is build, adding a coinbase to it has constant cost regardless of the size of the tree.

I think the real reason miners are building blocks with no transactions is because they don't want to have to validate transactions and a block with an invalid transaction would be rejected. I don't think it has anything to do with generating work units.

Doesn't have to.  Usually, the pool master does the merkle tree work and just passes the merkle root & the rest of the header to be hashed to the miners, so the miners don't see more with or without transactions.

We're talking about the work the pool master has to do per miner. My point is that it only has to build the merkle tree once. It can then inject a different coinbase transaction in without having to recompute the rest of the tree. (The coinbase is at the tip of the tree.)

Since all users are free to edit and/or delete their own posts, there is no way for me to prove absolutely what you said or did not say. That said, I can no longer find the post I was referring to, but that could be simply because I haven't looked far enough. Either way I can't find it on search and I don't feel like looking any further just to prove that you're a communist. Also, I could give a shit less what my post count is or what other people think.

However, there is something I'm curious about.


Is there a purpose for doing that besides intentionally destroying coins, or perhaps making a finney attack?


Securing tx and including tx are inseparable. Security with no tx and tx with no security are equally useless, since tx are ultimately what is being secured. We would never consider paying miners just to sit around spamming an empty blockchain, since that by itself serves no purpose.

That said, I can see that there may be other technical problems with the minimum inclusion proposal, but since everyone seems to have a different idea of how tx inclusion and mining pools work (which may be true, since there are different software versions), I can't even intelligibly decide whether I'm full of shit or not, however..

(1) If pool owners are purposefully excluding tx because it is inefficient for them in some way, that is entirely their fault, or possibly a shortcoming in the BTC protocol itself; but if that is the case, then shouldn't P2mining be impractical? If distributed mining isn't practical (especially now when there are only ~80tx per block), then how practical is BTC going to be when there are 1000tx+ per block? Otherwise what do we need major pools and pool operators for to begin with?

(2) If a pool owner is excluding tx because they can't keep the blockchain on their machines, we should force blockchain validation a la gmaxwell, assuming that his proposal (or something similar) would in fact solve the problem. I have my doubts.

(3) The monopolist/commons problem may or may not become relevant. A minimum inclusion requirement wouldn't necessarily solve that problem, either. The main problem is that, unlike a free market, only one "seller" gets to sell his product every ~10 minutes or so, no matter what the demand is, so there is an inherent limitation in price matching between buyers and sellers, which biases towards unilateral miner monopoly a bit more than in a "normal" market. If history serves as any reasonable guide, the next step is that monopolists arise (probably the owner of DeepBit, for that matter), then the next thing you know they own the currency and you've got full-on fascism all over again. An external fascist nation with enough resources (ie the USA) could easily produce the same result if they wanted to purposefully destroy the currency.

Some form of proof of stake might mitigate that possibility, but I don't know of any practical way to make that work. BTC combines both technological concerns along with economic concerns, which makes it far more complicated than any other currency, period. Considering it's less than 4 years old, it's not even clear that it will actually work at all at this point, but the sheer number of ways in which it could be attacked from within or without doesn't look very promising. Without tangible benefits over fiat or gold, there's no reason to use it at all. Anyone using BTC right now is really speculating on an experiment, which is fine for personal use, but unacceptable from a business standpoint.


Makes sense. Then the question is "how consistently could a tx list be kept across different nodes on the network?"

D&T seems to think this would make tx inclusion less efficient (which it might depending on the mechanics of the inclusion quota) however I don't see how that could be the general case. As long as miners are using similar inclusion criteria, their lists should be very similar overall, so it should be possible to reach a reliable agreement over the network.

The way I see it, if a miner is happily accepting charity from their 25-50BTC block reward, paid by all BTC stakeholders and potential stakeholders, then they should happily accept a minimum quota of charity returned to those stakeholders to ensure the reliability and longevity of the network. It doesn't even require that they necessarily accept free tx, just as long as they include enough tx to prove that they're contributing something. The only outlier case is where the next block is found before any new tx are broadcast, which is possible, but there are ways of allowing such blocks to be accepted anyway.

OK I'm not correct, but neither are you :)

It has to process the complete left side of the tree and the depth of the tree is based on the number of Txn's being included.
It is not a constant.

Edit: also before the tree it has to generate a coinbase txn for each miner and after the tree the midstate for each miner

So you lied.  Good enough.


It is right there in the quote.  The concept is called "coin melting".  It would render asinine ideas like "tainted coin lists" and "Bitcoin Police" beyond infeasible.  Currency must be fungible.  Having blacklists reduces the fungibility of currency and I see it as a threat to Bitcoin.

That's mostly true, but so what?  I don't understand the point you are trying to make?  The coinbase transaction is the one that all miners must make, and correctly, for their block to be valid at all, whether that is done by a pool master, a lone miner or a botnet.

Nope, still not following. I'm afraid I don't know what you mean by "memory pool". What does it mean for a tx to be in the memory pool? What does it mean for a coin to be included in the memory pool without being broadcast?

AFAIK a tx not broadcast will not be included in a block, and therefore is effectively not a tx. It sounds to me like you want miners to be able to include arbitrary tx which are neither received nor coinbase into their blocks, the only purpose of which that I'm aware of would be for executing a finney attack with the cooperation of a mining pool.

This is correct.  If you create a transaction but don't broadcast it, it doesn't exist.  The same is true for a block.

He wants to mine it himself, not broadcast it to other miners.

I'm guessing it's a transaction with tainted coins as input, and much less (or zero, if that's allowed) output value.  The difference between input value and output value goes to whoever mines the transaction, which D&T wants to be himself.  That way the tainted coins lose their taint (unless the taint calculation takes fees into account, of course).

Exact!

You can put all the tainted coins as fee or only a percentage in a transaction to another address you own.

This way you melt the tainted coins with the real transactions fee mined in the block and the block finding reward.

I can see how that might work to undermine blocklists, but selling the private keys works just as well.  Anyway, he might have to wait a long time to mine his one transaction.

There is no need to sell the private key.

transaction fees aren't subject to any taint list so ....

tainted coins in -----> coin melter* ----> untained coins out  :)

* coin melter simply makes a tx using tainted coins as the input a new address as the output and send a % to coinbase as a tx fee.

Using enough blocks one could convert all tainted coins into coinbase rewards.  At which point they can be used just like any other coins.

That would be a great business(and probably very profitable) for the one who is brave enough to do it ;)

If 'tainted coin tracing' is ever implemented, it would be pretty easy to extend the idea to 'tainted mined coins'.

Mine 75 BTC (50 plus 25 in 'melted, tainted coins') and it'd be considered 33% tainted.

Possibly but I would hope the taint idiots would see the catastrophic damage that would cause.

Thieves don't need to get 100% of the take to be profitable (things like a fence are just a cost of doing "business").

Say i steal 100K BTC from the bank of Bitcoin.
I have the coins melted down for a 10% fee by someone with a lot of hashing power and willingness to maximize that return (105% PPS pools anyone).  To camouflage that I create a large number of tx involving both tainted and untained coins with hefty fees which I broadcast to the network.    They get picked up by dozens of pools and the fees dispersed to thousands of miners.  Likely the taint is spread unevenly between miners adding to the confusion and uncertainty. Some of those miners use those tainted inputs in downstream txs.  Sure I didn't clear the 100K but no business is 100% profit (not even theft).  

What is MtGox going to do then?   Freeze accounts of ten thousands miners simultaneously because they have tainted coins from mining?

In most of the discussions of Bitcoin Police & taint databases the consensus seemed to be that taint couldn't follow tx fees due to the massive overhead and false positives that would create.  My interest in coin melting is to simply to show that coin tainting is beyond stupid,  Mt.Gox actions are beyond stupid.

Taint lists and account freezing undermines the fundamental concept that currency is fungible.Taint lists have a real risk of killing Bitcoin.  More than protocol flaws, more than SHA compromise, more that internal disruptions, more than govt prohibition, more than attacks by conventional financial interests.  Currency must be fungible.  Period.  If it isn't fungible it isn't a currency.

I have two $1 bills in my wallet.  Each has a purchasing power of exactly $1.00000000000000.  It doesn't matter which one I spend I get same amount of goods in return. That fungibility is a major element of the intrinsic value of the dollar.   If Bitcoin can't remain fungible IMHO it will die and will be replaced by an alternative which preserves that essential quality of currencies.

Another big can of worms.

My point was ANY scheme you come up with will eventually come down to "Does this transaction input have a high enough percentage of 'badness' for me to say no, I won't take it."

Whatever percentage you choose, the bad guys will very likely figure out ways to make their transactions just barely pass your purity test.

Which is why I mostly think starting down that road is probably a bad idea.

On the other hand... "security theatre" can be good public relations.  Make the bad guys jump through two hoops and then feel good about how tough you are on crime....

I misunderstood and then went on a long winded rant.  Need some coffee.   I agree the point of coin melting would be to make that determination too complex and subjective to be of any effective value. I have no interest in running a coin melting service but I think having it out there is useful for changing public opinion.  If it can be done it eventually will be done so might as well show people it can be done.

Mt.Gox will only change their policies when it costs them money.  Tainting thousands of accounts while at the same time ensuring the thief's accounts look no different than the tainted ones will cost Mt.Gox money.  It will also ensure coins will never be returned to the person who claimed to have lost them.  Coins will remain fungible and irreversible.


Well it is good for controlling the masses (Patriot Act, no fly list w/ 1.8 million "terrorists" on it, TSA, etc).  Some people just want power and control even in a decentralized network.  Protocol or no protocol they want to be the one controlling it.  Freedom is scary and they want the comfort of controlling the uncontrollable.

Is there a transaction-type where I can force the receiver to make a decision on taking or not taking the coins? That way I can imagine this one working.

Sorry if this is a re-post:

http://www.rdmag.com/News/Feeds/2012/03/general-sciences-bitcoin-currency-system-offers-negative-incentive/

Apparently this was presented at a convention in Spain back in July.
Is this the same concept as only mining blocks with no transactions?

No.  The article you link is just pointing out that nodes have no incentive to forward transactions.

The whole problem with any such approach is that "bad" is not a mathematical calculation.

Bad is a 100% purely subjective opinion - there is no calculation that says something IS bad.
It is simply just an opinion and no more than that.

Worse, what is happening it is actually based on the ideals of religion.
The idea that there is a god that will set everything right in the end.
And worse, people are trying to call MtGox their god in this situation.

The absolutely simplest point to make is - why should MtGox decide what bitcoins are good and what bitcoins are bad?

If someone steals 10 BTC from me, how on earth am I going to be able to get MtGox to ban the perpetrator?
Why would MtGox trust me saying that those coins are "bad"
Who in MtGox is going to be given this god power to decide which transactions in the block-chain are "bad"?

It's actually even worse again - it boils down to a person or a certain small group of people having the power to decide that when someone claims that their bitcoins were stolen, that claim is decided to be true (or false) - and who will this person or small group believe?
Oh hang on - is it only for big transactions? - people with 'more money' that this will work? ....

It will become a situation where certain people will have certain extra power over bitcoins that others do not.
Who in their right mind would be stupid enough to be happy with that?

I'm feeling like I should start an anti-MtGox campaign here on the forum and tell people to NOT use MtGox and try to shut them down.

Of course MtGox will be happy if you give them this power ... that they also seem to want ...

Well, I think this has gotten a bit off topic, but I agree. MtGox got what they deserved because they were fucking stupid and let themselves get social engineered and had all their passwords stolen. Not only that, but their recovery of people's legitimate accounts was piss poor, and AFAIK is still leaving some people stranded.

There are plenty of ways to avoid that sort of thing, and plenty more being developed (like BTC address signatures). If you're going to run a bank, exchange, or other business which deals with other people's money, you need to do better than that. If anything, MtGox should go out of business. Sure, the people who robbed them were douches, but it was ultimately MtGox's responsibility to safeguard their clients information, and they failed, and then tried to blame it on someone else.

I don't know that "coin melting" will actually solve that problem to any extent, or make it any harder for them to track "tainted" coins. Not like keeping some kind of coin blacklist is going to do anything anyway. MtGox is never, ever getting those coins back, so trying to put restrictions on them is impeding the currency in general just to make MtGox look less irresponsible than they really are. If you want to police other people's lives, you're in the wrong place; BTC is all about personal responsibility.

Your points on fungibility are correct. There are plenty of laundry services to do this. Starting down some road to build laundry ("coin melting") into the protocol sounds like a bad idea, indeed. Investigating bitcoin crime seems like it's super fun for law enforcement if they're trained on how to do it (and taught correctly what "taint" really does not mean). Supporting their training, while the laundry services churn away certainly sounds like two good hoops.

Bitcoin lends itself to a free market - if large bitcoin handlers persist in seizures based on taint, more people with larger holdings will just go to the laundry as a matter of course upon receiving bitcoin.

It's costing MtGox money already. (something is bound to w/ 80% marketshare) - Since they're the most likely first contact for law enforcement they need to send the LEOs wherever they need to go to get trained. Their bottom line will teach them this, but they probably can't teach all the LE community by themselves.

I know I'm new here, but it seems to me that the "trackability" of coins is a prime feature of bitcoin - gives the LEOs something to do, and a way to use it to look at / for crimes. Because there is no such "thing" as a bitcoin, "taint" is just probability, and that's what LEOs are supposed to work with: probable cause. It seems that security theater is exactly what's built into the design. Why change that?

Cool, then more people will use our trading platform :)
Decentralization FTW.

Futures & Options FTW!

Why don't we use our resources to educate computer users and help prevent botnets from becoming as large a problem as they are.

Oh, I'm sorry, forgive me. I forgot that this was about money and not people.

Isn't it really about how people relate to each other and money?

Yes it is about money.
That's the reason bitcoin exists ... even the name tells you that.

Preventing botnets is about money.
The botnets exist because they can make money, people want to stop them coz they are using other peoples computers to make money.
http://arstechnica.com/business/news/2012/03/p2p-botnets-the-bigger-they-come-the-faster-they-fall.ars

It helps to be clear, it seems you are talking about -involuntary- botnets.  Bitcoin is a giant botnet, a mining rig is also a smaller botnet.  Just to be clear, there is absolutely no a priori reason that the bitcoin network would care who pays for the power. 

Yes the term botnets has more than one meaning in English.
Great language isn't it :)

In reference to bitcoin the term has a specific meaning:
It's a collection of computers infected with a torjan/virus/such that are running software that the computer owners are either unaware of or unable to stop from running.

Of course, "bitcoins" do not care if you hacked into someone's computer and stole them, stole the network to generate them or used your own resources to generate them.

Just the same as your local supermarket doesn't know if the $10 you gave them to buy a packet of cigarettes was obtained by murdering someone down the road and stealing it from the wallet in their lifeless body ... and then when the supermarket gives it to the next person who receives it as change, the shop that they take the $10 change to, to buy their grandmother flowers doesn't know that it is "tainted" money and will happily let you use it to buy the flowers ... then the child of the murdered person may get the $10 back as change when they buy flowers for the funeral of their parent ...

Lol I think this is more about counterfeit than it is about murder x.x

It is not possible to counterfeit Bitcoins.

yet ;) no system is 100% error free.

Counterfeit collars don't exist either, if they are counterfeit they aren't dollars :p

You could trick someone into accepting something that isn't actually a bitcoin, that's really what counterfeiting is. Don't let your guard down.

+1
While I find that's will be very hard to do online it can become somewhat easy with physical coins.