[If tx limit is removed] Disturbingly low future difficulty equilibrium

[deadlizard]

Consider for now, we're in the future ...

Let me stop you there.
Before I read on I have to warn you that if this hypothetical future does not contain Sexy Robot Slaves I will be very disappointed ....

[Mike Hearn]

You're misusing "irrational". Accepting low-transaction fees is very rational for the one who accepts them, because the profits are all his own, while the free rider problem is everyone's.

Maybe I should use the word "reasonable" or "smart" rather than rational then, which is a loaded word in economics. Again just look around and you won't find very many companies selling their product using auctions. The exceptions are those where the thing they are selling is naturally finite and they can't really increase supply. AdWords is one such example because there are only so many ad slots that can fit on a screen, and demand often outstrips supply.

I'm not sure that the locality of the market makes much difference. Probably in future fees would be calculated by client software algorithmically. If a miner enters the market who behaves irrationally (ie, allows free riders) and has significant amounts of hash power then fees would drop dramatically and very quickly. They'd immediately undermine their own business as well as everyone elses and that would be very visible. For one guy using his GPU so what? But by the time fees drive mining it'll be in the hands of large, professional businesses. You wouldn't be able to get funding for hardware if your business model boiled down to undermining the market.

Someone who doesn't currently mine can rent some hardware, swipe all the low fees and exit.

They can do that if they spend enough in electricity to find some blocks, and then make enough profit on fees to pay for the cost of hardware rental. It only works for them if there are lots of low fee transactions sitting in the memory pool all the time waiting to be collected and they don't make it into blocks.

But the big miners using the "smart/long term thinking" model are still working on these transactions. It's just not for very long. And because they are running a sustainable business they are probably bigger and have more hardware than you. Once a high-fee transaction that depends on a bunch of low-fee or free transactions come along, those miners will be working on all of them anyway in order to collect the entire groups fees. So at any given time the dregs of the market aren't worth very much. The people creating them didn't care much about confirmation time. You could mine them yourself but you wouldn't get much money and the bigger miners wouldn't care as the extra profit from them is pretty small anyway. It's the high-fee transactions that drive the system, the low-fee transactions just get pulled along behind.

[Meni Rosenfeld]

It's the high-fee transactions that drive the system, the low-fee transactions just get pulled along behind.

Low-fee transactions will be much more numerous than high-fee, and I wouldn't be surprised if their total fee will be higher.

Suppose there's a pool that includes all transactions. It will be much more profitable per hash than pools that play hard-ball, and thus will enjoy lots of participants.

[da2ce7]

This has already been talked about here: https://www.bitcoin.org/smf/index.php?topic=1847.0

To summarise, the miners will from some sort of self-regulating cartel to make sure that the fees don't go to 0.  The cartel will reject block that have too many 'free/low-fee' transactions in it.

The will set their own rules like:  A transaction that has a fee less than 0.0001 BTC must wait at least 4 blocks before being accepted.

If the cartel becomes too profitable, some members will leave it and accept transactions under their own rules.

Overall, it should work out that the system naturally balances to the security required by the network.

[Vandroiy]

@da2ce7, [mike]:

Can you please debunk the exact failure modes I described for your suggested system, and provide the suggested equilibrium of the miner network size? In my eyes, the system configuration proposed has a macroscopic and a microscopic configuration, and both fail and both provide no reassuring equilibrium estimation. This has been handled before in a logic argument, with no part of the argument shown wrong so far! Why are we going over this again, with just claims again? (In fact, I just found a minor error in my end state of the macroscopic "cartel" configuration -- but I don't find any configuration that splits the block chain in standard operation good, and no other stable state has been provided so far. I'm still editing the post for clarity now. I'm still worried nobody noticed the error, it feels like nobody is checking my posts, no wonder I don't get arguments against them.)

My suggestion is that we move on from the "arbitrary fee height agreed on by miners" model. It doesn't get the discussion anywhere, unless someone can back it more strongly. Especially, I want to note that the model does not even provide any equilibrium difficulty at all, so even if we'd take it as true, there is no reason to believe this equilibrium would be at a good value for the system. This is building on sand, investors will not be satisfied with such an answer. We're not doing a trial-and-error game here, we want a solid currency for the future!

@da2ce7: You claim there is a link between required security and fee height. I do not see a substantial link. Can you explain where this claim comes from?

[Mike Hearn]

I guess I don't fully understand your position. The natural equilibrium of the model I proposed has security as strong as the participants require and no stronger. That model does not collapse down to nothing even if there are no block size limits at all. The fees are not arbitrary.

I'll try explaining again. Perhaps I have a problem in my thinking, but I don't understand your current criticisms so it's hard to say.

Let's start with the base case. If the recipient trusts the sender absolutely, then they don't need any confirmations to make a trade. Zero blocks is sufficient because the sender isn't going to try and reverse the transaction (by doing a Finney attack). They pay no fees because they don't need the services of miners ... they are perfectly happy with using the memory pool as just a public scoreboard. In fact they don't even need the p2p network at all, the sender can make the transaction and send it via email to the recipient, bypassing the public infrastructure entirely.

In many trades the recipient does not fully trust the sender. There is risk of fraud and the recipient can probably estimate that risk based on the reputation of the sender, size of the transaction and cost of reversal. However reversal risk is born by the recipient not the sender (who "knows" he is trustworthy). Instead the risk premium is built into the product price, just like today where merchants add on a risk overhead to deal with credit card fraud. The sender sends his BTC to the recipient with no fees (via the network). The recipient immediately broadcasts another transaction spending that money back to himself but this time including a fee.

The fee pays for a certain amount of computation. Consider 10 miners each of which own 10% of the networks hashing power. Each one can do 100 Gigahashes/sec and it costs them $1000 a day to mine continuously. A single miner can find a block in (on average) 100 minutes (10 mins per block * 10x harder than when everyone works on it).

If all miners were to mine on an empty network, there would be 1 Terahash/sec of work done at a cost of $10,000 per day globally. But nobody would do that so in fact there is 1TH/sec lying idle waiting for transactions (let's pretend we just entered this model so the difficulty is really high).

I wish to buy from you something very valuable which costs 20,000 BTC and let's assume 1:1 dollar:btc conversion. I sent the coins to you with no fees and as a result no miner starts work on it. You immediately spend them back to yourself, but how many fees to include? If you include none I can mine a new block with my CPU and reclaim the coins.

If you include a 1 BTC/$ fee, each miner can afford to spend 1/1000th of a day (86.4 seconds) working on it before their costs would be higher than any potential gain. Because there are 10 miners each one works on it for 86.4 seconds in an attempt to find a block with that transaction in - probably not enough to find any block unless you get lucky. With a 10 BTC fee, miners will work on it for around 14 minutes which will hopefully be enough to find a block but probably not two. Whichever miner finds a block with the transaction in will immediately start work on finding the next (empty) block and do however much computation is left based on how much of the fee was spent so far. Eg, if a miner finds a block after 3 minutes then the next block will be worked on for 11 minutes by that miner alone (the others shut down). So you'd get one block but probably not another one.

What if you want two blocks? A 100 BTC fee would buy you 1/10th of a day (2.4 hours) of computation. The first block would be knocked out within a few minutes as the entire network tries to claim it. The second block would take much longer but would still be found as the miner who claimed the fees lives up to their end of the bargain.

This buys you as much security as you paid for it .... 100 BTC fee means the sender has to spend at least 100 BTCs worth of computation to split the chain and reclaim his coins. Not a great deal given the size of the trade.

But in reality there will be many overlapping transactions. Let's imagine 30 mins after you send your tx somebody else does the same thing. Now all miners try to find a block including that second tx (including the one which was already mining to finish the work you paid for). A different miner claims the second tx and starts work. Now there are two miners working on securing your transaction .... but at some point your miner will shut down whilst the second keeps on going. The later transaction re-inforces the earlier but that's OK because it's not free riding - both senders are paying the same price for the same amount of work done.

In practice at high transaction rates no miner would ever fully shut down. They might do partial shutdowns in which some of their hash power lies idle some of the time, but the flow of transactions would always be enough to keep the chain moving .... though it might move a bit faster during the daytime than at night, it'd still move.

No miner would include free transactions at this point because to do so would immediately make lots of previously fee-paying transactions entirely free, yet that miner wouldn't get any greater share of the remaining fee paying ones. It'd kill their own bottom line.


[edited to clarify a few points]

[casascius]

I too have posted the same scenario as Vandroiy and can't say I have been satisfied by any attempt to answer i've seen.

Let's start with the base case. If the recipient trusts the sender absolutely, then they don't need any confirmations to make a trade. Zero blocks is sufficient because the sender isn't going to try and reverse the transaction (by doing a Finney attack). They pay no fees because they don't need the services of miners ... they are perfectly happy with using the memory pool as just a public scoreboard. In fact they don't even need the p2p network at all, the sender can make the transaction and send it via email to the recipient, bypassing the public infrastructure entirely.

The recipient is not going to be able to spend the coins until confirmed, because the transaction doesn't have an ID that anybody can reference.  As far as anybody knows, the coins still belong to the original holder.

No miner would include free transactions at this point because to do so would immediately make lots of previously fee-paying transactions entirely free, yet that miner wouldn't get any greater share of the remaining fee paying ones. It'd kill their own bottom line.

I have disputed this assertion in the past, see "tragedy of the commons".  It's like suggesting that no one would fish more than his fair share at a lake, or let his cattle graze more than his fair share in the field.  So long as it's beneficial in the short term for one person to take a little extra at the expense of the community - and especially if he can do so anonymously - people are going to do it.  The suggestion that "no miner would include free transactions" means Bitcoin is forever dependent upon miners anonymously sacrificing personal short-term gain for the good of the community.  Human nature shows that's pretty much impossible - were it not so, we could always just trust our Federal Reserve to act in our best interests too.

[da2ce7]

The current breed of stock generators, generate continuously, and only ignore a block if it is deemed illegal.

When transaction fees become the main incentive to generate a new breed of generators will be developed that will maximize the profit made.  One of the features of this bread of generator is that certain 'rules' will be voluntarily agreed upon, some will not accept blocks that have two many low fee transactions, and will wait for blocks that have only higher fee transactions.

The risk of a generator who accepts too many (new) low fee transactions is that the most of the network will ignore and orphan it's block.

This is an interesting, market based solution. Generators can stop competitors from undercutting them, by "voting" out blocks that don't adhere to their own customized rules. I'm not convinced that it is sustainable. It looks like a problem for game theory.

If generators get too greedy, accepting all transactions, They run the risk that most of their peers will reject their hard work generating a block. If they reject blocks that their peers have accepted, they'll waste time working on a smaller block chain.

If this sort of thing works, then miners should be doing it now! This will stop the spam problem: If miners start rejecting blocks that are too generous to spamers, we shouldn't need a block size limit. I suppose, though, that the block size limit IS an implementation of a voluntary rule. Though a rather primitive one, it's sufficient for now.

Perhaps miners will converge to using an optimal rule, similar to what I've described and I'm trying to preempt this development. This definitely looks like the problem of finding an "optimal strategy" in game theory.

The cartel will naturally charge as much as the market can withstand: If there is a serious thread from double spending then the market will accept higher fees.  If there isn't much of a threat to double spending, then the transaction cost will and should go down to virtually zero.

[db]

Vandroiy and casascius have it right. There are two tragedies of the commons here; one with the miners and one with the users.

Each miner gets the full benefit of including low-fee transactions but the loss from lower fees are shared by all miners.

Each user pays the fee on its own transaction but the resulting protection is shared by all users.

This will inevitably push fees towards zero.

I can think of a few things that might not make this as bad as it sounds.

Transaction fees are also the opposite of a tragedy of the commons (comedy of the commons?). Each user gets the full benefit of each fee. Only a few substantial fees are needed to protect everyone. Maybe the whole thing can be run on donatins?

If volumes are high enough, maybe even extremely tiny fees will add up to provide enough double-spending protection. But if not?

If not then there will be competition among double spending scammers. Only about six double spending scams can be done per hour. With lots of transactions the risk for each transaction will be very low and the double spenders will push up difficulty.

[Gavin Andresen]

You are forgetting that there are (at least) three players in the bitcoin economy:  miners, users... and merchants.

If transaction fees are driven to zero so miners start dropping out, then merchants have an incentive to step in and start mining themselves so their transactions get processed in a timely manner.  Otherwise they don't get paid.

[db]

You are forgetting that there are (at least) three players in the bitcoin economy:  miners, users... and merchants.

If transaction fees are driven to zero so miners start dropping out, then merchants have an incentive to step in and start mining themselves so their transactions get processed in a timely manner.  Otherwise they don't get paid.

That's also a tragedy of the commons. It won't work. Every individual merchant has an incentive to free ride on everyone else, not to contribute themselves. If one does take the cost and makes an effort to generate it won't benefit that merchant personally.

[stillfire]

That's also a tragedy of the commons. It won't work. Every individual merchant has an incentive to free ride on everyone else, not to contribute themselves. If one does take the cost and makes an effort to generate it won't benefit that merchant personally.

The merchants will therefore have an incentive to spend as little as possible on it. But if very few mine, mining will be easy and costs will go down. So even if there was a 'staring contest' between merchants where each waited for the next one to be the 'sucker' to turn on their mining rig first, the opportunity cost of waiting will quickly overtake the cost of just running the rig.

[BitterTea]

Merchants could also form a guild and purchase mining hardware that way. Membership in the guild (one of many, perhaps) gets your transactions confirmed.

[nextnonce]

It seems to me that there's some disagreement in this thread about one particular technicality of the bitcoin system.  It boils down to this question:

Does including a transaction in a block also require including all previous unprocessed transactions for those same bitcoins or doesn't it?  

[mike] seems to think that it does and bases his models on that.  The OP claims that instead, miners can just include the transaction w/ the fee and forget about the rest.  So who's right?

[Mike Hearn]

You can't include transactions in the block chain if their dependents aren't also in the chain, obviously, otherwise that block would be invalid.

I don't think it's worth arguing about this any more right now. This thread is going round in circles. If you're convinced miners will all undermine the entire market in return for tiny short term profits then you probably aren't going to change your mind based on analogies to real world markets where it doesn't happen.

[BitterTea]

It seems to me that there's some disagreement in this thread about one particular technicality of the bitcoin system.  It boils down to this question:

Does including a transaction in a block also require including all previous unprocessed transactions for those same bitcoins or doesn't it?  

[mike] seems to think that it does and bases his models on that.  The OP claims that instead, miners can just include the transaction w/ the fee and forget about the rest.  So who's right?

Let's say I send some newly generated BTC to a friend. I create transaction A with no fee, which signs ownership over from my address X (represents private key X) to their address Y. They create transaction B with a fee, which signs ownership from their address Y to their address Z.

If B makes it into a block, but A doesn't, the transaction will be invalid. This is because the previous output of the transaction is not in the block chain. The network (as it exists today), along with my client, would consider me the owner of the coins transferred in transactions A and B, as there exists no transaction in the block chain signing from X to Y or X to Z.

[MoonShadow]

It seems to me that there's some disagreement in this thread about one particular technicality of the bitcoin system.  It boils down to this question:

Does including a transaction in a block also require including all previous unprocessed transactions for those same bitcoins or doesn't it?  

[mike] seems to think that it does and bases his models on that.  The OP claims that instead, miners can just include the transaction w/ the fee and forget about the rest.  So who's right?

A transaction is only valid to the network if it's inputs are transactions already in the blockchain.  I think that it is actually impossible to referrence an input transaction without it's permanent block number.  (transaction hash + block number, I believe,  Someone correct me if I'm wrong)  So any other transactions sent after the first one, fee or no fee, would be rejected by the sender's peers as invalid and never even forwarded to a miner.

[Meni Rosenfeld]

The sender sends his BTC to the recipient with no fees (via the network). The recipient immediately broadcasts another transaction spending that money back to himself but this time including a fee.

What casascius said, AFAIK you can't reference a transaction if it's not in a block. The block chain establishes a timeline of transactions and each transaction builds on those in previous blocks.
That's not a problem though, it just means the sender will need to include the fees after all.

You immediately spend them back to yourself, but how many fees to include? If you include none I can mine a new block with my CPU and reclaim the coins.

You can't find a block on a CPU in reasonable time, fees have nothing to do with this.

If you include a 1 BTC/$ fee, each miner can afford to spend 1/1000th of a day (86.4 seconds) working on it before their costs would be higher than any potential gain. Because there are 10 miners each one works on it for 86.4 seconds in an attempt to find a block with that transaction in - probably not enough to find any block unless you get lucky.

The time to find a block follows the exponential distribution, which is memoryless. If it's profitable for me to try to solve a block now, it will still be exactly as profitable after 1000 seconds of trying. Nobody will suddenly stop mining after some arbitrary time, unless there's a mining police that monitors when someone tries to mine beyond the quota.

Also (a moot point in light of the above), if 10 miners each spend 1 BTC to try find this block, the total spent is 10 BTC which isn't covered by the fee.

What if you want two blocks? A 100 BTC fee would buy you 1/10th of a day (2.4 hours) of computation. The first block would be knocked out within a few minutes as the entire network tries to claim it. The second block would take much longer but would still be found as the miner who claimed the fees lives up to their end of the bargain.

And why would the miner want to work on a new block that has no fees? He's more likely to just say "I kept mining and didn't find any blocks. Honest."
And who does he report to anyway? The mining cartel police, or the sender who paid the fee? Does everyone who sends a transaction need to know the miner that collected his fee? Does the miner send difficulty-1 shares to the police to prove he's done his due work?

Also, someone who wants multiple blocks wants it in reasonable time, not the 100 minutes it takes to 1 miner out of 10 to find it, or the week it takes 1 miner out of 1000.

But in reality there will be many overlapping transactions. Let's imagine 30 mins after you send your tx somebody else does the same thing. Now all miners try to find a block including that second tx (including the one which was already mining to finish the work you paid for). A different miner claims the second tx and starts work. Now there are two miners working on securing your transaction .... but at some point your miner will shut down whilst the second keeps on going. The later transaction re-inforces the earlier but that's OK because it's not free riding - both senders are paying the same price for the same amount of work done.

I have no idea what your model is for multiple transactions per block. There are supposed to be thousands of them.

No miner would include free transactions at this point because to do so would immediately make lots of previously fee-paying transactions entirely free, yet that miner wouldn't get any greater share of the remaining fee paying ones. It'd kill their own bottom line.

Miners spend computing power to try solve a block, and choose which transactions to include in it. If they succeed, they collect all the fees for transactions they included. More included transactions = more fees = more revenue per hash = profit. The only way they would reject transactions out of fear of encouraging free-riders in the long term, is if they both expect their individual actions to significantly affect the market (which is suspect, given its size and complex dynamics) and intend to stay in the market long-term (which is even more suspect, given the low barrier of entry). See also Prisoner's dilemma.

And what if the mining cartel uses FUD, vendor lock-in, lobbying and bullying to prevent people from forming an efficient market? Well, I thought that was one of the things Bitcoin was supposed to be against.

[Meni Rosenfeld]

I don't think it's worth arguing about this any more right now. This thread is going round in circles. If you're convinced miners will all undermine the entire market in return for tiny short term profits then you probably aren't going to change your mind based on analogies to real world markets where it doesn't happen.

As was already explained, those analogies have few players and high barriers of entry so it's not even a remotely valid comparison. Also, your model is wrong for several reasons as I've explained above.

[MoonShadow]

Vandroiy and casascius have it right. There are two tragedies of the commons here; one with the miners and one with the users.

Each miner gets the full benefit of including low-fee transactions but the loss from lower fees are shared by all miners.

I don't see how all miners suffer from lower transaction fees, since it's presently provable that free transaction fees are possible with a working Bitcoin economy.

Each user pays the fee on its own transaction but the resulting protection is shared by all users.

This will inevitably push fees towards zero.

They are already zero, but you are overlooking time preferences.  Space in a block is effectively of zero value, since there is presently plenty that a zero fee transaction can get into the next block.  However, if Bitcoin ever reaches mainstream acceptance comparable to PayPal; the space inside a block (which is limited by convention) will start to command a premium.  As such, users who have a need/want to see their transaction included in the next block will be motivated to provide a fee towards that end.  And forcing all transactions that pay a fee less than them to wait until the next block or longer.  Such traders will be relatively rare, such as buying/selling a car from a dealer or stranger, or other high value transaction wherein confirmations are the only acceptable risk limiting mechanism.  Even relatively rare, such transactions are likely to justify enough of a transaction fee to motivate miners to incude as many of them as they can find.  Perhaps even motivating miners with marginal profits to jump into and out of mining based upon the collection of fees in the transactions in their queue.  This would create a short term "flexability" in the processing of transactions, as blocks would come faster than six an hour under high demand and slower than 6 per hour during off peak times; still averageing out to 6 per hour over the two week adjustment period.

I can think of a few things that might not make this as bad as it sounds.

Transaction fees are also the opposite of a tragedy of the commons (comedy of the commons?). Each user gets the full benefit of each fee. Only a few substantial fees are needed to protect everyone. Maybe the whole thing can be run on donatins?

If volumes are high enough, maybe even extremely tiny fees will add up to provide enough double-spending protection. But if not?

Then we are back to where we were about four months ago, when transaction fees were rare.

If not then there will be competition among double spending scammers. Only about six double spending scams can be done per hour. With lots of transactions the risk for each transaction will be very low and the double spenders will push up difficulty.

I'm not even sure how you get to this conclusion.