Potential disaster scenario

[gebler]

The difficulty for generating bitcoins is periodically adjusted using a method
that has worked well this far.  However, I am afraid there are plausible
scenarios where the current method would misbehave quite spectacularly.

One scenario goes like this:

1) As bitcoins become more known, competition among minters continues to
    increase, with corresponding increases in difficulty.  The increased
    difficulty will eventually make bitcoin minting clearly unprofitable for
    those who do not have access to good energy prices and cheap access to an
    energy-efficient HW/SW combination.

2) Some bitcoin users may continue to mint bitcoins even though it is not
    profitable for them.  This could be due to ideology, the fun factor, or
    just ignorance.  But it is quite plausible that the vast majority of
    bitcoins will be minted by those who profit from it.  Let's say that 99% of
    all bitcoins are eventually minted by for-profit-minters.

3) The competition among for-profit-minters will drive profit margins down, to
    the point where it is profitable to continue minting, but barely so.  Let's
    say that the typical profit margin during one difficulty adjustment period
    (2016 blocks) is 10%.

4) Since bitcoin minting is a decentralized uncoordinated process, we can
    expect random fluctuations in bitcoin minting activity.  This does not
    affect the difficulty during a specific 2016-block period, so the minting
    activity can e.g. increase by 20% during a period without making minting
    unprofitable within that period.

Given the above assumptions, we now have a disaster at hand at the next
difficulty adjustment.  As bitcoin production was 20% more than target,
difficulty is adjusted upwards by 20%.  But the profit margin was only 10%, so
for-profit-minters would now lose money if they continued minting.  They will
therefore stop minting, and as they make up 99% of the minting capacity,
generating the next 2016 blocks will take 100 times longer than normal.
Everything that depends on block generation will slow to a crawl, and this
slowness will persist for a very long time, since the next 2016 blocks will
take 100 times longer to generate (almost 4 years rather than two weeks).

Now, if this was to happen, I guess a new client could be released that resets
the difficulty to some sensible number and started using a better algorithm
for difficulty adjustment.  But it would be much better to do it proactively
before it becomes a problem (perhaps with a predetermined "flag day"
activating the new algorithm at a certain time in the future, giving the new
client a chance to propagate).

A simple(?) modification of the algorithm would be to apply the adjustment
after a certain amount of time rather than at a certain block number.  The
switch could still be synced to take effect for the next block, so time
synchronization between clients would not need to be super exact to have the
vast majority of them agree on when the new difficulty is to be applied.

Also, the difficulty adjustment should probably take into account the
adjustments of the number of bitcoins minted per event (now 50, halved every 4
years).  Halving the number of bitcoins generated each time is equivalent to
doubling the difficulty as far as profitability is concerned, and such a
drastical drop in profitability is unnecessary if it can be avoided easily.
I'm not sure if the current adjustment algorithm already takes that into
account somehow, but I couldn't see any obvious adjustment for it in the
source code.

[NewLibertyStandard]

I think there is a limit to the amount that the difficulty can increase at each step.

The more the bitcoin network grows, the less likely it is to have large spikes in difficulty. The likelihood of a jump of that magnitude from a single entity is very unlikely. If that kind of jump does occur, it will more likely be from a large interested demographic discovering Bitcoin all at once, such as if it was featured in a major magazine. Bitcoin will cope with such increases and subsequent decreases just fine.

In the very unlikely event of blocks taking hours or days to be completed, the transaction fee feature would quickly be added back to the program and people would start including a ฿0.01 transaction fee with each transaction they sent. The mega-minter would have a modified client which would monitor how many bitcoins in transaction fees were available and once it was enough to be profitable, they would turn on their mega-hash-cruncher and you'd get your block within ten minutes on average.

It doesn't much matter whether minting is profitable or not. In not too many years, total transaction fees per block will be higher than new bitcoins per block.

[gebler]

I think there is a limit to the amount that the difficulty can increase at each step.

The more the bitcoin network grows, the less likely it is to have large spikes in difficulty. The likelihood of a jump of that magnitude from a single entity is very unlikely. If that kind of jump does occur, it will more likely be from a large interested demographic discovering Bitcoin all at once, such as if it was featured in a major magazine. Bitcoin will cope with such increases and subsequent decreases just fine.

There is a limit to the difficulty adjustment, but for upward adjustments it is 300% (factor of 4 between new and old) rather than the 20% in my scenario, so it wouldn't help. 

I would argue that a 20% difficulty adjustment is in no sense extreme - the last five adjustments were 44%, 35%, 300%, 93%, and 21%.  This hasn't been a problem up to now, but obviously not because a 20% adjustment would be too extreme, but more likely because it hasn't been enough to make minting unprofitable (or otherwise uninteresting).  My scenario does not assume that a single entity is responsible for a jump in difficulty - it is just as problematic if e.g. publicity makes minting 20% more popular at a time when the profit margin is less than 20%.  The extreme effects come from the proportion of minters who are unwilling to continue minting at a loss.  If this proportion is large while the profit margin is low, the system becomes very unstable.

Also, if I am right about the 4-year halving of bitcoins generated (i.e. that it doesn't affect the difficulty adjustment) such events alone would be equivalent to a 100% upward difficulty adjustment.  And it seems very reasonable to assume that most minters will have a lower profit margin than 100% in the future.

[NewLibertyStandard]

It's already not interesting to mint bitcoins. Also, for the average person, I think they're hardly profitable. Even if the average person can sell bitcoins at slightly above production costs, it's difficult to invest more than pocket change worth of electricity per day. Profit on pocket change in exchange for hearing your fan running full blast all day everyday is not very interesting to the average person. If there is a sudden spike in new users, they're not going to all show up at the beginning of a difficulty cycle, stay until the cycle until it's over and then all leave. Most will not last a week generating bitcoins on our already boring to generate network, so as all these flaky new users quit generating, we'll have half a week without them and half a week with them. Those that do stick around will help adjust difficulty back to what it should be.

And like I said about the worst case scenario, if we're going on a week without generating a block, then we start including some transaction fees and somebody from the community rents a cloud to get to the next adjustment.

In short, I don't think there's much anything to worry about. But you're welcome to go on worrying about it.

[lachesis]

The problem with your analysis is that you assume that all for-profit minters will have the same profit margin. They won't. Among other things, larger minters will have economies of scale in their favour, making them more profitable.

In addition, as Bitcoin grows, people will develop dedicated hardware that maximizes the khash/dollar spent. In addition, people will tune the software in more and more precise ways to squeeze slightly more khash/second out of the same hardware. The people who invest a large fixed cost to do that will receive a correspondingly lower variable cost per Bitcoin minted in return, so they'll be able to mint at price levels that would drive others out.

Finally, at the point that this becomes an issue, everyone will be including transaction fees with their transactions to incentivize minting.

[gebler]

It's already not interesting to mint bitcoins. Also, for the average person, I think they're hardly profitable.

Well, there might be an important psychological difference between being marginally profitable and by clearly losing money.  The range of efficiency in bitcoin minting is such that some people may be profitable at a difficulty level where most are clearly losing money.  If this happens, I suspect the share of nonprofit minting activity would dwindle quite quickly.  We are then in the unstable situation I warned about.

In short, I don't think there's much anything to worry about. But you're welcome to go on worrying about it. 

Of course I'm worrying - my entire bitcoin fortune (150 BC) is at stake here! 

[FreeMoney]

You ignore all kinds of things that help solve this.

Transaction fees will increase if blocks come more slowly because it will be more desirable to get your transactions in the next block.

If an entity calculates that (fixed costs + variable costs < revenue) adjusted for risk the it is pretty unlikely that they will hit the (variable costs > revenue) easily at all and that is the shutdown point.

These minters will usually have and use bitcoins themselves. This gives two extra incentives to operate at a "loss". They want their transactions to get processed, and they want to maintain bitcoin as a working system.

[gebler]

The problem with your analysis is that you assume that all for-profit minters will have the same profit margin. They won't. Among other things, larger minters will have economies of scale in their favour, making them more profitable.

Actually, I was not assuming that.  In the scenario, the profit margin of 10% was the typical profit margin, by which I meant that most of the minting was done by players with a profit margin around 10%.  The scenario assumed the fluctuation in minting activity to be 20%, so it allows quite a lot of variance around the 10% mean without affecting the outcome as long as the bulk is within the 0%-20% interval (and the bulk cannot very well be above 20% - then the typical profit margin would not be 10%).

Regarding the effect of transaction fees, they may alleviate the problem somewhat, but at an unnecessary cost.  There will be some balance between monetary costs and convenience costs, but avoiding the costs altogether seems preferable to me.

[FreeMoney]

The problem with your analysis is that you assume that all for-profit minters will have the same profit margin. They won't. Among other things, larger minters will have economies of scale in their favour, making them more profitable.

Actually, I was not assuming that.  In the scenario, the profit margin of 10% was the typical profit margin, by which I meant that most of the minting was done by players with a profit margin around 10%.  The scenario assumed the fluctuation in minting activity to be 20%, so it allows quite a lot of variance around the 10% mean without affecting the outcome as long as the bulk is within the 0%-20% interval (and the bulk cannot very well be above 20% - then the typical profit margin would not be 10%).

Regarding the effect of transaction fees, they may alleviate the problem somewhat, but at an unnecessary cost.  There will be some balance between monetary costs and convenience costs, but avoiding the costs altogether seems preferable to me.

What if I had a factory and hired you as a consultant? Would you tell me avoiding costs altogether was the best thing? Many, many costs are worth it.

Having a secure, open, P2P, pseudonymous, limited quantity money, is worth a ton. People who agree will pay the .001BTC or whatever to make a transaction if that's about what it ends up costing. 

[gebler]

What if I had a factory and hired you as a consultant? Would you tell me avoiding costs altogether was the best thing? Many, many costs are worth it.

Sure, I just thought this was an unnecessary cost, best avoided by fixing the algorithm before it causes real problems and associated costs.  I understand that the 4x limit to the difficulty adjustment was added quite recently, so there is some precedent in fixing problematic aspects of the algorithm.

[FreeMoney]

What if I had a factory and hired you as a consultant? Would you tell me avoiding costs altogether was the best thing? Many, many costs are worth it.

Sure, I just thought this was an unnecessary cost, best avoided by fixing the algorithm before it causes real problems and associated costs.  I understand that the 4x limit to the difficulty adjustment was added quite recently, so there is some precedent in fixing problematic aspects of the algorithm.

Really? Source?

[knightmb]

What if I had a factory and hired you as a consultant? Would you tell me avoiding costs altogether was the best thing? Many, many costs are worth it.

Sure, I just thought this was an unnecessary cost, best avoided by fixing the algorithm before it causes real problems and associated costs.  I understand that the 4x limit to the difficulty adjustment was added quite recently, so there is some precedent in fixing problematic aspects of the algorithm.

I'll bite. 

How would you fix the algorithm?

[gebler]

I understand that the 4x limit to the difficulty adjustment was added quite recently, so there is some precedent in fixing problematic aspects of the algorithm.

Really? Source?

Sorry, I was wrong about that.  I though I had read something on the forum to that effect, but now I can't find it.  Looking at old sources, I see that the 4x cap was there as early as bitcoin-0.1.0.rar, so this is definitely not a new modification.

[FreeMoney]

I understand that the 4x limit to the difficulty adjustment was added quite recently, so there is some precedent in fixing problematic aspects of the algorithm.

Really? Source?

Sorry, I was wrong about that.  I though I had read something on the forum to that effect, but now I can't find it.  Looking at old sources, I see that the 4x cap was there as early as bitcoin-0.1.0.rar, so this is definitely not a new modification.

I thought so, but still, some changes can be made. What is the change that solves this problem?

[gebler]

What if I had a factory and hired you as a consultant? Would you tell me avoiding costs altogether was the best thing? Many, many costs are worth it.

Sure, I just thought this was an unnecessary cost, best avoided by fixing the algorithm before it causes real problems and associated costs.  I understand that the 4x limit to the difficulty adjustment was added quite recently, so there is some precedent in fixing problematic aspects of the algorithm.

I'll bite. 

How would you fix the algorithm?

It's in my original post:

A simple(?) modification of the algorithm would be to apply the adjustment after a certain amount of time rather than at a certain block number.  The switch could still be synced to take effect for the next block, so time synchronization between clients would not need to be super exact to have the vast majority of them agree on when the new difficulty is to be applied.

Also, the difficulty adjustment should probably take into account the adjustments of the number of bitcoins minted per event (now 50, halved every 4 years).  Halving the number of bitcoins generated each time is equivalent to doubling the difficulty as far as profitability is concerned, and such a drastical drop in profitability is unnecessary if it can be avoided easily. I'm not sure if the current adjustment algorithm already takes that into account somehow, but I couldn't see any obvious adjustment for it in the source code.

[knightmb]

It's in my original post:

A simple(?) modification of the algorithm would be to apply the adjustment after a certain amount of time rather than at a certain block number.  The switch could still be synced to take effect for the next block, so time synchronization between clients would not need to be super exact to have the vast majority of them agree on when the new difficulty is to be applied.

Also, the difficulty adjustment should probably take into account the adjustments of the number of bitcoins minted per event (now 50, halved every 4 years).  Halving the number of bitcoins generated each time is equivalent to doubling the difficulty as far as profitability is concerned, and such a drastical drop in profitability is unnecessary if it can be avoided easily. I'm not sure if the current adjustment algorithm already takes that into account somehow, but I couldn't see any obvious adjustment for it in the source code.

How will you account for latency and time hacking then?

[gebler]

It's in my original post:

A simple(?) modification of the algorithm would be to apply the adjustment after a certain amount of time rather than at a certain block number.  The switch could still be synced to take effect for the next block, so time synchronization between clients would not need to be super exact to have the vast majority of them agree on when the new difficulty is to be applied.

Also, the difficulty adjustment should probably take into account the adjustments of the number of bitcoins minted per event (now 50, halved every 4 years).  Halving the number of bitcoins generated each time is equivalent to doubling the difficulty as far as profitability is concerned, and such a drastical drop in profitability is unnecessary if it can be avoided easily. I'm not sure if the current adjustment algorithm already takes that into account somehow, but I couldn't see any obvious adjustment for it in the source code.

How will you account for latency and time hacking then?

Could you elaborate on what kind of scenarios you see where the proposed algorithm would be more vulnerable than the current one? Note that the current algorithm also uses wallclock time in the difficulty adjustment calculation and is synchronized using newly generated blocks.

[GoldRush]

You greatly under estimate the power of the profit motive and a market.  Bitcoins will continue to be generated as people find more clever ways to reduce hardware and power costs.  A lot of people are generating them at zero cost by using their employer's computer.  It's unfair to the employer, but it is the reason why bitcoins will continue to generate.  The current algorithm for difficulty is fine the way it is.

But let's assume you are correct and people stop generating bitcoins early.  So what?  There will always be a market price for the remaining bitcoins.

[NewLibertyStandard]

But let's assume you are correct and people stop generating bitcoins early.  So what?  There will always be a market price for the remaining bitcoins.

Transactions are verified by generating blocks, so although I don't think it will happen, blocks being too difficult to generate would cause the economy to grind to a halt.

[GoldRush]

Why would it "grind to a halt"?  Economies don't function just because money is being printed.  Are you using the same argument for the US dollar and assuming our "economy would grind to a halt" if the government didn't print more dollars?  Of course not.