Difficulty adjustment needs modifying

[Transisto]

The fact that the difficulty retarget is slow and late (and the result is wrong since it represents the previous 2 weeks not the current environment) and directly promotes the recent BTC economic climate.

If BTC climate it decreasing, it's slow to decrease the difficulty (and can be very slow to decrease it) and thus promotes making the decreasing environment worse.

There is no such thing as "current environment" , be it the previous 3 day or previous hour.

It take roughly 2 week to purchase GPU and set them up for mining.

I don't think Day traders and weekly miners count for even 1% of the ecosystem.

weekly miner mean, oh this week the price is low I will stop mining, oh the difficulty we be low in 2 week, lets purchase tons of GPU.

[kano]

Unfortunately that ignores reality.

Just because the bitcoin re-target time line occurs approximately once every 2 weeks, doesn't mean that everyone who mines makes decisions the minute the re-target occurs.
It is a continuous time line with decisions made constantly all along it.

In fact the rather amusing side comment I could make is that yes there is one recent event that probably does ignore what I just said - the release of BF3 - maybe suddenly a lot of people decided to stop mining and play BF3? I don't know - but it would be interesting to see if there is a sudden dip in block generation rate around now (somewhere during the past 24 hours and the next 24 hours)

Anyway, for the past 2 re-targets (at least) the 3day average has always been worse than the 14day average
That isn't related to it having a higher SD, that's directly related to the environment.

Why? For the very obvious reason, there has been an almost continuous decline in miners over the past month.

What does that mean?

Well that's simple also, that the estimate at the end of the 14 days (over the past month) is always too high, it is pulled up by the higher results of the first 11 days versus the results of the last few days.

Thus there is a current environment and that current environment is decreasing at the moment, not going down in steps every 2 weeks.

The re-target maths ignores that possibility and in fact makes it worse.

[sadpandatech]

Unfortunately that ignores reality.

Just because the bitcoin re-target time line occurs approximately once every 2 weeks, doesn't mean that everyone who mines makes decisions the minute the re-target occurs.
It is a continuous time line with decisions made constantly all along it.

In fact the rather amusing side comment I could make is that yes there is one recent event that probably does ignore what I just said - the release of BF3 - maybe suddenly a lot of people decided to stop mining and play BF3? I don't know - but it would be interesting to see if there is a sudden dip in block generation rate around now (somewhere during the past 24 hours and the next 24 hours)

Anyway, for the past 2 re-targets (at least) the 3day average has always been worse than the 14day average
That isn't related to it having a higher SD, that's directly related to the environment.

Why? For the very obvious reason, there has been an almost continuous decline in miners over the past month.

What does that mean?

Well that's simple also, that the estimate at the end of the 14 days (over the past month) is always too high, it is pulled up by the higher results of the first 11 days versus the results of the last few days.

Thus there is a current environment and that current environment is decreasing at the moment, not going down in steps every 2 weeks.

The re-target maths ignores that possibility and in fact makes it worse.

  +1

  Slighty OT, but have you noticed the 2-day 'oscillation' in the global hash rate? Its weird but I've been observing it long enough to be pretty certain its really there. Probably coincidental and meaningless but I swear I see it. ;p

[Transisto]

...What does that mean?

Well that's simple also, that the estimate at the end of the 14 days (over the past month) is always too high, it is pulled up by the higher results of the first 11 days versus the results of the last few days.

Thus there is a current environment and that current environment is decreasing at the moment, not going down in steps every 2 weeks.

The re-target maths ignores that possibility and in fact makes it worse.

You're seeing a pattern out of two number (last two rediff).

What will you have to say when hashrate start to increase in the middle of a 2016 round and the difficulty even out and stay the same ?

We could discuss the effect on hashrate of BF3 Vs. temperature soon reaching sub-zero for a big part of N-America and many other things,
But please let's do so on another thread.

Unless low hash-rate is threatening the transactions delay beyond hours, I see no reason to worry.

If you were to use the last 3 day for setting the next difficulty it would create a point of vulnerability where people with 30-40% of total power could sway the diff to fit their agenda.

[sadpandatech]

Unless low hash-rate is threatening the transactions delay beyond hours, I see no reason to worry.

  That's pretty much the primary concern besides it just being more accurate.  It's the 'what if' type scenario that is more concerning. What if, the global hash power drops more than 50% in one 2016 period? The current setup will cause further crashing because of the delay in blocks and increased time for the miners to reach whatever normal equilibrium would be there.

[Transisto]

... What if, the global hash power drops more than 50% in one 2016 period? The current setup will cause further crashing because of the delay in blocks and increased time for the miners to reach whatever normal equilibrium would be there.

IMO The only thing that would cause a 50%+ drop is the release of a 2nd better crypto currency.  (namecoin scenario)

I'm all for adjusting more frequently but read there are incumbents to implement it.  (can't recall, sorry)

[gmaxwell]

 That's pretty much the primary concern besides it just being more accurate.  It's the 'what if' type scenario that is more concerning. What if, the global hash power drops more than 50% in one 2016 period? The current setup will cause further crashing because of the delay in blocks and increased time for the miners to reach whatever normal equilibrium would be there.

Then so what?  So transactions take longer to confirm. If we really lost more than half our hash power overnight then we'd probably have bigger concerns than somewhat slower txn processing for a couple of weeks.

The slow introduction of FPGA miners also makes sudden losses less likely, because their costs are even more frontloaded than the GPUs which make up most of our hash power now (and which appear to have significant hysteresis).

would it be worth the time and effort to code such a change

As I pointed out upthread, there is a real tradeoff vs security here. Some minor improvement against some hypothetical case isn't obviously a win vs a change in security exposure.   Making the change also requires a complete blockchain forking flag day event which would itself create non-trivial security risks (old clients ending up on a dead chain fork but believing txn on it).

The change you want would be far more clearly defined and far more easily implemented in the presence of an actual problem. E.g. if blocks were actually taking 6 hours it would be a lot easier to convince everyone to upgrade at once, and a lot easier to know exactly what change would be needed to address the actual situation at hand.

[sadpandatech]

I'm all for adjusting more frequently but read there are incumbents to implement it.  (can't recall, sorry)

  Two main ones; Higher potential rate of error with fewer blocks, and time and effort to change that part of the code.

  Now that you are up to speed, part of the issue is figuring out how big of a problem a big drop could have? Is it important enough to have it more accurate? If the rate did drop so drasticly, would we just say, "oh well, we tried" or would it be worth the time and effort to code such a change?

[sadpandatech]

  That's pretty much the primary concern besides it just being more accurate.  It's the 'what if' type scenario that is more concerning. What if, the global hash power drops more than 50% in one 2016 period? The current setup will cause further crashing because of the delay in blocks and increased time for the miners to reach whatever normal equilibrium would be there.

Then so what?  So transactions take longer to confirm. If we really lost more than half our hash power overnight then we'd probably have bigger concerns than somewhat slower txn processing for a couple of weeks.

The slow introduction of FPGA miners also makes sudden losses less likely, because their costs are even more frontloaded than the GPUs which make up most of our hash power now (and which appear to have significant hysteresis).

  I could picture a situation or two that could cause significant hash rate crashes but not be tied to the health of Bitcoin as a whole. However such crashes could make Bitcoin unhealthy as a whole.. Just my humble take on it.

[kano]

Yes I guess a few people have missed the point of why I created this thread in the first place

Simple, I don't like the idea that: well if it happens damn shame there's some worse wrong with bitcoin anyway ... ... (and I'll add - ok bye bye bitcoin)

Gavin has clearly implied that, gmaxwell has just above, I'm not going to re-read through all the posts, to find out if anyone else, but anyone else who wants to add their name to that list let me know.

There is clear obvious flaw in the current maths to handle the 'what if' I'm discussing and the only 'reasonable' argument given against changing it so far is that the slush or deepbit pool owners may use it to their advantage to lower the difficulty (and then either lose all their members due to being paid a lot less during the difficulty hack, or have to pay their members all that extra they didn't earn and somehow hide the fact that they did the difficulty hack)

Current difficulty change fully back to last is -15% so again it will extend at least an extra 2 days before the drop will cut in ... late ...

... and I love the suggestion

The change you want would be far more clearly defined and far more easily implemented in the presence of an actual problem. E.g. if blocks were actually taking 6 hours it would be a lot easier to convince everyone to upgrade at once, and a lot easier to know exactly what change would be needed to address the actual situation at hand.

Directly implying that it will be easier to fix the problem after the fact than to prepare for and try to prevent it before hand

You really prefer to hack in a quick fix to a drastic problem after it happens than to design a solution in advance that resolves the problem if it occurs?

[Transisto]

 I could picture a situation or two that could cause significant hash rate crashes but not be tied to the health of Bitcoin as a whole. However such crashes could make Bitcoin unhealthy as a whole.. Just my humble take on it.

If by unhealthy you mean, people panic and sell their bitcoin for USD because transaction speed is 50% slower than is used to be.

Then I guess anyone who know what cause lower hashrate and understand it is not linked to health of the bitcoin as a whole will make a killing buying low.

Do someone here have a better algorithms for adjusting the difficulty ?

[kano]

...
Do someone here have a better algorithms for adjusting the difficulty ?

I have a suggestion of course but the problem is getting someone else to come up with one ...

and the arguments against tend to be the same 'the security issue' without really giving a good real world example of it that isn't already an issue to BTC in other ways.

[gmaxwell]

and the arguments against tend to be the same 'the security issue' without really giving a good real world example of it that isn't already an issue to BTC in other ways.

I gave a concrete example with figures for a proposed change to a windowed system which wasn't even faster than the current one (making it faster just exacerbates the issue further)....  Which is a lot more specific than people waving their arms speculating "I could picture a situation or two" with respect to enormous hash rate falloffs.

And yes, I do think it's easier to fix a problem after the fact when you don't actually understand the potential problem concretely before the fact, and when there are a great many possible specific problems which may require different solutions— and when the solution carries real practical costs plus non-trivial theoretical security harms _AND_ must have pretty much complete and universal consensus.

Sure, security issues exist otherwise. But it's not like any difficulty adjustment change you could propose would completely solve all difficulty adjustment issues. If you take every security reduction which is just a change in magnitude but not kind, eventually you have no security at all.

Just an edit to make this clear:

Here are two examples that require drastically different difficulty handling solutions (of course there are many other possibilities than these): There is an energy crisis and electrical power jumps to the equivalent of $10/kwh everywhere  vs  another blockchain cryptocurrency becomes as popular as bitcoin and roughtly as profitable to mine.   To the first a one time step in difficulty reduction is the proper tool, not a faster syncup in general though faster might be sufficient. A single step is better than a faster change because it would address the need without harming security in the future.   To the second case merged mining with the new cryptocurrency would be the rational change: Making the difficulty algorithm faster would only cause deeper highly disruptive oscillations as hash power moved between chains.

[sadpandatech]

and the arguments against tend to be the same 'the security issue' without really giving a good real world example of it that isn't already an issue to BTC in other ways.

I gave a concrete example with figures for a proposed change to a windowed system which wasn't even faster than the current one (making it faster just exacerbates the issue further)....  Which is a lot more specific than people waving their arms speculating "I could picture a situation or two" with respect to enormous hash rate falloffs.

And yes, I do think it's easier to fix a problem after the fact when you don't actually understand the potential problem concretely before the fact, and when there are a great many possible specific problems which may require different solutions— and when the solution carries real practical costs plus non-trivial theoretical security harms _AND_ must have pretty much complete and universal consensus.

  Thanks for that, though I'd like to believe I put more effort into than just waving my arms about. Well, I may not understand it from the perspective you seem to proport to, I did state that shortening the number of blocks to recalc would cause more problems. I have also offered up atleast a vague idea about adding in another adjustment based on the most recent blocks.

  The very last part of your post is what I was hoping to see more of. The discussion that was intended in the thread, or atleast I thought, as to what the implications are of 'changing the formula' or 'not chaning it'...

[gmaxwell]

  Thanks for that, though I'd like to believe I put more effort into than just waving my arms about.

I'm interested in knowing what you expect to happen to cause the hash rate to fall faster than the current adjustment can keep up with it which won't in itself present material problems which dwarf transactions being slow while the difficulty catches up?

The only things I can think of that would cause massive super-fast hash-rate losses — things like major governments outlawing mining without warning— are both highly unlikely and devastatingly terribly completely independent of the difficulty calculation.

I would have previously bought a theory that if the exchange rate fell to the point where many GPU miners were unprofitable over electrical cost that hash rate would plummet, but this appears to be disproven by the evidence— it seems that there is enough mixture of energy costs (e.g. people with higher costs that shut off first), people whos energy usage is offset by being able to use the surplus heat, people with FPGAs and free power (partial or total energy cost insensitivity), and people with faith that bitcoin will gain value in the future that downward changes caused by exchange vs energy cost are fairly slow in practice.  It's gone down, of course, but not so quickly that its problematic.

In the meantime,  alt-chains with modified difficulty algorithms have been exploited thoroughly by attackers who used overpowering attacks which were assisted by the difficulty being too easily ramped.

So perhaps this was just the wrong time to have this debate: At the moment I don't think the evidence for changing the difficulty has ever been weaker and I don't think the evidence against it has ever been stronger.

[sadpandatech]

  Thanks for that, though I'd like to believe I put more effort into than just waving my arms about.

I'm interested in knowing what you expect to happen to cause the hash rate to fall faster than the current adjustment can keep up with it which won't in itself present material problems which dwarf transactions being slow while the difficulty catches up?

The only things I can think of that would cause massive super-fast hash-rate losses — things like major governments outlawing mining without warning— are both highly unlikely and devastatingly terribly completely independent of the difficulty calculation.

I would have previously bought a theory that if the exchange rate fell to the point where many GPU miners were unprofitable over electrical cost that hash rate would plummet, but this appears to be disproven by the evidence— it seems that there is enough mixture of energy costs (e.g. people with higher costs that shut off first), people whos energy usage is offset by being able to use the surplus heat, people with FPGAs and free power (partial or total energy cost insensitivity), and people with faith that bitcoin will gain value in the future that downward changes caused by exchange vs energy cost are fairly slow in practice.  It's gone down, of course, but not so quickly that its problematic.

In the meantime,  alt-chains with modified difficulty algorithms have been exploited thoroughly by attackers who used overpowering attacks which were assisted by the difficulty being too easily ramped.

  Thank you for taking the time to better clearify what you meant by 'waving arms' 

So perhaps this was just the wrong time to have this debate: At the moment I don't think the evidence for changing the difficulty has ever been weaker and I don't think the evidence against it has ever been stronger.

  You make a good case and hopefully this holds true for a while.

  Can we still debate what some of the better methods for changing the difficulty algo could be IF there were to be a need to? 

   Cheers,
      Derek

[gmaxwell]

  You make a good case and hopefully this holds true for a while.
  Can we still debate what some of the better methods for changing the difficulty algo could be IF there were to be a need to? 

Sorry, — you don't need my permission. I apologize if I was too heavy handed and made it seem that I was suggesting otherwise.  I'll try to confine any further comments I make to the thread to the assumption that a change would be made, and simply to questions about the best/worst ways to change it.