Difficulty adjustment needs modifying

[kano]

OK luke-jr I shouldn't have agreed with you about the time in a block.

I just ran another block table generation (not at the link but these are in the old link anyway) and it shows this:

148111   04:04:32 5-Oct-2011 UTC   0x1a09ee5d (1689334.4045971)   -73m 00s   9m 38.15s   +3.64%
148110   05:17:32 5-Oct-2011 UTC   0x1a09ee5d (1689334.4045971)   84m 01s   10m 01.40s   -0.23%
148109   03:53:31 5-Oct-2011 UTC   0x1a09ee5d (1689334.4045971)   13m 25s   10m 02.45s   -0.41%

Yes of course 148110 is Eligius - more than a whole hour?!?

If you look through my table probably every time you find a block with a negative time it is after an Eligius block
True for the first 3 and easy to check the block look at the coinbase for the word Eligius, or a whole bunch of outs in the generation transaction.

If the diff change occurred on block 148110, that one block would affect the calculation by extending the average by more than 2 seconds thus making the difficulty lower (yeah I said higher - fixed ...

I guess this is what was being referred to before?

[kano]

Updated diffcalc.php -> http://tradebtc.net/diffcalc.html (N.B. that's a copy of when I just ran it - it takes 3min to run)
Hmm firstly, 2weeks is ~17 hours from now but still 300 blocks to go ...

Counting back to last change the new diff would be -11.48%

However:
 432 blocks (~3 days) is -16.78%
 144 blocks (~1 day) is -25.48%

(and -84 blocks is -30.37% <- yeah I chose that one on purpose - the highest one close to 100)

Again, I really think this needs to be addressed BEFORE it happens ... and if this past week is any indication of the near future ...

[sadpandatech]

Updated diffcalc.php -> http://tradebtc.net/diffcalc.html (N.B. that's a copy of when I just ran it - it takes 3min to run)
Hmm firstly, 2weeks is ~17 hours from now but still 300 blocks to go ...

Counting back to last change the new diff would be -11.48%

However:
 432 blocks (~3 days) is -16.78%
 144 blocks (~1 day) is -25.48%

(and -84 blocks is -30.37% <- yeah I chose that one on purpose - the highest one close to 100)

Again, I really think this needs to be addressed BEFORE it happens ... and if this past week is any indication of the near future ...

 After looking at the top 5 pools(not counting 'other') today to get average user counts it was pretty obvious that a HUGE chunk of the hash power is in a very small percentage of hands. That said, I am in very strong aggreeance with you on this!

  Hopefully we will get lucky and those that may decide to switch off will do so during the last 1/4 of a 2016 chunk before a diff change. That would have the least amount of negative impact versus the more likely scenario that they would switch off right after dif change. 

[Transisto]

Updated diffcalc.php -> http://tradebtc.net/diffcalc.html (N.B. that's a copy of when I just ran it - it takes 3min to run)
Hmm firstly, 2weeks is ~17 hours from now but still 300 blocks to go ...

Counting back to last change the new diff would be -11.48%

However:
 432 blocks (~3 days) is -16.78%
 144 blocks (~1 day) is -25.48%

(and -84 blocks is -30.37% <- yeah I chose that one on purpose - the highest one close to 100)

Again, I really think this needs to be addressed BEFORE it happens ... and if this past week is any indication of the near future ...

At what point do we start to care ? If 85% of hashing power was to disappear overnight it would take an hours on average between blocks and we would have 3 month to explain to the user-base that it is not the end of the world yet.

If BTCs were to drop at 10c usd we would still have a fairly large % of population with free electricity that would not stop mining.

After looking at the top 5 pools(not counting 'other') today to get average user counts it was pretty obvious that a HUGE chunk of the hash power is in a very small percentage of hands. That said, I am in very strong aggreeance with you on this...

Pool centralization does not affect transaction speed nor difficulty change. If a big pool was to disappear it wouldn't take half a day for 2/3 of it's miners to move elsewhere.

[Luke-Jr]

If BTCs were to drop at 10c usd we would still have a fairly large % of population with free electricity that would not stop mining.

Very few people have free electricity.

[Transisto]

If BTCs were to drop at 10c usd we would still have a fairly large % of population with free electricity that would not stop mining.

Very few people have free electricity.

Free electricity for mining is about using generated heat for other purposes.  

AKA. Winter

Updated diffcalc.php -> http://tradebtc.net/diffcalc.html (N.B. that's a copy of when I just ran it - it takes 3min to run)

@Kano , Very nice stats, Shouldn't your average restart at zero after adjustments ? , block 1802   147167

[kano]

If BTCs were to drop at 10c usd we would still have a fairly large % of population with free electricity that would not stop mining.

Very few people have free electricity.

Free electricity for mining is about using generated heat for other purposes.  

AKA. Winter

Updated diffcalc.php -> http://tradebtc.net/diffcalc.html (N.B. that's a copy of when I just ran it - it takes 3min to run)

@Kano , Very nice stats, Shouldn't your average restart at zero after adjustments ? , block 1802   147167

Well the extra data after the adjustment is somewhat meaningless since it is a different difficulty,
but I decided to go with the simple calculation of just do 2017 blocks and mark 3 of interest.
In this case when we are close to the end, the data before the diff adjustment is pretty useless, but soon after a diff adjustment, it is worth looking at.
Anyway, only really those three % values are of real interest, the rest is just here to work out the numbers or run on a bit after the end.

I've just run it again (and updated it at the link)

Back to last difficulty: -12.67%
-432 blocks (~3 days): -23.06%
-144 blocks (~1 day): -37.09%

Interesting that -144 seems to be the lower limit and it's rising a bit since then
(though it's hard to tell for sure since the figures get very unreliable as you get closer to 1)

But earlier today I had some scary numbers:
back to last diff=-12.77% ~3days=-24.61% ~1day=-34.91% worst around 100 is -99=-51.84%

[sadpandatech]

After looking at the top 5 pools(not counting 'other') today to get average user counts it was pretty obvious that a HUGE chunk of the hash power is in a very small percentage of hands. That said, I am in very strong aggreeance with you on this...

Pool centralization does not affect transaction speed nor difficulty change. If a big pool was to disappear it wouldn't take half a day for 2/3 of it's miners to move elsewhere.

My point had nothing to do with the pools themselves. I was refering to the individul users who hold the majority of the hash power. If even a small number of them decide to flip the switch, it will hurt.....

And yes, many will keep going no matter what. I am one of those. But how much hash power do 'we' hold?

[kano]

Hmm - more than 24 hours late on the diff change and still more than 100 blocks to go
(so that means a guaranteed drop of at least 11%)

... and DeepBit died about an hour ago ...
(Edit: and Slush and some of BTCGuild)

I wonder what will happen to the trailing end of this difficulty ...
(start time was "22:40:40 27-Sep-2011 UTC")

[sadpandatech]

Hmm - more than 24 hours late on the diff change and still more than 100 blocks to go
(so that means a guaranteed drop of at least 11%)

... and DeepBit died about an hour ago ...
(Edit: and Slush and some of BTCGuild)

I wonder what will happen to the trailing end of this difficulty ...
(start time was "22:40:40 27-Sep-2011 UTC")

Aye, showing about 12% here now. I would think with so few blocks left the effect will be very minimal. Mainly due to the affected pool ops working their butts off to counter the attacks.

[kano]

Hmm - more than 24 hours late on the diff change and still more than 100 blocks to go
(so that means a guaranteed drop of at least 11%)

... and DeepBit died about an hour ago ...
(Edit: and Slush and some of BTCGuild)

I wonder what will happen to the trailing end of this difficulty ...
(start time was "22:40:40 27-Sep-2011 UTC")

Aye, showing about 12% here now. I would think with so few blocks left the effect will be very minimal. Mainly due to the affected pool ops working their butts off to counter the attacks.

Hmm - but I was thinking more: how long it might extend out for the remaining ~100 blocks.

[sadpandatech]

Aye, showing about 12% here now. I would think with so few blocks left the effect will be very minimal. Mainly due to the affected pool ops working their butts off to counter the attacks.

Hmm - but I was thinking more: how long it might extend out for the remaining ~100 blocks.

Aye, I caught your drift, m8. sittin at 13% now. ;p 149184 13/10/2011 22:55   the time is about what it was a few days ago when all the large pools had 24h+ of super bad luck. It has the same effect.

Judging by the last 10 blocks being back up a bit from a few hours ago (btc guild atleast mining again), I think the attacks are losing some steam. 

  The only thing they will accomplish doing such petty things to multiple places will be getting their botnet busted quicker than usual by 'the man'.

[kano]

I've just run it again (and updated it at the link http://tradebtc.net/diffcalc.php )

Back to last difficulty: -14.91%
-432 blocks (~3 days): -31.78%
-144 blocks (~1 day): -45.75%

Worst figure around -100 blocks: -82 blocks: -59.94%

So I guess the following means

Nope, bitcoin still has the off-by-1, as for now it's deemed "too big to fail" and fixing it would mean a backwards-incompatible change to the blockchain.

First: sorry for conflating the off-by-1 and the asymmetric adjustment issues.  As I said, I haven't taken the time to consider changing the bitcoin difficulty adjustment algorithm (too big a change for too little benefit, in my opinion; if we lose 90% of hashing power in a month then there that is a sign something much more serious is wrong with bitcoin than the difficulty algorithm; ...

there is ALMOST something seriously wrong with bitcoin now ...
(1 day worth of hashing suggests we might be half way to 90%)

[sadpandatech]

I've just run it again (and updated it at the link http://tradebtc.net/diffcalc.php )

Back to last difficulty: -14.91%
-432 blocks (~3 days): -31.78%
-144 blocks (~1 day): -45.75%

Worst figure around -100 blocks: -82 blocks: -59.94%

So I guess the following means

in my opinion; if we lose 90% of hashing power in a month then there that is a sign something much more serious is wrong with bitcoin than the difficulty algorithm; ...

there is ALMOST something seriously wrong with bitcoin now ...
(1 day worth of hashing suggests we might be half way to 90%)

  I sure hope you don't really mean that, Mr. Anderson.  I like to take a slightly more optimistic stance on it and attribute such a loss in hashing power in the near future to 'something much more serious wrong with the present state of the community at large', rather than Bitcoin itself. You and everyone else who have put in so much time and effort to build what is in concept nearly flawless certainly deserve more optimism. I'd like to think a massive loss in hash any time soon would be more of a 'weeding' out of all the crap. And potentially, could afford many who do see the value of the protocol a more stable environment to rebuild in.. just my 2 bitcents worth.


  Yea, kano, we are down to 14% drop now and if the current little trend continues into the next difficulty, Ddosing aside, we are looking at a 25-40% for the next one. Which is fine save for the potential for it being a bloody 4 week+ change time. :/  There is however another variable that may or may not enter the equation. That is our lil botnet buddies, if they decide to actually find a place to nest and mine. They are capable of about 3T worth on their own from best I can be calcumaltin'..  Interesting timess ahead, none the less.


  Again, I must agree with Kano that atleast some discussion should be had amongst those that know more than I about the 'What If' scenario of losing a massive chunk of hash early on in a dif change......
 

[kano]

OK I've been thinking about this more again.

I think the simplest solution, without the issues of having 2 different calculations interfering with each other, is to make the 2016 block diff recalc a 432 block diff recalc
Yep drop it from ~14 days to ~3 days.

Points:

If some bot net can mess up the diff recalc for 1.5 days of a short 3 day diff recalc
I really can't see them having all that much more of a problem also reaching 7 days of a 14 day diff recalc.
It's 4.7 times longer - not even an order of magnitude more.
It also means, however, that the amount of time the current difficulty will be extended due to a bot difficulty attack is quite obviously smaller for a smaller diff recalc

The difficulty is not adjusted based on the current network, it's based on the past network - since we cannot know what the current network really is.
It also is an estimate simply calculated from the past block generation rate.
This guarantees it will always be late to change.
Is that there by desire, or due to consideration of the issue with calculating it?

The last diff change extended 2 days longer that the expected 14 days.
I wonder how long this current one will go ...

If some statistics pro wants to work out and compare the expected standard deviation of 432 blocks vs 2016 blocks that would probably simplify a yes/no opinion of this.

[sadpandatech]

If some statistics pro wants to work out and compare the expected standard deviation of 432 blocks vs 2016 blocks that would probably simplify a yes/no opinion of this.

  432 would put it pretty high up on the curve there. Maybe 432 blocks sooner verses every 432 would keep us from being too far off. Its not as big of an issue coming back below 1mil difficulty. I think the problem lies, and probably what the thinking is for wanting it where it is, would be for the 10mil+ diff where a few points would make a much bigger impact on the deviation.

From Maaku's post on page 1;

[kano]

Oh forgot about that

Anyway, again the problem with the current is it isn't accurate at all - it's based on the past 2 weeks which means it is always wrong except when the network hash rate is unchanging.
Even though 3 days has a high (but not extreme) SD, the actual result over time shouldn't be that bad IMO
... and again in situations like now, it will speed up the diff change and help retard the major negative effects the 14 day delay is currently having

[Transisto]

kano, What problem are your trying to solve exactly ?

Estimating future difficulty adjustment or increasing re-target frequency ?

[kano]

The fact that the difficulty retarget is slow and late (and the result is wrong since it represents the previous 2 weeks not the current environment) and directly promotes the recent BTC economic climate.

If BTC climate it decreasing, it's slow to decrease the difficulty (and can be very slow to decrease it) and thus promotes making the decreasing environment worse.

[sadpandatech]

kano, What problem are your trying to solve exactly ?

Estimating future difficulty adjustment or increasing re-target frequency ?

  Both.

  Yes, Kano I can definetly agree its always off anyhows. But one needs to consider the costs verses the benfits of coding such a change. And the potential issues a change in the dataset being used could have. Using less than ~1600 blocks starts to have a pretty big error rate when at high difficulty(10mil+). This, as you pointed out is more caused by the fact we are looking at past blocks for the data set and not so much by the deviation. The only true fix would have to be able to see the future. its just not possible.

  One possible solution that I was pondering, that would atleast make it a little more 'current' rate friendly would be to keep the same 2016 formula intact and simply adjust the weight given to the last 144 blocks mined to make the retarget more accurate by focusing more on the present hash rate. Now, the issue here would be the ability to game that. Not as much as only using 144 blocks would, but it is still 'hackable'. I believe this could be solved by using a 'Heartbeat Adjustment' to the last 144 current weight. I.e., we'd take the largest, consistent block reporters and adjust the next retarget by the % they are above or below it for x number of blocks. We would adjust how much weight each one of them gave to the 'HA' based on uptime. So if one were to be heavy ddos victems their last block set would count for only %N uptime into the adjustment.

  I know this idea would be much more useful with some example formulas, which I will gladly draw up real quick if anyone is really interested in my ramblings. Otherwise I am sure the better math minds here can see it in their heads like I can. *And are probably facepalming at me. ;p

  Cheers