Bitcoin Forum

The last and next difficulty adjustment (and anyone who has watched the problems with the other crypto copies and the patches they have attempted) suggests that the 2 week difficulty adjustment should have a second test to adjust it early if needed.

Basically I'd propose that from 144 blocks (~24hrs) after a difficulty adjustment, and tested then and each 12 blocks (~2hrs) after that, if the actual calculated adjustment based on all blocks since the last adjustment is higher or lower by 50% than the current difficulty, then an early difficulty adjustment should kick in.
Of course those approximate values in hours for when to do this are way out if the problem should actually occur, however, there would need to be 144 blocks to attempt to ensure it doesn't happen completely unnecessarily due to likely random probability (as opposed to unlikely random probability)

This may never be needed, however, if it is needed Bitcoin $ value will obviously drop badly if it isn't implemented.
Simple example would be a short time frame drop in the network capacity by 50% would mean that the number of days remaining of the 2 weeks, would double thus extending out the slow down of the transaction confirmation in the network and thus having all the roll on effects that this would lead to including the possible self perpetuating spiral down that it could cause.

It would (as I said) be an up or down adjustment test, not just for downward adjustments, though the problems caused by a late upward adjustment would not be as severe as a late downward adjustment, thus the up test could be removed.

Comments? Suggestions?

Bitcoin does not have the problem with miners coming and going at anywhere near the level that seen with the alternates.

The last four difficulty adjustment periods each were between fourteen and fifteen days in duration:
 - http://docs.google.com/spreadsheet/ccc?key=0AmcTCtjBoRWUdHVRMHpqWUJValI1RlZiaEtCT1RrQmc&authkey=CMrV-rYE&hl=en_US&authkey=CMrV-rYE

That means instead of 10 minutes, blocks, on average, took 10 minutes and a few seconds.

Thus, this extra adjustment wouldn't take effect.

However, if the problem ever did occur, it would take effect.

Are you implying that could never happen?

At the moment it's at -7% due to perceived value and a game BF3 (it was -10% earlier today)
-7% seems a lot ...

So making no changes each block takes 10 minutes and 36 seconds, on average, for the 15 days, instead of 10 minutes.  Problem?

No problem ... and that is not what I was suggesting either ... :P
Is that why you ignored my question? You didn't read the 1st post?

I have a semi-constructive suggestion:

Anyone proposing a change in the feedback controller for difficulty should be required to show the stability region of his proposal. Pretty much everyone who tries to "improve" the PI controller implemented by Satoshi comes up with some hackneyed version of PID controller and then is surprised that it can be made to osciallte and is not even assymptotically stable in the Lyapunov sense if any nonlinearity is included.

http://en.wikipedia.org/wiki/Lyapunov_stability
http://en.wikipedia.org/wiki/PID_controller

I may have some software engineering disagreement with some of the Satoshi's choices, but the choice of PI regulator to adjust the difficulty is an example of excellent engineering: PI-s may be slow, but they are absolutely stable for every causal process and for every error signal.

PID controllers are completely standard, and the reason to use them (from my experience) is that they can be very easily fine tuned to optimal cybernetic states.

Can you explain your reasoning though for arguing that the difficulty adjustment is a PI controller? P = nActualTimespan / nTargetTimespan? Then where do you get the integral component from the difficulty re-calculation? To me it looks like a simple proportional scaling algorithm.

If I remember correctly it integrates the error over 2016 time intervals. This is some approximation of PI, more accurate approximation would be if the calculation for the expected block time was carried since the block 0 (time = -infinity).

Due to the esotheric block timekeeping the typical discrete-time transformation cannot be applied. If I remember correctly the block chain can be extended by a block up to two hours in the past. Because of that process under control is marginally acausal. This esotheric timekeeping precludes use of any standard mathematical tool from the theory of control systems.

So if you wanted to implement a discrete time approximation of P controller, you would measure the error over a single interblock interval. In case of non-monotonic block time you would have to set difficulty to some negative number.

PID, PI and P are terms from the the theory of Linear Time-Invariant systems and pretty much nobody deals with acausal systems. I agree that PID controllers are not that difficult to fine tune. But any tool (theorethic or practical) do do such tuning assumes causality. Moreover, the PID controller design with discrete time but variable time step are quite complex mathematically.

If the block time was changed from the current esotheric one to something which is monotonically increasing (like NTP time), then the change in the difficulty feedback loop could be considered and implemented with safety.

I did some simulation of blocktime variance (assuming honest nodes, and after calibration for network growth) for various difficulty adjustment intervals, posed on the freicoin forums (http://www.freicoin.org/implementations-details-and-bounty-7-3-btc-t13-10.html#p153). The nifty chart is copied below. The difference between one-week and two-week intervals was negligible, and is what I would recommend if a shorter interval was desired.

http://img836.imageshack.us/img836/276/clockvariance.jpg

A 24-hour interval (144 blocks) would have a variance/clock skew of 8.4%--meaning that one would *expect* the parameters governing difficulty adjustment to be in error by as much as 8.4% (vs bitcoin's current 2.2%). That's a significant difference. A 1-week retarget would have 3.8% variance. Twice-weekly would have 4.4% variance. I certainly wouldn't let it go any smaller than that..

Scenario 1: Something really bad happens and the Bitcoin exchange rate quickly drops to a tenth of its previous value. This is when mining was already close to breakeven, so for most miners it is no longer profitable to mine and they quit. Hashrate drops to a tenth of the previous value, blocks are found every 100 minutes, retargeting is in 5 months.

Scenario 2: Someone breaks the hashing function or builds a huge mining cluster, and uses it to attack Bitcoin. He drives the difficulty way up and then quits.

Scenario 3: It is decided to change the hashing function. Miners want to protect their investment so they play hardball bargaining and (threaten to) quit.

These can all be solved by hardcoding a new value for the difficulty, but wouldn't it be better to have an adjustment algorithm robust against this? Especially considering most Bitcoin activity will freeze until a solution is decided, implemented and distributed.

I know enough to understand the ideas of what you're saying, but not the specifics. And I think you're wrong about the last part. Integrating the error over 2016 time intervals is not an approximation for I, the integral from -infinity. It is an approximation for P, used rather than direct measurements (equivalent to 1 time interval) because the quantity measured is stochastic.

P is what exists currently.
I would fix problems with long-term trends. Currently, halving is done every less than 4 years because of the rising difficulty trend.
D would rapidly adapt to abrupt changes, basically what the OP is suggesting.

It's possible that the existing linear control theory doesn't apply directly to the block finding system and that P, I and D are only used metaphorically.
People who understand this stuff should gather and design a proper difficulty adjustment algorithm with all these components.

We're not talking about using the same primitive algorithm with a shorter timespan. We're talking about either a new intelligent algorithm, or a specific exception to deal with emergencies.

It is my understanding that at least Eligius pool isn't a "honest node" and intentionally produces acausal blocks (or at least as close to acausal as they deem practical).

Let me say what I said using a different terminology. The current regulator approximates a PI using a 2016-tap low-pass FIR filter with a sampe-and-hold (0-th order extrapolator) doing 2016 times subsampling of the output from the FIR.
Those are the famous last words. There were the days when average student of engineering would be splashed with an ink from the paper-tape analog data logger after cranking up the D term during his lab work. Nowadays I don't know how people learn the basics of system stability.

ArtForz had shown on one of the alternate chains both persistent oscillation and lack of assymptotic stability when people implement the diff-operator or some non-linear and/or time-varying approximations to a differentiator.

I know of the only one thing that could be safely done without fixing the problem of acausality or non-monotonicity of the time stamps.
From this I can see that it is possible to synthesize an absolutely stable 5-octave multirate filter bank that would work in front of a 63-times subsampler. There are two problems
for which I don't know the answer:
1) difficulty uses some custom floating point format with very low precision that may cause limit-cycle oscillations in a theorethically stable filter. If the format uses correct rounding then this wouldn't matter.
2) making changes to the difficulty 32 times more often would increase the probability of network splitting into disjoint block-chains, but I don't know how to estimate that.

Hmm, firstly I guess most notice, but I did not say to do away with the 2016 calculation.

What I am suggesting is specifically a way to handle large swings in the mining community and only that.
That's the reason for firstly waiting a full 144 blocks (normally ~1 day) before allowing it to even be checked and also for ignoring any changes less than 50%

I'm not talking about a general algorithm that would affect the normal working of the difficulty adjustment, but only actually have an effect if something drastic happened (i.e. network hash rate changed by at least 50%)

Namecoin has even had this specific problem happen already (and some of the scamcoins - I mean - alternate coins - have also seen this happen)

It's not something that would normally effect the difficulty adjustment calculation
i.e. it would give 'false' in any situation but some drastic network change

That's the reasoning behind the 50% check (and also of course the 144 block delay before even checking helps somewhat with network statistical variance)

At the moment the 2 week estimate is -9% (well I think that's right from the irc bot in the channel I visit) and that's still less than 1/5 of the value necessary for any early change to occur

The point is to have something that cuts in if something does go wrong but normally has no effect.

Waiting for it to happen first and then making a quick hack in bitcoin is IMO a bad idea, better to come up with something beforehand and yes discuss it as well.
However, the point of the original idea is to come up with a reasonable intervention and of course code that doesn't intervene except when needed.

What about ordering blocks with respect to time (just for the purposes of this calculation)?

If you can provide code or pseudo-code of or even just MATLAB code for this, I can bring it into an altchain for testing. I didn't pay attention enough in controls class to follow the discussion here :\

I think it is still risky. The risk would not be in out-of-order blocks but blocks-with-the-same-timestamps and a subsequent zerodivide or blocks-with-almost-the-same-timestamps and total numerical loss of accuracy.

Yeah, I was thinking about too. My copy of MATLAB is on an SGI O2 that is in the storage. I would not be able to deal with this yet. If I find some backups on a non-IRIX disk I'll post.

Allowing asymmetric adjustments can lead to security problems.  Artforz described a few when this came up on alternate chains.

Unless I'm seriously misunderstanding, no one is discussing asymmetric adjustments here..

how about real time difficulty adjustments? is that even possible?

No, coz all reasonable calculations of the network hash rate are based on the block finding rate - which is a random statistical value ...

You are.  This is exactly what is under discussion.

Yep that's what the first post directly implies :)
Asymmetric

However, making the vague 'can lead to security problems' comment is rather pointless.

The obvious security problem would be a sudden drop in difficulty well below the network hash rate.
That's the point of having a delay after a difficulty change and also have the % high

Feel free to suggest values based on some reasonably complex but viable calculation
(mine are just based on a feel of the actual network, watching the difficulty prediction after a change and seeing what happened with the hacks in the scam coins - but 'feeling' are often wrong and that's also why I've wanted to put the idea up for discussion)

I'm sure there are other 'security problems', but at least mention them ...
Even the current bitcoin has a WELL known one - 51% ... that seems to have the only well known reasonable fix of checkpoints that are implemented.

Edit: though I can't find any code to stop a fork from below the checkpoint ... hmm

He did. He said ArtForz brought them up in the Alternative Currencies board.

See this thread for details:
  https://bitcointalk.org/index.php?topic=43692.msg521772#msg521772 (https://bitcointalk.org/index.php?topic=43692.msg521772#msg521772)

Yes well that one has been discussed to death over there and there is even a fix for it, but no I've not looked at the bitcoin git to see if a related commit has gone in there ...

OK, so that one's OK to ignore. Anyone got some more? :)

I am quite serious about this idea because I hear a lot of people saying 'bitcoin will die soon' and although that really doesn't matter, I'm interested in it staying alive and this particular issue is one that could make a hiccough turn into a nightmare.
It is pretty easy to see the problem but (as I said) the solution should be to handle the unexpected case of it happening and otherwise have no effect.

I'm also certain that if the discussion changed to completely replacing the current 2016 method the discussion would go on forever and never be implemented.

Edit: all things in economics have ups and downs, but it's a really good idea to not help the bad downs go down faster yourself.

Fine, then ponder this for a while:
  https://bitcointalk.org/index.php?topic=42417.msg517020#msg517020

Executive summary:  asymmetric difficulty adjustments and adjustments based on time are really, really hard to get correct.

I haven't thought about it hard, but I trust ArtForz when he says he thinks it is IMPOSSIBLE to get it right.

How much hashing power do people running BF3 have to think this affect Bitcoin ?

10 min for a transaction is already so long than having it take even 30 min would not change usage much.

Sorry I don't see where this is going at all.

... As if not enough people have near free electricity and brought GPU specifically for mining that price of BTC or a game release threaten the network ...

Well that's a link back to around when he first pointed out the issue.
So your saying his fix that even he proposed doesn't work?
(yes I haven't looked at his fix - I guess I was silly to trust ArtForz's comments about there being a fix :P)

Anyway, there's a 144 block buffer in my suggestion already (the 12 block test can only re-diff once before the buffer hits again)
My reasoning for the buffer was to avoid statistical variance after a re-diff - but it seems to help with the time problem also.

The 'ArtForz' takeover requires 51% which really is a non issue for the main block chain.
If we suddenly fall low enough for someone to do a 51% takeover (other than a company like MS or google or a large government who could do it now anyway) then with the current software everyone will just ride it straight to the ground anyway due to no re-diff happening ever again.
/me ... wanders off to read up about the problems with the time fix and who actually implemented it ...

Going ... it's about the problem that if the network hash power did drop drastically, then the next difficulty recalculation could end up being weeks or even months away.
That would of course have obvious roll on effects on the new hashing power ... and could lead to a rather drastic downward spiral ... which could lead to all sorts of financial issues and decisions by the few companies/people who do allow BTC payments (among other things)

Edit: as stated in my first post

I'm of the opinion that if there is ever a huge sudden decline in the hashing power, there won't be a system to worry about.

But, since your scenario is about what to do in exactly that situation, you can't hide behind the "but bitcoin is huge" thing any more, because 51% isn't a big deal at that scale.

At any rate, the big problem is that it aligns the incentives in a bad way.  With an asymmetric system, each miner has a personal incentive to join the cabal.  The current system is careful about keeping the interests of each participant aligned with the interests of the system as a whole.

From the OP: “It would (as I said) be an up or down adjustment test, not just for downward adjustments.”

Maybe I'm misunderstanding what you mean by asymmetric?

EDIT: Regarding ArtForz's attack (the one you linked to), it was against against an off-by-one error in the implementation of bitcoin. That sort of thing is really orthogonal to the discussion here. Unless there is another attack he published that I'm not aware of?

The network hash could drop drastically if ... :

• New hype game come out : 1-3% drop (very likely, insignificant)
  
• Bitcoin price drop to ~1$: 40-50% drop depending on popularity of PFGAs,ASICs (Yes, many people actually pay this little and others use the heat for other purposes.)
  (very unlikely, can be tolerated)
• Natural disasters of the order to affect ~50% of world population at once for more than 1 month, (Yellowstone ?, Impact ?, Geomagnetic storm ?)
• Voluntary attempt at destabilizing the network in preparation of 51% attack,  Example: Massive sellout dropping the price to ~1$, combined with a 20-40% addition in hashing power at start of new adjustment (51% attack would now require ~33%) They'd need to attack fast because we'd have time to spread to word and keep mining at lost.
• ... ?

_{*Given 1.6m difficulty}

Hmm misunderstanding there I think.

I'm not hiding behind anything - you obviously misread.
My point says that if that happens it will probably be a scenario that falls into my suggested code 'intercept'.
However, without anything to handle it "everyone will just ride it straight to the ground anyway due to no re-diff happening ever again."

The estimate change I keep seeing for the current difficulty is 7% to 10% as it stands now.
If we see it reach 20% then not having something in place to handle 50% early would be tantamount to bitcoin suicide in my opinion.

I'd also say that everyone would have the same disincentive if the problem occurred.
The death of bitcoin (though that would be an incentive for those who want that ...)
I can't see "the interests of the system as a whole" related to the problem the system ignores if halfway through a 2016 the network hash rate has dropped to 50%
The possibility of a 50% drop causing a downward spiral are not nothing and are not related to aligned incentives at all ...

It seems to have dropped as much as 10% at the moment ...

I'm not saying this will happen, I'm suggesting a change in case it does happen.

Think of it this way: Is everyone who takes life insurance suicidal?

If the difficulty adjustment takes fewer blocks in certain circumstances, that is what I mean by asymmetric.  And it can be gamed.

That is the one he's referring to.
I've spent a little time finding related links:

ArtForz' last post about it (Forum time: September 13, 2011, 06:37:13 PM)
https://bitcointalk.org/index.php?topic=42417.msg523751#msg523751
saying that bitcoin doesn't have the fix included

I then went through the bitcoin git and read all the commits since 12/13 Sep:
https://github.com/bitcoin/bitcoin/commits/master

(AND a complete off topic comment on the last lot of commits:
LOL you let namecoin merged mining sneak in - did anyone realise that's what this is?
https://github.com/bitcoin/bitcoin/pull/476
Oh well, I guess it doesn't matter that they will be adding extra non-bitcoin data to the block-chain ...)

ArtForz' comment that there is a fix for the timewarp exploit:
https://bitcointalk.org/index.php?topic=42417.msg523714#msg523714

GG Git: ArtForz' NTP commit:
https://github.com/Lolcust/GeistGeld/commit/a25e1ce480f5bfbf7529bccd56df28b54c0eea77
basically trying to keep bitcoin's time accurate with the -ntp option
(on linux install ntpd as I always have :P)

But I can't find the change to the actual calculation ...
and I can't find anything like this link of Gavin's anywhere in GG or bitcoin:
https://bitcointalk.org/index.php?topic=42417.msg517020#msg517020

Yep it may take 1/14 of the blocks when needed.
So how can it be gamed ... unless you are simply referring to Artforz' attack he did on GG?
(that I have made a few comments about in the post above)

Nope, bitcoin still has the off-by-1, as for now it's deemed "too big to fail" and fixing it would mean a backwards-incompatible change to the blockchain.

The off-by-1 fix is completely orthogonal to the NTP stuff.
It's right here:
https://github.com/Lolcust/GeistGeld/commit/f9523f33ec22e26b7781f5a545fc74b4ecdc31a6#diff-3

Iirc my "why asymmetric diff adjustment is bad" was in a thread discussing SCs adjustment algo, and so far every asymmetric diff adjustment method I bothered to figure out a optimal strategy on has the same problem, a 51% attacker can create more blocks in the same nTime window than "he should be able to" while only doing the same # of hashes as the real chain over the same nTime window. And every time the factor gained happened to come out exactly as the maximum down adjustment over the maximum up adjustment for the same # of blocks.
Trivial example for how time-based has the same flaw, let's say daily retarget and /4 *4 limits
official chain hashes along at difficulty 1 for 2 days, so it has 2 difficulty-days and takes 2 days worth of nTime.
attackers chain has 0 blocks on day 1 (= diff retargets by /4), 8 times the normal blocks at diff 1/4 on day 2 (=diff retargets *4 and is back to 1 afterwards), so the attackers chain has a total of 1/4 * 8 = 2 difficulty-days, starts and ends with diff 1, also takes 2 days worth of nTime, but contains 4 times the # of blocks of the official chain.

Other clients won't accept the new blockchain unless it represents more work, which at no point it does... unless you side network has 51% the hash power of the main network, which would make this just a fancy 51% attack (50% in this example), and is not exploitable otherwise, no?

Yep, all the "fucking with timestamps" are "only" worsening the result of 51% attacks.
The problem is this changes the economic incentives of miners, as "defecting" is suddenly a lot more profitable...

Short version of the simple version, cooperators are mining "as planned", defectors fork the chain and communicate to collectively build a alternate version with optimally adjusted timestamps, N is the fraction of "cooperating" hashrate:

Bitcoin as planned:
N = 1 cooperate, everyone gets 1.
N > 0.5 cooperate, cooperators get 1/N, defectors get 0.
N < 0.5 cooperate, cooperators get 0, defectors get 1/(1-N).
N = 0 cooperate, everyone gets 1.

Solidcoin/ixcoin/i0coin/... with *1.1 /4 limits (edit: in theory, as the real chains all also have the same off-by-1 in their retargeting):
N = 1 cooperate, everyone gets 1.
N > 0.5 cooperate, cooperators get 1/N, defectors get 0.
N < 0.5 cooperate, cooperators get 0, defectors get 3.6/(1-N).
N = 0 cooperate, everyone gets 3.6.

Bitcoin with off-by-1:
N = 1 cooperate, everyone gets 1.
N > 0.5 cooperate, cooperators get 1/N, defectors get 0.
N < 0.5 cooperate, cooperators get 0, defectors get near infinite (really (what difficulty should be)/(minimum difficulty)).
N = 0 cooperate, everyone gets near INF.

edit: forgot, all current *1.1 /4 chains *also* have the off-by-1, so it's really "near infinite" for "defectors get a majority" there, too...

Under the “bitcoin as planned” alternative there is no incentive either way, so fix the off-by-1 and make sure adjustments are +/- the same maximum ("symmetric" by everyone but kjj's definition) and we're done, right? In other words, you need the off-by-1 or asymmetry for this to be exploitable, right?

Also, once new coins are not being generated, the issue goes away as well, right? It doesn't matter how many blocks you generate if the entire subsidy is coming from transaction fees (a blocknumber based demurrage would still suffer the same problem, however).

Can't this kind of problem be solved with a term corresponding to the I in PID? If more blocks than normal are found consistently, this will kick in and increase the difficulty.

Also, the suggestion in the OP isn't to change the limits asymmetrically. It's to expedite the adjustment when the recent history indicates the next adjustment is expected too far in the future. If the hashrate goes backup, the difficulty can rapidly increase just fine. Maybe even make it so that in this scenario the difficulty can even more easily go up, to further decrease any gameability.

All I'm saying is that just because some alts came up with a half-assed algo, doesn't mean the doomsday problem is unsolvable.


Done with ArtForz's attack, yes. But not with the problem of the OP.

I believe so. But the dynamics of that era are still not understood well enough even without considering potential attacks.

Wouldn't it be easiest to keep the original retarget algo (with 2016 block window etc), but just do the retargetting more often so that retarget windows overlap? You could retarget every block if you wanted (or every 10 or whatever).

With system like this, for example Namecoin difficulty would have dropped to "profitable" levels a long long time ago, and it would have suffered only a temporary slowdown.

Is there a downside to a system like this that I don't see? Some problem it doesn't solve or some exploit it introduces? It seems so simple :-)

Ah OK, your talking about getting the actual 50BTC per block also i.e. there's an actual direct gain of mining those low difficulty blocks as long as the network accepts the blocks and they are never rolled back (or you transfer them to some exchange and cash them out ... though that's pretty easy to trace as long as TradeHill or MtGox is willing to help)

But as stated further up (and I was going to post when he posted) there is the issue of the network accepting these low difficulty blocks.
I'd presume (without delving into the code) that it wouldn't accept them ...
Meh I'll look ... main.cpp ... yep GetNextWorkRequired is part of ProcessBlock/AcceptBlock/CheckBlock and it calculates the difficulty so the low difficulty blocks would be rejected anyway by the standard client
(and I just noticed it does the checkpoint there so you can't fork below the last checkpoint)

Meni Rosenfeld:
I never claimed otherwise, I only answered how I came to the conclusion that "naive algo with asymmetric limits is possibly disastrous".

Zibbo:
Sliding window is a interesting idea, you'd obviously have to reduce the per block adjustment factor and the limits to the 2016-th root to get the same overall adjustment speed as bitcoin, but at first look this should work without introducing new vulnerabilities and result in "smoother" adjustments (obv. not really *faster* if you use factors to end up with bitcoin-equiv adjustment rate, but less... well... blocky).

kano:
That's the whole point, the current network will happily accept chain-of-massive-number-of-low-diff-blocks over chain-of-less-harder-blocks as long as the sum of difficulty of the first is higher and it follows the "rules set in stone" (no invalid tx, generation amount <= calculated amount, difficulty == getNextDifficulty(prevblock), block nTime > median of prev 11 blocks, block nTime can't be more than 2 h in the future, ...).
Assuming someone forks the chain, creating a chain with more work than the real network over the same real time obviously requires the attacker(s) having more hashrate than the rest of the network... you know, kinda the definition of a 51% attack.
And as I already explained, creating such a low-difficulty chain starting at the same nTime as the real chain without having nTime of the forkchain blocks run off into the far future (as real clients wouldn't accept that, see above) *should* be impossible. But it isn't due to the off-by-1 in getNextDifficulty, am I talking Chinese here or something?

Well - reading through main.cpp

CBlock::AcceptBlock()
{
 checks for a dup hash (yeah that's an odd check to do still not sure why it does that)

 get the previous block

 height of this will be prev + 1

 check block nBits stated difficulty = GetNextWorkRequired(previous block)

 etc
}

GetNextWorkRequired(prev)
{
 if (height + 1) isn't a multiple of 2016
   return prev's difficulty

 otherwise find block old=2015 ago and use time diff of prev-old to work out new difficulty
}

Now unless CBlock::AcceptBlock() is the wrong function, I don't see how that can accept the first incorrect difficulty block.
Where is the off by 1 issue in what I'm looking at?

I'm not saying it isn't there, I'm simply saying I don't see it.

Not sure of the issues of negative time changes,
but it works it out based on the previous block compared to the block 2015 ago, not the current block (also)

Of course, if my change was added, then of course that's a whole different can of worms but my change only makes it change down to 144 blocks in the worst case instead of 2016 blocks

Edit: I guess once per 2016 blocks you could step it down 1/4 but then next 2016 would step it back up anyway
The only issue I do see is the negative time issues and that's real easy to ad a check if necessary in GetNextWorkRequired

This only has limited effectiveness against the doomsday problem. Even if you retarget on every block, you still have to wait for that first block which could take a while if things are really bad. And as ArtForz said you need to take a root of the correction (otherwise it explodes), so even after the block is found the correction is not enough.

It's taking the time over 2015 blocks. Every 2016 blocks. Last time I checked, 2015 != 2016.
In case you still don't get it, is the time taken from the last block with diff X to the first block with diff Y counted? Really?
But honestly, thats just making a elephant of a simple implementation flaw, nothing to do with the theoretical properties of the proper algorithm.
So back on topic... This idea actually looks promising, I yet have to run the numbers but I suspect it'll work "economically" as long as you keep the total adjustment possible over X blocks symmetrical for up vs. down (or even allow quicker up than down adjustment). Still somewhat worried about poisson noise causing issues for a faster-adjusting algo, but I guess that *should* average out over time...

Ah OK your referring to the fact that it's 2015 time ranges between 2016 blocks - typical start/end point issue
Since it starts from the first block after the last difficulty and counts that as 1 of 2016 which again gives 2015 time ranges or in average expected time, 2 weeks minus 10 minutes.
Don't worry I wont ask for the explanation of the problem that causes (but I'd guess it relates to alternates, not bitcoin though my guess is probably wrong)

I ran some simulations, I don't think there's anything wrong with it but by itself it won't solve the problem. I tried m=1 (adjustment every block), n=2016 (ratio calculated over the past 2016 blocks) and correction of ratio^(m/n). It takes 600 days to recover from a x100 drop in hashrate. Using a correction of ratio^(2m/n) it's still stable and improves it to 400 days. Maybe it's a matter of finding the highest stable exponent.

First: sorry for conflating the off-by-1 and the asymmetric adjustment issues.  As I said, I haven't taken the time to consider changing the bitcoin difficulty adjustment algorithm (too big a change for too little benefit, in my opinion; if we lose 90% of hashing power in a month then there that is a sign something much more serious is wrong with bitcoin than the difficulty algorithm; speculation on what will happen when the block reward drops seems pointless to me, I don't think we'll know until it happens).

Second: I've written 'discourage blocks' infrastructure:
  https://github.com/gavinandresen/bitcoin-git/tree/discourageblocks (https://github.com/gavinandresen/bitcoin-git/tree/discourageblocks)
  (code reviews welcome)

... which should give us the ability to nudge miners to Do The Right Thing.  Discouraging blocks that appear to be gaming the off-by-one bug should be enough incentive to prevent 50+% cartels from forming, without requiring a blockchain-splitting change.

Ah well then I guess we better hope it doesn't drop 50% for a reason other than the end of BTC.
Having nothing to handle that will almost certainly cause worse problems in that event :P

Current opinion of BTC, BTC $ value (and BTC $ vs electricity) and the release of BFC appear to be setting a drop of around 7% to 10% already this 2016 blocks

This is way off-topic from the OP... but Gavin, why couldn't ArtForz's patch be applied to core? Bumping the interval of consideration up from 2015 to 2016 will have no effect unless nodes are acting dishonestly. And this isn't far-fetched--as ArtForz points out the current situation gives miners significant incentive to collaborate and cheat.

"Shit happens" as they say. It would certainly mean there's something wrong, but not necessarily fatal. It will be fatal if we're not prepared. Also, merely preparing for the possibility may make it less likely by preventing a "hash run" - price drops, miners quit, transactions are confirmed more slowly, causing loss of confidence and more price drops, more miners quit...

I'm all for making the fix but I don't think the incentive is really that great. If miners collude to generate coins ahead of schedule, it will cause a loss of confidence in Bitcoin making their mined coins worthless.

Wrong, it's a forward-and-backward-incompatible change, as at the first retarget 2015 and 2016 have a good chance on disagreeing what the next target should be.

You can do both calculations and accept either-or during a transitionary period (and obviously do the old calculation for validation of blocks prior to the transition).

Even that wouldn't work. If the fixed calculation was ever lower than the old calculation during the transition period, there would be a permanent chain fork starting with the first block that didn't meet the old difficulty, but met the new difficulty, assuming over half of the hash power upgraded. If the fixed calculation was higher than the old calculation, nobody in their right mind would mine under the new calculation. Thus, the only solution is to permanently fork at a specific block in the future.

This is getting OT, but judging by the currently seen client versions on the network you'd probably need a prewarning/transitional period measured in years to have a good chance at all old clients upgrading in time...

Back on the original topic, ran some sims, doesn't look like there's a hard point where a faster difficulty adjustment using the original integrator becomes unstable (short of going to extremes so it slams into the limits from normal fluctuations in hashrate), "merely" makes difficulty adjustment more noisy. Near identical results with a sliding window and scaled factors (looks smoother, but pretty much the same deviation from optimal (as expected)).
For a interesting practical experiment in "just how short can you make the window and how extreme the adjustment", look at GeistGeld chain (retarget every 16 blocks, adjustment limit roughly * / 1.8. that comes out to a adjustment limit of */ somewhere around 1e30 over 2016 blocks). Somewhat surprisingly it more-or-less works so far...

Though, any change to the difficulty algo would require everyone to upgrade anyway.

You'd just have to put it in a specific version (e.g. 4.x release) with the new calculation only being used after block N some time in the future.

As for the 2015/2016 mistake, well that would have to work the same way, starting from some future block after a new release (but using the old count of 2015 timeframes for before that)

The code fix would be to simply count back 2016 instead of 2015 blocks since the time gap ignored (from the previous 2016 to 2015) is the current difficulty also - though the side effect would be to slightly decrease the difficulty - since the current calculation is ever so slightly high
(the correction will make the 2016 time longer thus the difficulty correction will be calculated to be smaller)

Hmm - need an incentive for everyone out there?
"The difficulty calculation in slightly high - upgrade so everyone can get the new corrected value that is slightly lower" :)

Consider this attack:   I have the ability to isolate the network connectivity many different nodes and I can pretend to be all of their peers (this isn't an especially hard attack— e.g. it's one people at large ISPs can pull)

I make a one time investment to fork the chain someplace after the highest of the currently used checkpoints, and mine enough blocks to get it down to a difficulty I can sustain on my own.  Once I've done this I can isolate fresh nodes and get them onto my fantasy chain (the fact that it's the sum-difficulty used for comparison is irrelevant because I've isolated them) and I can trigger their prudent must-have-n-confirms behavior before considering a txn safe.

Say I need to reduce it by 1024x to get it to where I can mine it on my own (which is about right, 1/1024 puts 10GH at ~11 minutes/block).  This would currently cost 134,268.75  BTC (the simple forgone income from the same amount of computation: 2016*(50/4^0) + 2016*(50/4^1) + 2016*(50/4^2) + 2016*(50/4^3) + 2016*(50/4^4.)).

If you switch to a sliding window with the same overall behavior your change clamp at each block will need to be at 0.25^1/2016.  You would still need to mine ~10080 blocks but total cost would be 72615 BTC because of the far fewer blocks calculated at the 'too high' difficulty.

So it would ~halve the cost of this attack.

The clamps in bitcoin are what make these attacks costly,  but the clamps also represent exciting non-linearities in the payout of the system which miners could exploit for increased profits.  The fact that the clamps are hard to reach currently makes it a non-issue, but with a sliding window the clamps would have to be very near a factor of 1.0 to preserve the resistance to these forged chain attacks, so the system would almost always be operating in the non-linear region.

Tightening the clamps to keep the attack cost the same would only worsen the non-linearity.

Moreover, (ignoring the screwed up calculation) the window plus node timestamp enforcement (the limitation against blocks from the future) limits the maximum gain miners miners can get from lying about the time to a couple percent.  A sliding window would make this worse because it would provide an incentive to lie for every block, and not just the final ones in a cycle.

People need to stop fixating on weirdness in other chains which are more or less irrelevant for bitcoin and realize that the design of bitcoin isn't an accident. Every one of the features of the distributed algorithm has a complicated relationship with all the others.

Eligius varies ntime to optimize getting work out to miners at a longpoll. Otherwise it could take many more seconds to generate work for all of them. It's not dishonest, just imprecise. The current variation allowances are not a problem, either.

Any change to how difficulty is calculated means a block chain fork. Any block chain fork should be avoided as long as feasibly possible. When it happens, we should have a large list of housekeeping changes to make at once.

You should post this in your FAQ. Many people are wondering whats wrong with your clock. If you happen to live in a country with an adversarial legal system, you will be invited to the court to explain this to the judge and the jury why the timestamps on your blocks are messing up the accounting ledgers of other people.
Very wise statement. Bitcoin is like amber, the bugs inside are to be admired for their beauty as long as possible.

Anyone using block times for anything important is quite simply a fool. It's not meant to be precise, and never will be. Even if I didn't (ab)use it for improved efficiency, it would be the time that (many) miners began work on the block, not the time it was found. And even if it were precise, it has no relevance.

This quote is worth preserving. There is a number of people on this forum who claim that bitcoin as-it-is-right-now is ready to maintain certifiable accounts for businesses. Who will be right?

Luke is not saying bitcon isn't good to go. He's saying the times shouldn't be used as times.

... and of course luke-jr is also correct about the times themselves.
They are the time you requested a getwork, which is NOT the time the block was found.

Looking at that in terms of a relationship between the two numbers: well the difference can be anything from 0 seconds up to 120 seconds.
(0 seconds if you are very lucky, have a very fast hashing device and very short network latency)
Default pushpool accepts work up to 120 seconds after the getwork request?
(msg.c:   WORK_EXPIRE_INT      = 120,)

So I guess that's also a possible hack to make your blocks always win if they are in an orphan battle?
Since you expect people to respond in 60 seconds (and most mining programs assume that also) the pool can push the time back almost 60 seconds and the miners would see no unexpected extra expired blocks? ...

Meanwhile ... back on topic.
So the suggestions are to actually redo the default 2016 calculation?
(due to the unstated but expected hacks due to having 2 different calculations working together ...)

Edit: though by the sounds of things it wont get implemented anyway ...

Is anyone maintaining a branch with these sort of housekeeping changes? A block chain fork may be years in the future, but it might be worth maintaining such a branch to keep such long-term changes in mind.

Good idea. Who wants to volunteer? I'm too busy...

I was thinking that would be a good idea. I can help, but I don't think I have time to maintain such a branch by myself.

Just an FYI
Difficulty estimator at #ozcoin (;;bc,diffchange) is saying:
 -9.4% expected
but calculated based on only the last 3 days:
 -13.2% ...

Is Ozcoi.in estimate more accurate than bitcoinchart @ -5% ?

Estimate based on 3 day are near meaningless because of luck factor and anyway difficulty adjust based on past 2016, (we're already half-way).

IMO what we're seeing was the delay for people in EU to realize unprofitably and stopping their rig.

Hell you don't even have to have an attack, and the "doomsday" scenario is written into the code.
Mining right now is marginally profitable if you have efficient GPUs and cheap electricity.
It fairly obviously isn't profitable for a decent number of people, as evidenced by the dropping hash rate and difficulty.
Now look into the future a little it to the 50% drop in rewards.
Presto!  Anybody without free electricity won't be mining profitably anymore, and bitcoin has a namecoin type issue.

Bitcoin prices better at least double by then, or bitcoin is in serious trouble.

* Have you pondered over ASICs efficiency to say "efficient GPUs"
* Winter in coming in the US so don't worry about people not having free electricity.
* Price does not have to double, only difficulty has to halve.
* Calm down.

Halving is not going to cause doomsday, for several reasons.
 - Capital expenditure is a major component in mining cost. First, this means that there will be plenty of people who are making more than twice their electricity cost.
 - Second, it means that in the time before halving, people will avoid buying hardware in anticipation of decreased profitability, so the difficulty will be less than it would have otherwise been.
 - The price will gradually increase in the time before halving in anticipation of the reduced supply.


Not a counterargument by itself, because the point with doomsday is that difficulty doesn't get a chance to adjust.

about THE halving to 25btc, I think volumes will be written on the subject, when time will come.

Lots of things may change by then like a bigger share of miners using ASIC-FPGA, and greater adoption overall.
And at that time there will still be people mining in winter scenario, enough for having the difficulty adjusted within a month or so.

What fatal difference does that make to wait 30min per confirmation instead of 10min ?
When investing into bitcoins this a basics concept to understand and live with.

The problem is that if you have a drop-off of 50%, that's 50% of the potential computing power of the network that is now disillusioned with mining. And in fact, once you cross that 50% threshold it becomes *more* profitable for such miners to collaborate on forking the bitcoin (in terms of BTC, ignoring for the moment what effect this would have on the exchanges).

In reality people come and go for their own reasons, and you're never going to get 100% buy-in to cheat the system from miners who left. But it becomes a possibility once you cross that 50% threshold, so a conservative approach would be to never let that happen.

OK then I calculated it myself - feel free to point out any errors in:
http://tradebtc.net/diffcalc.html

Firstly "3 day are near meaningless" - um you ever done statistics?
see if you understand this: a sample > 20% of population ...

OK so from my table:
last 432 times:
 147800   18:00:10 2-Oct-2011 UTC   0x1a09ee5d (1689334.4045971)   14m 40s   10m 51.66s   -8.61%
back to first block after last diff change:
 147168   22:47:04 27-Sep-2011 UTC   0x1a09ee5d (1689334.4045971)   6m 24s   10m 53.95s   -8.99%
last 2016 times:
 146216   00:35:02 21-Sep-2011 UTC   0x1a098ea5 (1755425.3203287)   1m 36s   10m 41.79s   -6.96%

So I guess bitcoinchart is a piece of ... also

at 1060/2016  52%
2011-09-28 we are on the 5th so
29-30-31-1-2-3-4-5 , 8day out of 14 57%
We're at expected 9.6% increase...

You are right BTCchart is POS.

Ayee, we went -3.76495171661444% 9/27 and, using http://dot-bit.org/tools/nextDifficulty.php has been the closest thing to being accurate I have seen.

according to dot-bit;
Instant as of block 148,241 = +1.35848534244243%
Expected at block  149,184 = -3.92677906619912%

That will likely change and the instant was -4%~ earlier due to variance causing 2 days worth of bad luck for a few of the larger pools.

My table is simply the data for the last 2017 blocks.
Do with it as you will.

However "Instant" yes that figure is COMPLETELY meaningless (yes that is an example of it being true)

as for the guess at -4% I'd love to know where they even get that from
Look at my actual figures and tell me how you can estimate -4% ?

Take the number at half way and divide by 2? :D :D :D

Sorry, did not see the table before. I was wondering where you got your %'s from.

 Couple of 'errors', well, questions really. Where are you getting your starting "running average" figure from?

  And, wouldn't the running average time be more accurate if at the point of difficulty change you bumped it up or down by the same % that difficulty adjusted. Being that in the case of it going down
the 'average' time to solve should be less so the penalty in adjustment % would be greater. Its a sloppy fix, and not being a smarty guy like you peeps I can't offer up a fancy, terminology laced explanation, but give it a try and see. ;p

i.e. 147167 - 147168 should consider the difficulty adjustment, instead it appears to be the same as 147166 - 147167
147168 22:47:04 27-Sep-2011 UTC 0x1a09ee5d (1689334.4045971) 6m 24s 10m 53.95s -8.99%
147167 22:40:40 27-Sep-2011 UTC 0x1a098ea5 (1755425.3203287) 2m 00s 10m 53.45s -8.91%
147166 22:38:40 27-Sep-2011 UTC 0x1a098ea5 (1755425.3203287) 1m 58s 10m 52.95s -8.83%


Should we not bump the running time average from 10m 53.95s up by .37647003433787% to 11m 18.57s ?

Edit; very sloppy on my part. Would like to see the actual spreadsheet to see how you are doing the math. I know the time is lacking the needed adjustment for dif change but am not sure about it being fixed by bumping at the 'running average'. We would want to add the % change I suggested into the formula you are using for running average before it actually calculates the new running average time. Which if my brain isn't completly fried would only be adding the .3%~ to the .50s, the expected calculated change in time. So instead of 50s up that it currently is, it would be .5188s added to 53.45s or a new current running average time of 10m 53.97s

Very trivial when I look at it like that, but across a few more dif changes it would begin to add up if not corrected for. imho.

My running average is simply the average of all block generation times from the top down to the line it's shown on.
The first one is of course the time to generate the last block, the 2nd is the average of 1 and 2 etc ...
So you can see what the average is at any point running back from 'now'
(now being block 148231 23:57:28 5-Oct-2011 UTC - in the past)
I can regenerate it (it's just php - and takes - as you can read - almost 4 minutes - to do 2017 getblockbycount's to my bitcoind)

The standard difficulty calculation is only interested in the blocks from calculation time back to the last difficulty change,
so I've made the averages and % relate to that.

Yeah I guess the 2016 line isn't really accurate since it crossed the diff change boundary,
ignore it :)

ahhh, must need sleep. Also did not see you were running backwords. ;/  Why not from bottom up?
edited my last post a bit to fix my half ass attempt at maths...


hehe, I am fairly certain even if I keep missing the rest that my bump idea will fix that. Just do it in reverse, since I for whatever dumb reason thought you were calculating from older first. so instead of it dropping .50s, drop it .52s

OK luke-jr I shouldn't have agreed with you about the time in a block.

I just ran another block table generation (not at the link but these are in the old link anyway) and it shows this:

148111   04:04:32 5-Oct-2011 UTC   0x1a09ee5d (1689334.4045971)   -73m 00s   9m 38.15s   +3.64%
148110   05:17:32 5-Oct-2011 UTC   0x1a09ee5d (1689334.4045971)   84m 01s   10m 01.40s   -0.23%
148109   03:53:31 5-Oct-2011 UTC   0x1a09ee5d (1689334.4045971)   13m 25s   10m 02.45s   -0.41%

Yes of course 148110 is Eligius - more than a whole hour?!?

If you look through my table probably every time you find a block with a negative time it is after an Eligius block
True for the first 3 and easy to check the block look at the coinbase for the word Eligius, or a whole bunch of outs in the generation transaction.

If the diff change occurred on block 148110, that one block would affect the calculation by extending the average by more than 2 seconds thus making the difficulty lower (yeah I said higher - fixed :) ...

I guess this is what was being referred to before?

Updated diffcalc.php -> http://tradebtc.net/diffcalc.html (N.B. that's a copy of when I just ran it - it takes 3min to run)
Hmm firstly, 2weeks is ~17 hours from now but still 300 blocks to go ...

Counting back to last change the new diff would be -11.48%

However:
 432 blocks (~3 days) is -16.78%
 144 blocks (~1 day) is -25.48%

(and -84 blocks is -30.37% <- yeah I chose that one on purpose - the highest one close to 100)

Again, I really think this needs to be addressed BEFORE it happens ... and if this past week is any indication of the near future ...

 After looking at the top 5 pools(not counting 'other') today to get average user counts it was pretty obvious that a HUGE chunk of the hash power is in a very small percentage of hands. That said, I am in very strong aggreeance with you on this!

  Hopefully we will get lucky and those that may decide to switch off will do so during the last 1/4 of a 2016 chunk before a diff change. That would have the least amount of negative impact versus the more likely scenario that they would switch off right after dif change.  :-\

At what point do we start to care ? If 85% of hashing power was to disappear overnight it would take an hours on average between blocks and we would have 3 month to explain to the user-base that it is not the end of the world yet.

If BTCs were to drop at 10c usd we would still have a fairly large % of population with free electricity that would not stop mining.


Pool centralization does not affect transaction speed nor difficulty change. If a big pool was to disappear it wouldn't take half a day for 2/3 of it's miners to move elsewhere.

Very few people have free electricity.

Free electricity for mining is about using generated heat for other purposes.  

AKA. Winter

@Kano , Very nice stats, Shouldn't your average restart at zero after adjustments ? , block 1802   147167

Well the extra data after the adjustment is somewhat meaningless since it is a different difficulty,
but I decided to go with the simple calculation of just do 2017 blocks and mark 3 of interest.
In this case when we are close to the end, the data before the diff adjustment is pretty useless, but soon after a diff adjustment, it is worth looking at.
Anyway, only really those three % values are of real interest, the rest is just here to work out the numbers or run on a bit after the end.

I've just run it again (and updated it at the link)

Back to last difficulty: -12.67%
-432 blocks (~3 days): -23.06%
-144 blocks (~1 day): -37.09%

Interesting that -144 seems to be the lower limit and it's rising a bit since then
(though it's hard to tell for sure since the figures get very unreliable as you get closer to 1)

But earlier today I had some scary numbers:
back to last diff=-12.77% ~3days=-24.61% ~1day=-34.91% worst around 100 is -99=-51.84%

My point had nothing to do with the pools themselves. I was refering to the individul users who hold the majority of the hash power. If even a small number of them decide to flip the switch, it will hurt.....

And yes, many will keep going no matter what. I am one of those. But how much hash power do 'we' hold?

Hmm - more than 24 hours late on the diff change and still more than 100 blocks to go :(
(so that means a guaranteed drop of at least 11%)

... and DeepBit died about an hour ago ...
(Edit: and Slush and some of BTCGuild)

I wonder what will happen to the trailing end of this difficulty ...
(start time was "22:40:40 27-Sep-2011 UTC")

Aye, showing about 12% here now. I would think with so few blocks left the effect will be very minimal. Mainly due to the affected pool ops working their butts off to counter the attacks.

Hmm - but I was thinking more: how long it might extend out for the remaining ~100 blocks.

Aye, I caught your drift, m8. sittin at 13% now. ;p 149184 13/10/2011 22:55   the time is about what it was a few days ago when all the large pools had 24h+ of super bad luck. It has the same effect.

Judging by the last 10 blocks being back up a bit from a few hours ago (btc guild atleast mining again), I think the attacks are losing some steam. 

  The only thing they will accomplish doing such petty things to multiple places will be getting their botnet busted quicker than usual by 'the man'.

I've just run it again (and updated it at the link http://tradebtc.net/diffcalc.php )

Back to last difficulty: -14.91%
-432 blocks (~3 days): -31.78%
-144 blocks (~1 day): -45.75%

Worst figure around -100 blocks: -82 blocks: -59.94%

So I guess the following means
there is ALMOST something seriously wrong with bitcoin now ...
(1 day worth of hashing suggests we might be half way to 90%)

  I sure hope you don't really mean that, Mr. Anderson.  I like to take a slightly more optimistic stance on it and attribute such a loss in hashing power in the near future to 'something much more serious wrong with the present state of the community at large', rather than Bitcoin itself. You and everyone else who have put in so much time and effort to build what is in concept nearly flawless certainly deserve more optimism. I'd like to think a massive loss in hash any time soon would be more of a 'weeding' out of all the crap. And potentially, could afford many who do see the value of the protocol a more stable environment to rebuild in.. just my 2 bitcents worth.


  Yea, kano, we are down to 14% drop now and if the current little trend continues into the next difficulty, Ddosing aside, we are looking at a 25-40% for the next one. Which is fine save for the potential for it being a bloody 4 week+ change time. :/  There is however another variable that may or may not enter the equation. That is our lil botnet buddies, if they decide to actually find a place to nest and mine. They are capable of about 3T worth on their own from best I can be calcumaltin'..  Interesting timess ahead, none the less.


  Again, I must agree with Kano that atleast some discussion should be had amongst those that know more than I about the 'What If' scenario of losing a massive chunk of hash early on in a dif change......
 

OK I've been thinking about this more again.

I think the simplest solution, without the issues of having 2 different calculations interfering with each other, is to make the 2016 block diff recalc a 432 block diff recalc
Yep drop it from ~14 days to ~3 days.

Points:

If some bot net can mess up the diff recalc for 1.5 days of a short 3 day diff recalc
I really can't see them having all that much more of a problem also reaching 7 days of a 14 day diff recalc.
It's 4.7 times longer - not even an order of magnitude more.
It also means, however, that the amount of time the current difficulty will be extended due to a bot difficulty attack is quite obviously smaller for a smaller diff recalc

The difficulty is not adjusted based on the current network, it's based on the past network - since we cannot know what the current network really is.
It also is an estimate simply calculated from the past block generation rate.
This guarantees it will always be late to change.
Is that there by desire, or due to consideration of the issue with calculating it?

The last diff change extended 2 days longer that the expected 14 days.
I wonder how long this current one will go ...

If some statistics pro wants to work out and compare the expected standard deviation of 432 blocks vs 2016 blocks that would probably simplify a yes/no opinion of this.

  432 would put it pretty high up on the curve there. Maybe 432 blocks sooner verses every 432 would keep us from being too far off. Its not as big of an issue coming back below 1mil difficulty. I think the problem lies, and probably what the thinking is for wanting it where it is, would be for the 10mil+ diff where a few points would make a much bigger impact on the deviation.

From Maaku's post on page 1;
http://img836.imageshack.us/img836/276/clockvariance.jpg

Oh forgot about that :)

Anyway, again the problem with the current is it isn't accurate at all - it's based on the past 2 weeks which means it is always wrong except when the network hash rate is unchanging.
Even though 3 days has a high (but not extreme) SD, the actual result over time shouldn't be that bad IMO
... and again in situations like now, it will speed up the diff change and help retard the major negative effects the 14 day delay is currently having

kano, What problem are your trying to solve exactly ?

Estimating future difficulty adjustment or increasing re-target frequency ?

The fact that the difficulty retarget is slow and late (and the result is wrong since it represents the previous 2 weeks not the current environment) and directly promotes the recent BTC economic climate.

If BTC climate it decreasing, it's slow to decrease the difficulty (and can be very slow to decrease it) and thus promotes making the decreasing environment worse.

  Both.

  Yes, Kano I can definetly agree its always off anyhows. But one needs to consider the costs verses the benfits of coding such a change. And the potential issues a change in the dataset being used could have. Using less than ~1600 blocks starts to have a pretty big error rate when at high difficulty(10mil+). This, as you pointed out is more caused by the fact we are looking at past blocks for the data set and not so much by the deviation. The only true fix would have to be able to see the future. its just not possible.

  One possible solution that I was pondering, that would atleast make it a little more 'current' rate friendly would be to keep the same 2016 formula intact and simply adjust the weight given to the last 144 blocks mined to make the retarget more accurate by focusing more on the present hash rate. Now, the issue here would be the ability to game that. Not as much as only using 144 blocks would, but it is still 'hackable'. I believe this could be solved by using a 'Heartbeat Adjustment' to the last 144 current weight. I.e., we'd take the largest, consistent block reporters and adjust the next retarget by the % they are above or below it for x number of blocks. We would adjust how much weight each one of them gave to the 'HA' based on uptime. So if one were to be heavy ddos victems their last block set would count for only %N uptime into the adjustment.

  I know this idea would be much more useful with some example formulas, which I will gladly draw up real quick if anyone is really interested in my ramblings. Otherwise I am sure the better math minds here can see it in their heads like I can. *And are probably facepalming at me. ;p

  Cheers

There is no such thing as "current environment" , be it the previous 3 day or previous hour.

It take roughly 2 week to purchase GPU and set them up for mining.

I don't think Day traders and weekly miners count for even 1% of the ecosystem.

weekly miner mean, oh this week the price is low I will stop mining, oh the difficulty we be low in 2 week, lets purchase tons of GPU.

Unfortunately that ignores reality.

Just because the bitcoin re-target time line occurs approximately once every 2 weeks, doesn't mean that everyone who mines makes decisions the minute the re-target occurs.
It is a continuous time line with decisions made constantly all along it.

In fact the rather amusing side comment I could make is that yes there is one recent event that probably does ignore what I just said - the release of BF3 - maybe suddenly a lot of people decided to stop mining and play BF3? I don't know - but it would be interesting to see if there is a sudden dip in block generation rate around now (somewhere during the past 24 hours and the next 24 hours)

Anyway, for the past 2 re-targets (at least) the 3day average has always been worse than the 14day average
That isn't related to it having a higher SD, that's directly related to the environment.

Why? For the very obvious reason, there has been an almost continuous decline in miners over the past month.

What does that mean?

Well that's simple also, that the estimate at the end of the 14 days (over the past month) is always too high, it is pulled up by the higher results of the first 11 days versus the results of the last few days.

Thus there is a current environment and that current environment is decreasing at the moment, not going down in steps every 2 weeks.

The re-target maths ignores that possibility and in fact makes it worse.

  +1

  Slighty OT, but have you noticed the 2-day 'oscillation' in the global hash rate? Its weird but I've been observing it long enough to be pretty certain its really there. Probably coincidental and meaningless but I swear I see it. ;p

You're seeing a pattern out of two number (last two rediff).

What will you have to say when hashrate start to increase in the middle of a 2016 round and the difficulty even out and stay the same ?

We could discuss the effect on hashrate of BF3 Vs. temperature soon reaching sub-zero for a big part of N-America and many other things,
But please let's do so on another thread.

Unless low hash-rate is threatening the transactions delay beyond hours, I see no reason to worry.

If you were to use the last 3 day for setting the next difficulty it would create a point of vulnerability where people with 30-40% of total power could sway the diff to fit their agenda.

  That's pretty much the primary concern besides it just being more accurate.  It's the 'what if' type scenario that is more concerning. What if, the global hash power drops more than 50% in one 2016 period? The current setup will cause further crashing because of the delay in blocks and increased time for the miners to reach whatever normal equilibrium would be there.

IMO The only thing that would cause a 50%+ drop is the release of a 2nd better crypto currency.  (namecoin scenario)

I'm all for adjusting more frequently but read there are incumbents to implement it.  (can't recall, sorry)

Then so what?  So transactions take longer to confirm. If we really lost more than half our hash power overnight then we'd probably have bigger concerns than somewhat slower txn processing for a couple of weeks.

The slow introduction of FPGA miners also makes sudden losses less likely, because their costs are even more frontloaded than the GPUs which make up most of our hash power now (and which appear to have significant hysteresis).


As I pointed out upthread, there is a real tradeoff vs security here. Some minor improvement against some hypothetical case isn't obviously a win vs a change in security exposure.   Making the change also requires a complete blockchain forking flag day event which would itself create non-trivial security risks (old clients ending up on a dead chain fork but believing txn on it).

The change you want would be far more clearly defined and far more easily implemented in the presence of an actual problem. E.g. if blocks were actually taking 6 hours it would be a lot easier to convince everyone to upgrade at once, and a lot easier to know exactly what change would be needed to address the actual situation at hand.

  Two main ones; Higher potential rate of error with fewer blocks, and time and effort to change that part of the code.

  Now that you are up to speed, part of the issue is figuring out how big of a problem a big drop could have? Is it important enough to have it more accurate? If the rate did drop so drasticly, would we just say, "oh well, we tried" or would it be worth the time and effort to code such a change?

  I could picture a situation or two that could cause significant hash rate crashes but not be tied to the health of Bitcoin as a whole. However such crashes could make Bitcoin unhealthy as a whole.. Just my humble take on it.

Yes I guess a few people have missed the point of why I created this thread in the first place :)

Simple, I don't like the idea that: well if it happens damn shame there's some worse wrong with bitcoin anyway ... ... (and I'll add - ok bye bye bitcoin)

Gavin has clearly implied that, gmaxwell has just above, I'm not going to re-read through all the posts, to find out if anyone else, but anyone else who wants to add their name to that list let me know.

There is clear obvious flaw in the current maths to handle the 'what if' I'm discussing and the only 'reasonable' argument given against changing it so far is that the slush or deepbit pool owners may use it to their advantage to lower the difficulty (and then either lose all their members due to being paid a lot less during the difficulty hack, or have to pay their members all that extra they didn't earn and somehow hide the fact that they did the difficulty hack)

Current difficulty change fully back to last is -15% so again it will extend at least an extra 2 days before the drop will cut in ... late ...

... and I love the suggestion
Directly implying that it will be easier to fix the problem after the fact than to prepare for and try to prevent it before hand :)

You really prefer to hack in a quick fix to a drastic problem after it happens than to design a solution in advance that resolves the problem if it occurs?

If by unhealthy you mean, people panic and sell their bitcoin for USD because transaction speed is 50% slower than is used to be.

Then I guess anyone who know what cause lower hashrate and understand it is not linked to health of the bitcoin as a whole will make a killing buying low.

Do someone here have a better algorithms for adjusting the difficulty ?

I have a suggestion of course but the problem is getting someone else to come up with one ...

and the arguments against tend to be the same 'the security issue' without really giving a good real world example of it that isn't already an issue to BTC in other ways.

I gave a concrete example with figures for a proposed change to a windowed system which wasn't even faster than the current one (making it faster just exacerbates the issue further)....  Which is a lot more specific than people waving their arms speculating "I could picture a situation or two" with respect to enormous hash rate falloffs.

And yes, I do think it's easier to fix a problem after the fact when you don't actually understand the potential problem concretely before the fact, and when there are a great many possible specific problems which may require different solutions— and when the solution carries real practical costs plus non-trivial theoretical security harms _AND_ must have pretty much complete and universal consensus.

Sure, security issues exist otherwise. But it's not like any difficulty adjustment change you could propose would completely solve all difficulty adjustment issues. If you take every security reduction which is just a change in magnitude but not kind, eventually you have no security at all.

Just an edit to make this clear:

Here are two examples that require drastically different difficulty handling solutions (of course there are many other possibilities than these): There is an energy crisis and electrical power jumps to the equivalent of $10/kwh everywhere  vs  another blockchain cryptocurrency becomes as popular as bitcoin and roughtly as profitable to mine.   To the first a one time step in difficulty reduction is the proper tool, not a faster syncup in general though faster might be sufficient. A single step is better than a faster change because it would address the need without harming security in the future.   To the second case merged mining with the new cryptocurrency would be the rational change: Making the difficulty algorithm faster would only cause deeper highly disruptive oscillations as hash power moved between chains.

  Thanks for that, though I'd like to believe I put more effort into than just waving my arms about. Well, I may not understand it from the perspective you seem to proport to, I did state that shortening the number of blocks to recalc would cause more problems. I have also offered up atleast a vague idea about adding in another adjustment based on the most recent blocks.

  The very last part of your post is what I was hoping to see more of. The discussion that was intended in the thread, or atleast I thought, as to what the implications are of 'changing the formula' or 'not chaning it'...

I'm interested in knowing what you expect to happen to cause the hash rate to fall faster than the current adjustment can keep up with it which won't in itself present material problems which dwarf transactions being slow while the difficulty catches up?

The only things I can think of that would cause massive super-fast hash-rate losses — things like major governments outlawing mining without warning— are both highly unlikely and devastatingly terribly completely independent of the difficulty calculation.

I would have previously bought a theory that if the exchange rate fell to the point where many GPU miners were unprofitable over electrical cost that hash rate would plummet, but this appears to be disproven by the evidence— it seems that there is enough mixture of energy costs (e.g. people with higher costs that shut off first), people whos energy usage is offset by being able to use the surplus heat, people with FPGAs and free power (partial or total energy cost insensitivity), and people with faith that bitcoin will gain value in the future that downward changes caused by exchange vs energy cost are fairly slow in practice.  It's gone down, of course, but not so quickly that its problematic.

In the meantime,  alt-chains with modified difficulty algorithms have been exploited thoroughly by attackers who used overpowering attacks which were assisted by the difficulty being too easily ramped.

So perhaps this was just the wrong time to have this debate: At the moment I don't think the evidence for changing the difficulty has ever been weaker and I don't think the evidence against it has ever been stronger.

  Thank you for taking the time to better clearify what you meant by 'waving arms'  :)


  You make a good case and hopefully this holds true for a while.

  Can we still debate what some of the better methods for changing the difficulty algo could be IF there were to be a need to?  :-*

   Cheers,
      Derek

Sorry, — you don't need my permission. I apologize if I was too heavy handed and made it seem that I was suggesting otherwise.  I'll try to confine any further comments I make to the thread to the assumption that a change would be made, and simply to questions about the best/worst ways to change it.

:)