are there any actual stats on chain reorgs, by depth?

[jl777]

I know the odds for a deep reorg are small, but I was hoping some website is keeping stats on how often the small reorgs are happening. I guess it would be relative to a specific node, unless there is some internal bitcoind stats that can be queried across all peers.

I am assuming that 10 block deep reorg is very rare, but wanted to get statistical confirmation.

It would incur a several minute recalculation cost to have to regenerate the data for all the blocks after the reorg, so dont want to set this at something that happens every day. But no sense to set it so it would be expected to happen once a year.

The shorter the delay, the faster things will work for realtime queries as it minimizes the amount of data that needs to be scanned that is not in the parallel format. Though, now I think about it, maybe I can make a special case for smaller bundle at the end.

Regardless, of the realtime calc speed, I currently delay creation of a bundle until 10 blocks after the last block of that bundle and wanted to know what is the probability that it will need to be regenerated base on real world stats

James

[achow101]

Although I don't think Bitcoin Core has stats on reorgs, it does keep all of the valid blocks it receives. You could scan through the blk*.dat files and see if there were any chain reorgs.

Edit: I stand corrected:

The official Bitcoin client has "getchaintips" command

[2112]

The official Bitcoin client has "getchaintips" command that gives you the output like this:

Code:

[
    {
        "height" : 403263,
        "hash" : "00000000000000000210fee63635c72da771c6b136a63f20ff9d058ce778618d",
        "branchlen" : 0,
        "status" : "active"
    },
    {
        "height" : 402922,
        "hash" : "00000000000000000692c289cc4a4e9726018d6a030ad10c84f70adc2e920597",
        "branchlen" : 1,
        "status" : "valid-fork"
    },
    {
        "height" : 402751,
        "hash" : "00000000000000000180755378324c16847b2f49af428c763b8ff411d51d7767",
        "branchlen" : 1,
        "status" : "valid-headers"
    },
    {
        "height" : 402723,
        "hash" : "000000000000000000263e1722b119e2547a4167b2f9d5bfc89f0d9dc28e0e18",
        "branchlen" : 1,
        "status" : "valid-headers"
    },
    {
        "height" : 399879,
        "hash" : "0000000000000000005642fd14f92213a8ff68aa5071588f84dd2f1c61e93d0e",
        "branchlen" : 1,
        "status" : "valid-fork"
    },
    {
        "height" : 399706,
        "hash" : "000000000000000006aef254935a76c8b321519dcd86d4f5f5092f51ef374251",
        "branchlen" : 1,
        "status" : "valid-fork"
    },
    {
        "height" : 399332,
        "hash" : "000000000000000006972f107e48524ce3e01752c24a282beafc7b046f29f221",
        "branchlen" : 1,
        "status" : "valid-headers"
    },
    {
        "height" : 399247,
        "hash" : "00000000000000000228b92c2c2c8a91673c45f6aa10fb9acabfb587f73ccc81",
        "branchlen" : 1,
        "status" : "valid-fork"
    },
    {
        "height" : 399138,
        "hash" : "00000000000000000382da0f1d9cb0bffc6edbbdd53ba8f87877ffe11263100d",
        "branchlen" : 1,
        "status" : "valid-headers"
    },
    {
        "height" : 399098,
        "hash" : "000000000000000001bb9eac7a0684d2d0481b9a7fe74568ada10ada5734c857",
        "branchlen" : 1,
        "status" : "valid-fork"
    },
    {
        "height" : 398487,
        "hash" : "0000000000000000008fb4567640f05fddc79d28fcb0601a187b8e5cb46cb84c",
        "branchlen" : 1,
        "status" : "valid-headers"
    },
    {
        "height" : 398217,
        "hash" : "0000000000000000078f1ed2154a4db3d7cfa5615c584585c07e3f53ddba6aa8",
        "branchlen" : 1,
        "status" : "valid-fork"
    },
    {
        "height" : 398205,
        "hash" : "0000000000000000067886752a9524c8043341b5adc4100fbe11e7e45af1020d",
        "branchlen" : 1,
        "status" : "valid-headers"
    },
    {
        "height" : 397486,
        "hash" : "00000000000000000158e2c73ded7c24f0087514d646ae59ff017347bfff4c15",
        "branchlen" : 1,
        "status" : "valid-fork"
    },
    {
        "height" : 395487,
        "hash" : "000000000000000002782e988a2da3aa252752f3accae8624242527ca99de06e",
        "branchlen" : 1,
        "status" : "headers-only"
    }
]

[jl777]

The official Bitcoin client has "getchaintips" command that gives you the output like this:

Code:

[
    {
        "height" : 403263,
        "hash" : "00000000000000000210fee63635c72da771c6b136a63f20ff9d058ce778618d",
        "branchlen" : 0,
        "status" : "active"
    },
    {
        "height" : 402922,
        "hash" : "00000000000000000692c289cc4a4e9726018d6a030ad10c84f70adc2e920597",
        "branchlen" : 1,
        "status" : "valid-fork"
    },
    {
        "height" : 402751,
        "hash" : "00000000000000000180755378324c16847b2f49af428c763b8ff411d51d7767",
        "branchlen" : 1,
        "status" : "valid-headers"
    },
    {
        "height" : 402723,
        "hash" : "000000000000000000263e1722b119e2547a4167b2f9d5bfc89f0d9dc28e0e18",
        "branchlen" : 1,
        "status" : "valid-headers"
    },
    {
        "height" : 399879,
        "hash" : "0000000000000000005642fd14f92213a8ff68aa5071588f84dd2f1c61e93d0e",
        "branchlen" : 1,
        "status" : "valid-fork"
    },
    {
        "height" : 399706,
        "hash" : "000000000000000006aef254935a76c8b321519dcd86d4f5f5092f51ef374251",
        "branchlen" : 1,
        "status" : "valid-fork"
    },
    {
        "height" : 399332,
        "hash" : "000000000000000006972f107e48524ce3e01752c24a282beafc7b046f29f221",
        "branchlen" : 1,
        "status" : "valid-headers"
    },
    {
        "height" : 399247,
        "hash" : "00000000000000000228b92c2c2c8a91673c45f6aa10fb9acabfb587f73ccc81",
        "branchlen" : 1,
        "status" : "valid-fork"
    },
    {
        "height" : 399138,
        "hash" : "00000000000000000382da0f1d9cb0bffc6edbbdd53ba8f87877ffe11263100d",
        "branchlen" : 1,
        "status" : "valid-headers"
    },
    {
        "height" : 399098,
        "hash" : "000000000000000001bb9eac7a0684d2d0481b9a7fe74568ada10ada5734c857",
        "branchlen" : 1,
        "status" : "valid-fork"
    },
    {
        "height" : 398487,
        "hash" : "0000000000000000008fb4567640f05fddc79d28fcb0601a187b8e5cb46cb84c",
        "branchlen" : 1,
        "status" : "valid-headers"
    },
    {
        "height" : 398217,
        "hash" : "0000000000000000078f1ed2154a4db3d7cfa5615c584585c07e3f53ddba6aa8",
        "branchlen" : 1,
        "status" : "valid-fork"
    },
    {
        "height" : 398205,
        "hash" : "0000000000000000067886752a9524c8043341b5adc4100fbe11e7e45af1020d",
        "branchlen" : 1,
        "status" : "valid-headers"
    },
    {
        "height" : 397486,
        "hash" : "00000000000000000158e2c73ded7c24f0087514d646ae59ff017347bfff4c15",
        "branchlen" : 1,
        "status" : "valid-fork"
    },
    {
        "height" : 395487,
        "hash" : "000000000000000002782e988a2da3aa252752f3accae8624242527ca99de06e",
        "branchlen" : 1,
        "status" : "headers-only"
    }
]

very nice! didnt know it listed all reorgs.
so if I read that right since block 395487, the max depth reorg has been: 1

if so, does that mean a node that has been constantly running for 1 year would be needed to run getchaintips, to get the entire year's history of reorgs?

It seems 10 blocks at current hashrate would be very rare

James

[2112]

very nice! didnt know it listed all reorgs.
so if I read that right since block 395487, the max depth reorg has been: 1

if so, does that mean a node that has been constantly running for 1 year would be needed to run getchaintips, to get the entire year's history of reorgs?

It seems 10 blocks at current hashrate would be very rare

James

More or less. But remember that the reorganizations aren't really globally identical. You'll really want several long-running nodes on several ISPs to have meaningful statistics.

Also, testnet has way more and longer reorgs, very good for testing the handling of the reorgs.

[jl777]

very nice! didnt know it listed all reorgs.
so if I read that right since block 395487, the max depth reorg has been: 1

if so, does that mean a node that has been constantly running for 1 year would be needed to run getchaintips, to get the entire year's history of reorgs?

It seems 10 blocks at current hashrate would be very rare

James

More or less. But remember that the reorganizations aren't really globally identical. You'll really want several long-running nodes on several ISPs to have meaningful statistics.

Also, testnet has way more and longer reorgs, very good for testing the handling of the reorgs.

reorgs are totally dependent on each node, yes. but if your node is seeing max reorg depth of 1, i would think any node that is actually connected to the internet and not being flooded, would not be seeing reorgs of 10 blocks

For testing I can just manually force a reorg, I just wanted to calibrate things with the right initial offset using current network characteristics

3 seems too low, but 10 should be at most a monthly occurance, which is fine for a several minute recalc. But if anybody has more extensive data or much worse reorgs, please post!

it could be 2112's node is special

[2112]

it could be 2112's node is special

The results I posted were from a portable laptop that had seen multiple ISPs but runs only occasionally. On the other laptop that had seen only one ISP but runs more often I had 165 branches instead of 15.

You'll need way more nodes to have meaningful statistics.

[jl777]

it could be 2112's node is special

The results I posted were from a portable laptop that had seen multiple ISPs but runs only occasionally. On the other laptop that had seen only one ISP but runs more often I had 165 branches instead of 15.

You'll need way more nodes to have meaningful statistics.

what was max depth with the one of 165 branches?

[2112]

what was max depth with the one of 165 branches?

Code:

    {
        "height" : 363736,
        "hash" : "000000000000000013fe26675faa8f7dccd55ce5485bb6d0373fa66345901436",
        "branchlen" : 6,
        "status" : "valid-fork"
    },

[jl777]

what was max depth with the one of 165 branches?

Code:

    {
        "height" : 363736,
        "hash" : "000000000000000013fe26675faa8f7dccd55ce5485bb6d0373fa66345901436",
        "branchlen" : 6,
        "status" : "valid-fork"
    },

Has anybody seen a 10 block reorg in the last year?

It sure seems like a good value for a backstop. The cost being a few minutes recalc and offline time if a bigger reorg happens

[Syke]

It seems 10 blocks at current hashrate would be very rare

Extremely! This is closely related to the odds of pulling off a double-spend 10 blocks deep. D&T explained nicedly at https://bitcoin.stackexchange.com/questions/1170/why-is-6-the-number-of-confirms-that-is-considered-secure.

The worst was the overflow bug. That would have been around 53. https://en.bitcoin.it/wiki/Value_overflow_incident

My node only goes back to 2012, so I missed out on the overflow fork. But I have the next big one. My node saw the .7/.8 fork at a reorg of 30:

Code:

  {
    "height": 225459,
    "hash": "000000000000036e459f26b1dc02683c9beeb061eedf18689995143354467245",
    "branchlen": 30,
    "status": "valid-headers"
  }, 

Feel free to analyze my data at http://pastebin.com/LZxst5vD.

[jl777]

It seems 10 blocks at current hashrate would be very rare

Extremely! This is closely related to the odds of pulling off a double-spend 10 blocks deep. D&T explained nicedly at https://bitcoin.stackexchange.com/questions/1170/why-is-6-the-number-of-confirms-that-is-considered-secure.

The worst was the overflow bug. That would have been around 53. https://en.bitcoin.it/wiki/Value_overflow_incident

My node only goes back to 2012, so I missed out on the overflow fork. But I have the next big one. My node saw the .7/.8 fork at a reorg of 30:

Code:

  {
    "height": 225459,
    "hash": "000000000000036e459f26b1dc02683c9beeb061eedf18689995143354467245",
    "branchlen": 30,
    "status": "valid-headers"
  }, 

Feel free to analyze my data at http://pastebin.com/LZxst5vD.

Fantastic!! thanks for this invaluable history, 2012 is quite a ways back. Also reorgs from bugs/new versions are different from "normal" reorgs caused by propagation

and the hashrate was a small fraction back around block 225K. Not sure, but I think it is harder to reorg now vs then? If only since we now have ASIC so the tech leaps in hashrate from CPU to GPU to FPGA to ASIC is already done.

It seems vast majority are just 1 deep.
around 10 2 deep
just a few 3 deep

the 6 block reorg seems to be an outlier along with the 30. All the talk about eventual consensus, but with bitcoin it looks like 99.999% of the time, within 3 blocks it is in consensus

unless your node is special.

with propagation of around 15 seconds though, it seems other than at the edge of one block, it would take three fast blocks to require a 3 block reorg. Maybe that 6 block reorg was an actual attack? it seems way out of bounds and both your node and 2112's has it so it is unlikely that is it some local connectivity issue

since nowadays the miners are all probably doing their best to push the new blocks as fast as possible, that combined with the nextgen ASIC probably not going to be much more than 2x increase, maybe as little as 5 blocks would be enough, but there isnt much difference between 5 and 10, so probably best to setback 10 blocks

[Syke]

unless your node is special.

Standard PC running the default Linux client with basic cable modem connectivity. It's online probably 95% of the time, which would mean it could have missed a few reorgs. There could be an interesting story about that 6 block reorg. That does appear to be the biggest naturally occurring reorg.

[jl777]

unless your node is special.

Standard PC running the default Linux client with basic cable modem connectivity. It's online probably 95% of the time, which would mean it could have missed a few reorgs. There could be an interesting story about that 6 block reorg. That does appear to be the biggest naturally occurring reorg.

hopefully a few more long time nodes can post and we can get a much clearer idea.

just having the second node confirm the 6 block reorg changes that from probable isolated node issue to a likely global reorg. and it wasnt that long ago, so the hashrate required would have been a lot. maybe it was during the DDOs attack?

[monsterer]

and the hashrate was a small fraction back around block 225K. Not sure, but I think it is harder to reorg now vs then?

It was the same then as it is now - back then, network power was less than it is now, so although difficulty was lower than it is now, blocks were still 10 minutes each.

[jl777]

and the hashrate was a small fraction back around block 225K. Not sure, but I think it is harder to reorg now vs then?

It was the same then as it is now - back then, network power was less than it is now, so although difficulty was lower than it is now, blocks were still 10 minutes each.

Sure, theoretically, but I dont like to just assume that the theory is precisely followed by reality.

My concern is with the practical. When did the ASIC's come out? That is a nonlinear increase in hashrate per amount of capital.

Moving forward is it likely to have the same hashrate improvements was was possible a couple years ago. Not sure on the state of ASIC geometries, but if already the smallest practical geometry is being used, then it cant really get much more cost effective. Larger, yes, but with proportional cost for the silicon. Cleverer, yes, but possibly most of the low hanging fruit is already done

Not sure the state of mining at block 225K, but given those factors it is possible that things are still exactly the same, but I am skeptical about that. What are the currently envisioned improvements in hashrate efficiency?

Actually instead of just guessing, lets look at https://blockchain.info/charts/hash-rate?timespan=all&showDataPoints=false&daysAverageString=1&show_header=true&scale=0&address=

the alltime chart shows a definite slowdown of hashrate increase, so that does not seem the same to me. Granted from one block to the next, not much changes, but the discontinuity of new tech is what would create the temporary ability for a bigger reorg.

01/01/2010 18:15:05,0.006997468320659102
01/01/2011 18:15:05,128.16206770778084      18315x 2010
01/01/2012 18:15:05,8591.40080069935          67x 2011
01/01/2013 18:15:05,24291.440498714688      2.82x 2012
01/01/2014 18:15:05,11740802.089632047      483x 2013
01/01/2015 18:15:05,335365290.0920246       28.56x 2014
01/01/2016 18:15:05,697129166.4058079       2x 2015

As things went from CPU -> GPU -> FPGA -> ASIC and as market price is changing dramatically we see the change in hashrate vary from 18315x to 2x, so it does not seem safe to assume that things are the same. The 2014 gain was probably skewed by the bull market peak price bringing in a lot of new capital, but maybe that is around the same time ASIC made a big leap?

The most likely way for an attack against bitcoin PoW seems to me for the creator of advanced tech to have a disproportionate hashrate per capital available, especially when efficiency gains are measured in the orders of magnitude. However, once the orders of magnitude technology gains are already done, it becomes a matter of much more "incremental" improvements of doubling per year.

At least that is what I had guessed and the numbers indicate something similar to that. One way to look at this is back in 2013 what was the likelihood of a 500x hashrate increase? Was it really such a remote possibility? And what is the chance of a 500x hashrate increase now?

Is there really any chance that next year we see 1,122,825 * 500 terahashes? or even 10x.
Yes I know the difficulty adjusts, but that is every 2016 blocks, there is a window of time where a technical leap increases the chance of larger reorg, but as time passes I think this chance is less and less.

James

[2112]

Sure, theoretically, but I dont like to just assume that the theory is precisely followed by reality.

My concern is with the practical. When did the ASIC's come out? That is a nonlinear increase in hashrate per amount of capital.

Yes I know the difficulty adjusts, but that is every 2016 blocks, there is a window of time where a technical leap increases the chance of larger reorg, but as time passes I think this chance is less and less.

Your analysis is bogus. Your are mistaking correlation with causation.

The long term changes in the orphan rates are caused not by changes in hash rate but by changes in the number and connectivity of the mining pools. The number of pools decreased and the connectivity of the remaining pools improved. In particular Matt Corallo's Relay Network caused decrease in the orphan races.

The probability of reorgs probably going to increase in the future due to the bugs in the competing implementations, due to DDoS attacks on the mining pool's infrastructure and due to Matt Corallo finally abandoning the maintenance of his Relay Network.

The difficulty increase is just an innocent bystander.

[jl777]

Sure, theoretically, but I dont like to just assume that the theory is precisely followed by reality.

My concern is with the practical. When did the ASIC's come out? That is a nonlinear increase in hashrate per amount of capital.

Yes I know the difficulty adjusts, but that is every 2016 blocks, there is a window of time where a technical leap increases the chance of larger reorg, but as time passes I think this chance is less and less.

Your analysis is bogus. Your are mistaking correlation with causation.

The long term changes in the orphan rates are caused not by changes in hash rate but by changes in the number and connectivity of the mining pools. The number of pools decreased and the connectivity of the remaining pools improved. In particular Matt Corallo's Relay Network caused decrease in the orphan races.

The probability of reorgs probably going to increase in the future due to the bugs in the competing implementations, due to DDoS attacks on the mining pool's infrastructure and due to Matt Corallo finally abandoning the maintenance of his Relay Network.

The difficulty increase is just an innocent bystander.

Why are you always using such harsh words? Maybe I am an idiot and you are super genius that is always right about everything.

I specifically limited my question to issues unrelated to software versions. just trying to understand things one piece at a time.

Now monsterer is saying it is always the same, you say it is probably going to increase. But I agree with you about the non PoW factors most likely dominating and since I am always wrong, does that mean you are wrong too?

James

[2112]

Why are you always using such harsh words? Maybe I am an idiot and you are super genius that is always right about everything.

I specifically limited my question to issues unrelated to software versions. just trying to understand things one piece at a time.

Now monsterer is saying it is always the same, you say it is probably going to increase. But I agree with you about the non PoW factors most likely dominating and since I am always wrong, does that mean you are wrong too?

You seem to have an excellent command of English for a non-native speaker.

At the same time you seem to lack basic scientific education at the level of high school, where typically the "correlation vs. causation" is explained.

In a previous thread you've exhibited 0/10 knowledge of basics of DBMS.

I have no idea where you've received your education (home schooling? religious school?) but I haven't used any harsh words. I've used standard sentences that would be said by a teacher to a completely unprepared student face-to-face in any normal school. I'm not going to even delve into further scholastic experience like participating in competitive sports or military training.

[jl777]

Why are you always using such harsh words? Maybe I am an idiot and you are super genius that is always right about everything.

I specifically limited my question to issues unrelated to software versions. just trying to understand things one piece at a time.

Now monsterer is saying it is always the same, you say it is probably going to increase. But I agree with you about the non PoW factors most likely dominating and since I am always wrong, does that mean you are wrong too?

You seem to have an excellent command of English for a non-native speaker.

At the same time you seem to lack basic scientific education at the level of high school, where typically the "correlation vs. causation" is explained.

In a previous thread you've exhibited 0/10 knowledge of basics of DBMS.

I have no idea where you've received your education (home schooling? religious school?) but I haven't used any harsh words. I've used standard sentences that would be said by a teacher to a completely unprepared student face-to-face in any normal school. I'm not going to even delve into further scholastic experience like participating in competitive group sports or military training.

I dropped out of kindergarten, I couldnt handle being shamed and put in the corner when I asked questions I wasnt supposed to.

Rather than disprove how my analysis that 10 blocks is sufficient, you state that my analysis is bogus. So I should set it to 100 blocks? 1000 blocks? Some random number? That is your recommendation? Or maybe I should go to DBMS school since apparently without a DBMS Phd nothing I write has any validity.

What does my education have to do with the analysis that 10 blocks appears to be sufficient? It might make others question your objectivity if you keep taking every chance to try to make me look stupid. Or you dont mind if people just categorize you as some type of troll? I would think a smart person would want his point of view appreciated and not just dismissed as troll blather.

I appreciated your post about the getchaintips RPC, that was useful. Maybe you can stick to making useful posts? Like math based analysis, which is what I tried to do. Did I make any math errors?

If the statistical model is dominated in the real world by non-PoW factors, then that is certainly an important issue and might need to be factored in. If so, what should the setback be? would 1 year be enough time to be sure?

James