|
Title: are there any actual stats on chain reorgs, by depth? Post by: jl777 on March 18, 2016, 08:12:21 PM I know the odds for a deep reorg are small, but I was hoping some website is keeping stats on how often the small reorgs are happening. I guess it would be relative to a specific node, unless there is some internal bitcoind stats that can be queried across all peers.
I am assuming that 10 block deep reorg is very rare, but wanted to get statistical confirmation. It would incur a several minute recalculation cost to have to regenerate the data for all the blocks after the reorg, so dont want to set this at something that happens every day. But no sense to set it so it would be expected to happen once a year. The shorter the delay, the faster things will work for realtime queries as it minimizes the amount of data that needs to be scanned that is not in the parallel format. Though, now I think about it, maybe I can make a special case for smaller bundle at the end. Regardless, of the realtime calc speed, I currently delay creation of a bundle until 10 blocks after the last block of that bundle and wanted to know what is the probability that it will need to be regenerated base on real world stats James Title: Re: are there any actual stats on chain reorgs, by depth? Post by: achow101 on March 18, 2016, 09:23:38 PM Edit: I stand corrected: The official Bitcoin client has "getchaintips" command Title: Re: are there any actual stats on chain reorgs, by depth? Post by: 2112 on March 18, 2016, 09:44:36 PM The official Bitcoin client has "getchaintips" command that gives you the output like this:
Code: [ Title: Re: are there any actual stats on chain reorgs, by depth? Post by: jl777 on March 18, 2016, 09:54:43 PM The official Bitcoin client has "getchaintips" command that gives you the output like this: very nice! didnt know it listed all reorgs.Code: [ so if I read that right since block 395487, the max depth reorg has been: 1 if so, does that mean a node that has been constantly running for 1 year would be needed to run getchaintips, to get the entire year's history of reorgs? It seems 10 blocks at current hashrate would be very rare James Title: Re: are there any actual stats on chain reorgs, by depth? Post by: 2112 on March 18, 2016, 10:03:25 PM very nice! didnt know it listed all reorgs. More or less. But remember that the reorganizations aren't really globally identical. You'll really want several long-running nodes on several ISPs to have meaningful statistics.so if I read that right since block 395487, the max depth reorg has been: 1 if so, does that mean a node that has been constantly running for 1 year would be needed to run getchaintips, to get the entire year's history of reorgs? It seems 10 blocks at current hashrate would be very rare James Also, testnet has way more and longer reorgs, very good for testing the handling of the reorgs. Title: Re: are there any actual stats on chain reorgs, by depth? Post by: jl777 on March 18, 2016, 10:21:03 PM very nice! didnt know it listed all reorgs. More or less. But remember that the reorganizations aren't really globally identical. You'll really want several long-running nodes on several ISPs to have meaningful statistics.so if I read that right since block 395487, the max depth reorg has been: 1 if so, does that mean a node that has been constantly running for 1 year would be needed to run getchaintips, to get the entire year's history of reorgs? It seems 10 blocks at current hashrate would be very rare James Also, testnet has way more and longer reorgs, very good for testing the handling of the reorgs. For testing I can just manually force a reorg, I just wanted to calibrate things with the right initial offset using current network characteristics 3 seems too low, but 10 should be at most a monthly occurance, which is fine for a several minute recalc. But if anybody has more extensive data or much worse reorgs, please post! it could be 2112's node is special Title: Re: are there any actual stats on chain reorgs, by depth? Post by: 2112 on March 18, 2016, 10:34:57 PM it could be 2112's node is special The results I posted were from a portable laptop that had seen multiple ISPs but runs only occasionally. On the other laptop that had seen only one ISP but runs more often I had 165 branches instead of 15.You'll need way more nodes to have meaningful statistics. Title: Re: are there any actual stats on chain reorgs, by depth? Post by: jl777 on March 18, 2016, 11:00:54 PM it could be 2112's node is special The results I posted were from a portable laptop that had seen multiple ISPs but runs only occasionally. On the other laptop that had seen only one ISP but runs more often I had 165 branches instead of 15.You'll need way more nodes to have meaningful statistics. Title: Re: are there any actual stats on chain reorgs, by depth? Post by: 2112 on March 18, 2016, 11:16:48 PM what was max depth with the one of 165 branches? Code: {Title: Re: are there any actual stats on chain reorgs, by depth? Post by: jl777 on March 18, 2016, 11:26:15 PM what was max depth with the one of 165 branches? Code: {It sure seems like a good value for a backstop. The cost being a few minutes recalc and offline time if a bigger reorg happens Title: Re: are there any actual stats on chain reorgs, by depth? Post by: Syke on March 18, 2016, 11:57:23 PM It seems 10 blocks at current hashrate would be very rare Extremely! This is closely related to the odds of pulling off a double-spend 10 blocks deep. D&T explained nicedly at https://bitcoin.stackexchange.com/questions/1170/why-is-6-the-number-of-confirms-that-is-considered-secure. The worst was the overflow bug. That would have been around 53. https://en.bitcoin.it/wiki/Value_overflow_incident My node only goes back to 2012, so I missed out on the overflow fork. But I have the next big one. My node saw the .7/.8 fork at a reorg of 30: Code: {Feel free to analyze my data at http://pastebin.com/LZxst5vD. Title: Re: are there any actual stats on chain reorgs, by depth? Post by: jl777 on March 19, 2016, 12:26:03 AM It seems 10 blocks at current hashrate would be very rare Extremely! This is closely related to the odds of pulling off a double-spend 10 blocks deep. D&T explained nicedly at https://bitcoin.stackexchange.com/questions/1170/why-is-6-the-number-of-confirms-that-is-considered-secure. The worst was the overflow bug. That would have been around 53. https://en.bitcoin.it/wiki/Value_overflow_incident My node only goes back to 2012, so I missed out on the overflow fork. But I have the next big one. My node saw the .7/.8 fork at a reorg of 30: Code: {Feel free to analyze my data at http://pastebin.com/LZxst5vD. and the hashrate was a small fraction back around block 225K. Not sure, but I think it is harder to reorg now vs then? If only since we now have ASIC so the tech leaps in hashrate from CPU to GPU to FPGA to ASIC is already done. It seems vast majority are just 1 deep. around 10 2 deep just a few 3 deep the 6 block reorg seems to be an outlier along with the 30. All the talk about eventual consensus, but with bitcoin it looks like 99.999% of the time, within 3 blocks it is in consensus unless your node is special. with propagation of around 15 seconds though, it seems other than at the edge of one block, it would take three fast blocks to require a 3 block reorg. Maybe that 6 block reorg was an actual attack? it seems way out of bounds and both your node and 2112's has it so it is unlikely that is it some local connectivity issue since nowadays the miners are all probably doing their best to push the new blocks as fast as possible, that combined with the nextgen ASIC probably not going to be much more than 2x increase, maybe as little as 5 blocks would be enough, but there isnt much difference between 5 and 10, so probably best to setback 10 blocks Title: Re: are there any actual stats on chain reorgs, by depth? Post by: Syke on March 19, 2016, 01:43:56 AM unless your node is special. Standard PC running the default Linux client with basic cable modem connectivity. It's online probably 95% of the time, which would mean it could have missed a few reorgs. There could be an interesting story about that 6 block reorg. That does appear to be the biggest naturally occurring reorg. Title: Re: are there any actual stats on chain reorgs, by depth? Post by: jl777 on March 19, 2016, 02:02:49 AM unless your node is special. Standard PC running the default Linux client with basic cable modem connectivity. It's online probably 95% of the time, which would mean it could have missed a few reorgs. There could be an interesting story about that 6 block reorg. That does appear to be the biggest naturally occurring reorg. just having the second node confirm the 6 block reorg changes that from probable isolated node issue to a likely global reorg. and it wasnt that long ago, so the hashrate required would have been a lot. maybe it was during the DDOs attack? Title: Re: are there any actual stats on chain reorgs, by depth? Post by: monsterer on March 19, 2016, 10:53:44 AM and the hashrate was a small fraction back around block 225K. Not sure, but I think it is harder to reorg now vs then? It was the same then as it is now - back then, network power was less than it is now, so although difficulty was lower than it is now, blocks were still 10 minutes each. Title: Re: are there any actual stats on chain reorgs, by depth? Post by: jl777 on March 19, 2016, 06:30:54 PM and the hashrate was a small fraction back around block 225K. Not sure, but I think it is harder to reorg now vs then? It was the same then as it is now - back then, network power was less than it is now, so although difficulty was lower than it is now, blocks were still 10 minutes each. My concern is with the practical. When did the ASIC's come out? That is a nonlinear increase in hashrate per amount of capital. Moving forward is it likely to have the same hashrate improvements was was possible a couple years ago. Not sure on the state of ASIC geometries, but if already the smallest practical geometry is being used, then it cant really get much more cost effective. Larger, yes, but with proportional cost for the silicon. Cleverer, yes, but possibly most of the low hanging fruit is already done Not sure the state of mining at block 225K, but given those factors it is possible that things are still exactly the same, but I am skeptical about that. What are the currently envisioned improvements in hashrate efficiency? Actually instead of just guessing, lets look at https://blockchain.info/charts/hash-rate?timespan=all&showDataPoints=false&daysAverageString=1&show_header=true&scale=0&address= the alltime chart shows a definite slowdown of hashrate increase, so that does not seem the same to me. Granted from one block to the next, not much changes, but the discontinuity of new tech is what would create the temporary ability for a bigger reorg. 01/01/2010 18:15:05,0.006997468320659102 01/01/2011 18:15:05,128.16206770778084 18315x 2010 01/01/2012 18:15:05,8591.40080069935 67x 2011 01/01/2013 18:15:05,24291.440498714688 2.82x 2012 01/01/2014 18:15:05,11740802.089632047 483x 2013 01/01/2015 18:15:05,335365290.0920246 28.56x 2014 01/01/2016 18:15:05,697129166.4058079 2x 2015 As things went from CPU -> GPU -> FPGA -> ASIC and as market price is changing dramatically we see the change in hashrate vary from 18315x to 2x, so it does not seem safe to assume that things are the same. The 2014 gain was probably skewed by the bull market peak price bringing in a lot of new capital, but maybe that is around the same time ASIC made a big leap? The most likely way for an attack against bitcoin PoW seems to me for the creator of advanced tech to have a disproportionate hashrate per capital available, especially when efficiency gains are measured in the orders of magnitude. However, once the orders of magnitude technology gains are already done, it becomes a matter of much more "incremental" improvements of doubling per year. At least that is what I had guessed and the numbers indicate something similar to that. One way to look at this is back in 2013 what was the likelihood of a 500x hashrate increase? Was it really such a remote possibility? And what is the chance of a 500x hashrate increase now? Is there really any chance that next year we see 1,122,825 * 500 terahashes? or even 10x. Yes I know the difficulty adjusts, but that is every 2016 blocks, there is a window of time where a technical leap increases the chance of larger reorg, but as time passes I think this chance is less and less. James Title: Re: are there any actual stats on chain reorgs, by depth? Post by: 2112 on March 19, 2016, 06:54:04 PM Sure, theoretically, but I dont like to just assume that the theory is precisely followed by reality. Your analysis is bogus. Your are mistaking correlation with causation.My concern is with the practical. When did the ASIC's come out? That is a nonlinear increase in hashrate per amount of capital. Yes I know the difficulty adjusts, but that is every 2016 blocks, there is a window of time where a technical leap increases the chance of larger reorg, but as time passes I think this chance is less and less. The long term changes in the orphan rates are caused not by changes in hash rate but by changes in the number and connectivity of the mining pools. The number of pools decreased and the connectivity of the remaining pools improved. In particular Matt Corallo's Relay Network caused decrease in the orphan races. The probability of reorgs probably going to increase in the future due to the bugs in the competing implementations, due to DDoS attacks on the mining pool's infrastructure and due to Matt Corallo finally abandoning the maintenance of his Relay Network. The difficulty increase is just an innocent bystander. Title: Re: are there any actual stats on chain reorgs, by depth? Post by: jl777 on March 19, 2016, 08:09:33 PM Sure, theoretically, but I dont like to just assume that the theory is precisely followed by reality. Your analysis is bogus. Your are mistaking correlation with causation.My concern is with the practical. When did the ASIC's come out? That is a nonlinear increase in hashrate per amount of capital. Yes I know the difficulty adjusts, but that is every 2016 blocks, there is a window of time where a technical leap increases the chance of larger reorg, but as time passes I think this chance is less and less. The long term changes in the orphan rates are caused not by changes in hash rate but by changes in the number and connectivity of the mining pools. The number of pools decreased and the connectivity of the remaining pools improved. In particular Matt Corallo's Relay Network caused decrease in the orphan races. The probability of reorgs probably going to increase in the future due to the bugs in the competing implementations, due to DDoS attacks on the mining pool's infrastructure and due to Matt Corallo finally abandoning the maintenance of his Relay Network. The difficulty increase is just an innocent bystander. I specifically limited my question to issues unrelated to software versions. just trying to understand things one piece at a time. Now monsterer is saying it is always the same, you say it is probably going to increase. But I agree with you about the non PoW factors most likely dominating and since I am always wrong, does that mean you are wrong too? James Title: Re: are there any actual stats on chain reorgs, by depth? Post by: 2112 on March 19, 2016, 08:31:24 PM Why are you always using such harsh words? Maybe I am an idiot and you are super genius that is always right about everything. You seem to have an excellent command of English for a non-native speaker.I specifically limited my question to issues unrelated to software versions. just trying to understand things one piece at a time. Now monsterer is saying it is always the same, you say it is probably going to increase. But I agree with you about the non PoW factors most likely dominating and since I am always wrong, does that mean you are wrong too? At the same time you seem to lack basic scientific education at the level of high school, where typically the "correlation vs. causation" is explained. In a previous thread you've exhibited 0/10 knowledge of basics of DBMS. I have no idea where you've received your education (home schooling? religious school?) but I haven't used any harsh words. I've used standard sentences that would be said by a teacher to a completely unprepared student face-to-face in any normal school. I'm not going to even delve into further scholastic experience like participating in competitive sports or military training. Title: Re: are there any actual stats on chain reorgs, by depth? Post by: jl777 on March 19, 2016, 08:53:30 PM Why are you always using such harsh words? Maybe I am an idiot and you are super genius that is always right about everything. You seem to have an excellent command of English for a non-native speaker.I specifically limited my question to issues unrelated to software versions. just trying to understand things one piece at a time. Now monsterer is saying it is always the same, you say it is probably going to increase. But I agree with you about the non PoW factors most likely dominating and since I am always wrong, does that mean you are wrong too? At the same time you seem to lack basic scientific education at the level of high school, where typically the "correlation vs. causation" is explained. In a previous thread you've exhibited 0/10 knowledge of basics of DBMS. I have no idea where you've received your education (home schooling? religious school?) but I haven't used any harsh words. I've used standard sentences that would be said by a teacher to a completely unprepared student face-to-face in any normal school. I'm not going to even delve into further scholastic experience like participating in competitive group sports or military training. Rather than disprove how my analysis that 10 blocks is sufficient, you state that my analysis is bogus. So I should set it to 100 blocks? 1000 blocks? Some random number? That is your recommendation? Or maybe I should go to DBMS school since apparently without a DBMS Phd nothing I write has any validity. What does my education have to do with the analysis that 10 blocks appears to be sufficient? It might make others question your objectivity if you keep taking every chance to try to make me look stupid. Or you dont mind if people just categorize you as some type of troll? I would think a smart person would want his point of view appreciated and not just dismissed as troll blather. I appreciated your post about the getchaintips RPC, that was useful. Maybe you can stick to making useful posts? Like math based analysis, which is what I tried to do. Did I make any math errors? If the statistical model is dominated in the real world by non-PoW factors, then that is certainly an important issue and might need to be factored in. If so, what should the setback be? would 1 year be enough time to be sure? James Title: Re: are there any actual stats on chain reorgs, by depth? Post by: 2112 on March 19, 2016, 09:26:25 PM I dropped out of kindergarten, I couldnt handle being shamed and put in the corner when I asked questions I wasnt supposed to. Take mountain climbing as a hobby. Your main adversary will be then impersonal gravity. But you'll learn to deal with reality and how to overcome obstacles. Or you'll get yourself killed.Or you dont mind if people just categorize you as some type of troll? I would think a smart person would want his point of view appreciated and not just dismissed as troll blather. I'm actually proud of getting called troll by now know fraudsters like DeathAndTaxes or shtylman. On you the question is still open: are you leaning more towards harmless crackpottery or towards willful fraud? We'll see.I appreciated your post about the getchaintips RPC, that was useful. Maybe you can stick to making useful posts? Like math based analysis, which is what I tried to do. Did I make any math errors? This is public forum. I make post useful to the global audience of the readers, even if they aren't useful to you or any particular poster of any thread.Your math error is called: GIGO https://en.wikipedia.org/wiki/Garbage_in%2C_garbage_out . The mistake you are making is common enough that there's lots of educational texts regarding the "correlation vs. causation" issue. When I was in school profs used to refer to an excellent joke paper written by some Scandinavian scientists about presence of storks and childbirth rates in Scandinavia. I'm not sure if it was ever translated to English. Edit: More seriously: many law schools offer "personal development" classes on how to deal with adversity in a public courtroom. Apparently it is a common problem for even excellent students that were home schooled or went to religious schools. Title: Re: are there any actual stats on chain reorgs, by depth? Post by: jl777 on March 19, 2016, 09:36:28 PM I dropped out of kindergarten, I couldnt handle being shamed and put in the corner when I asked questions I wasnt supposed to. Take mountain climbing as a hobby. Your main adversary will be then impersonal gravity. But you'll learn to deal with reality and how to overcome obstacles. Or you'll get yourself killed.Or you dont mind if people just categorize you as some type of troll? I would think a smart person would want his point of view appreciated and not just dismissed as troll blather. I'm actually proud of getting called troll by now know fraudsters like DeathAndTaxes or shtylman. On you the question is still open: are you leaning more towards harmless crackpottery or towards willful fraud? We'll see.I appreciated your post about the getchaintips RPC, that was useful. Maybe you can stick to making useful posts? Like math based analysis, which is what I tried to do. Did I make any math errors? This is public forum. I make post useful to the global audience of the readers, even if they aren't useful to you or any particular poster of any thread.Your math error is called: GIGO https://en.wikipedia.org/wiki/Garbage_in%2C_garbage_out . The mistake you are making is common enough that there's lots of educational texts regarding the "correlation vs. causation" issue. When I was in school profs used to refer to an excellent joke paper written by some Scandinavian scientists about presence of storks and childbirth rates in Scandinavia. I'm not sure if it was ever translated to English. so we just need to wait stork number of blocks and that will be the optimal for mountain climbing. I guess you think satoshi was an idiot too, as he said something about 10 blocks. Or was he committing fraud or whatever this crockpottery is. ad hominem attacks vs. historical results with some actual math And what exactly is this fraud you imply I am conducting? That using a hard coded lookup table (generated from p2p network and validated) instead of a DB is more efficient. Is that my supposed fraud? James Title: Re: are there any actual stats on chain reorgs, by depth? Post by: 2112 on March 19, 2016, 10:16:47 PM Thanks! This really helps me find the math error in my analysis. I was missing the storks. I don't know. What I do know is that fraudster and gangsters (more generally common criminals) have exceptional sensitivity to being given "no respect" (cue Marlon Brando in The Godfather). Of course, correlation is not causation. But see how DeathAndTaxes reacted to the mention of money orders in 2012. I think that even he didn't know at that time what scam he's going to pull of years later.so we just need to wait stork number of blocks and that will be the optimal for mountain climbing. I guess you think satoshi was an idiot too, as he said something about 10 blocks. Or was he committing fraud or whatever this crockpottery is. ad hominem attacks vs. historical results with some actual math And what exactly is this fraud you imply I am conducting? That using a hard coded lookup table (generated from p2p network and validated) instead of a DB is more efficient. Is that my supposed fraud? James https://bitcointalk.org/index.php?topic=93655.msg1036760#msg1036760 Your reaction to the mention of high-school-level science curriculum is very similar. Again, time will show. Title: Re: are there any actual stats on chain reorgs, by depth? Post by: jl777 on March 19, 2016, 11:04:51 PM Thanks! This really helps me find the math error in my analysis. I was missing the storks. I don't know. What I do know is that fraudster and gangsters (more generally common criminals) have exceptional sensitivity to being given "no respect" (cue Marlon Brando in The Godfather). Of course, correlation is not causation. But see how DeathAndTaxes reacted to the mention of money orders in 2012. I think that even he didn't know at that time what scam he's going to pull of years later.so we just need to wait stork number of blocks and that will be the optimal for mountain climbing. I guess you think satoshi was an idiot too, as he said something about 10 blocks. Or was he committing fraud or whatever this crockpottery is. ad hominem attacks vs. historical results with some actual math And what exactly is this fraud you imply I am conducting? That using a hard coded lookup table (generated from p2p network and validated) instead of a DB is more efficient. Is that my supposed fraud? James https://bitcointalk.org/index.php?topic=93655.msg1036760#msg1036760 Your reaction to the mention of high-school-level science curriculum is very similar. Again, time will show. Like: you wake up in the morning. criminals wake up in the morning. Therefore you are a criminal. or: you breathe air. criminals breathe air. Therefore you are a criminal. what is the name for this logic, I dont know about it and want to find out more. I know about transitive https://en.wikipedia.org/wiki/Transitive_relation when A -> B and B -> C that A -> C, but the reverse, this is quite novel. I guess if people operated as strict field elements with specific properties of reverse transivitism(?), maybe there is some basis for this, but last I checked I wasnt a finite field element. Are you? What is a bit frustrating is that you mix in some highly intelligent stuff with total nonsense, but I am just supposed to accept it all. otherwise I am like a fraudster or mobster? I dont ask for any respect that I dont deserve. If I make a logic error or math error, then point it out. make fun that I am a bad calculator, fine. But this nonsense attack by implication that is supported by absurd "logic", well maybe you are the one that needs to go to school? Maybe I need to be your teacher about some things. If you cant find any flaws with the logic or math, then you could just say that or be silent about it. James P.S. I have done most of my learning by using google to find relevant pages, so there are likely gaps in my knowledge as compared to someone who spent many years learning all the vital things that are learned in schools. Title: Re: are there any actual stats on chain reorgs, by depth? Post by: Syke on March 20, 2016, 01:48:17 AM Regardless, of the realtime calc speed, I currently delay creation of a bundle until 10 blocks after the last block of that bundle and wanted to know what is the probability that it will need to be regenerated base on real world stats As far as my node data, there have been only 2 instances of a reorg over 10 blocks, both due to significant unique events. 1 instance of a normal 6-block reorg. 2 instances of a 3 block reorg. 8 instances of a 2 block reorg. And lots of 1 block reorgs. 4 or more block reorgs are *extremely* rare. The standard client considers anything under 6 blocks to be unconfirmed. That seems like a good number to use for your purposes too. Title: Re: are there any actual stats on chain reorgs, by depth? Post by: jl777 on March 20, 2016, 03:49:29 AM Regardless, of the realtime calc speed, I currently delay creation of a bundle until 10 blocks after the last block of that bundle and wanted to know what is the probability that it will need to be regenerated base on real world stats As far as my node data, there have been only 2 instances of a reorg over 10 blocks, both due to significant unique events. 1 instance of a normal 6-block reorg. 2 instances of a 3 block reorg. 8 instances of a 2 block reorg. And lots of 1 block reorgs. 4 or more block reorgs are *extremely* rare. The standard client considers anything under 6 blocks to be unconfirmed. That seems like a good number to use for your purposes too. My plan is to have smaller bundles of 100 and 10, so at most there would be 19 mini-bundles of 100, 9 microbundles of 10 and 9 blocks for a total of 37 things to search through |