Bitcoin Forum
September 30, 2022, 10:58:04 PM *
News: Latest Bitcoin Core release: 23.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Stale block rates  (Read 309 times)
buckyball (OP)
Newbie
*
Offline Offline

Activity: 4
Merit: 0


View Profile
July 04, 2020, 04:45:00 PM
 #1

Hello - does anybody know of any decent sites which track stale block rates for BTC vs other cryptocurrencies?

Thanks!
1664578684
Hero Member
*
Offline Offline

Posts: 1664578684

View Profile Personal Message (Offline)

Ignore
1664578684
Reply with quote  #2

1664578684
Report to moderator
1664578684
Hero Member
*
Offline Offline

Posts: 1664578684

View Profile Personal Message (Offline)

Ignore
1664578684
Reply with quote  #2

1664578684
Report to moderator
Each block is stacked on top of the previous one. Adding another block to the top makes all lower blocks more difficult to remove: there is more "weight" above each block. A transaction in a block 6 blocks deep (6 confirmations) will be very difficult to remove.
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
1664578684
Hero Member
*
Offline Offline

Posts: 1664578684

View Profile Personal Message (Offline)

Ignore
1664578684
Reply with quote  #2

1664578684
Report to moderator
1664578684
Hero Member
*
Offline Offline

Posts: 1664578684

View Profile Personal Message (Offline)

Ignore
1664578684
Reply with quote  #2

1664578684
Report to moderator
1664578684
Hero Member
*
Offline Offline

Posts: 1664578684

View Profile Personal Message (Offline)

Ignore
1664578684
Reply with quote  #2

1664578684
Report to moderator
ranochigo
Legendary
*
Offline Offline

Activity: 2744
Merit: 3355

@ me if you need my response


View Profile
July 04, 2020, 04:55:09 PM
 #2

Stale or orphan? You probably won't get reliable data on stale blocks since reference client doesn't usually accept blocks that are stale.

If two blocks are generated at the same time and a block is built ontop of one of them, then the other block gets orphaned. Its much more common to see these. Blockchain.com does have a page for Bitcoin.

Pmalek
Legendary
*
Offline Offline

Activity: 2170
Merit: 4844


Stop the FUD! https://bitcoincleanup.com


View Profile
July 05, 2020, 12:41:11 PM
 #3

I have also never come across any sites that keep statistics about stale blocks. As ranochigo pointed out, blockchain.com does keep a record of orphaned blocks and you can view those stats here: https://www.blockchain.com/charts/n-orphaned-blocks

o_e_l_e_o
Legendary
*
Offline Offline

Activity: 1792
Merit: 12318


Slava Ukraini


View Profile
July 05, 2020, 02:08:27 PM
Merited by Pmalek (1)
 #4

If two blocks are generated at the same time and a block is built ontop of one of them, then the other block gets orphaned. Its much more common to see these. Blockchain.com does have a page for Bitcoin.
We need to be clear on terminology here. What you are referring to here is best called a stale or extinct block - a block which was once part of the main chain, but a longer chain took over. It's a misnomer to call these blocks "orphan" because their parent blocks are fully known.

Orphan blocks were blocks which had no parent. These were seen locally by individual nodes which had not yet received details of their parent blocks. These no longer exist since "headers-first synchronization" was implemented in Bitcoin Core 0.10.0.

As ranochigo pointed out, blockchain.com does keep a record of orphaned blocks and you can view those stats here: https://www.blockchain.com/charts/n-orphaned-blocks
That page hasn't been kept up to date in a couple of years.

In terms of keeping track of actual stale blocks - using the definition of stale blocks I provided above - then the best resource I found to do so is here: https://forkmonitor.info/feeds/stale_candidates/btc.rss. They also have RSS feeds for both BCH and BSV - just change the three letters at the end up the link.

gmaxwell
Moderator
Legendary
*
expert
Offline Offline

Activity: 3752
Merit: 6861



View Profile
July 06, 2020, 02:28:54 AM
 #5

It's really hard to measure stale block rates right now.

Fibre and compact blocks have made propagation tremendously in the network, so now when a block is stale it's usually stale at the first or first couple hops.

As a result, the stale block doesn't propagate well (because it just makes it through zero to a few nodes before intersecting the wavefront of the earlier block).

You can think of it like this: stale blocks come from latency, latency arises from the mining process (miner devices, pool servers), from bitcoind, and from the network.  Advances in block propagation have really zeroed out the latter two classes. But mining process latency hasn't decreased (and in some places it likely has increased as miners have fewer stales to worry about), but the faster propagation means that your node will learn about all stales (including ones due to mining process delays) at a much lower rate.

If you want an even remotely useful figure you need to get the output from getchaintips rpc from a bunch of nodes spread all over the world.

Last time I collected stats I got data from a dozen nodes and I was still learning about more new stale blocks from the dozenth.

Really, there should be a P2P message added that allows nodes to announce the headers which are extremely close to the current tip for network monitoring purposes and which doesn't imply block availability like ordinary announcement does. ... but there isn't much activity in improving the Bitcoin protocol anymore, and improving stale monitoring would be one of the lowest priorities for that sort of work.
Cubic Earth Two
Newbie
*
Offline Offline

Activity: 2
Merit: 0


View Profile
July 07, 2020, 10:55:44 PM
 #6

... but there isn't much activity in improving the Bitcoin protocol anymore...

Huh Really?
gmaxwell
Moderator
Legendary
*
expert
Offline Offline

Activity: 3752
Merit: 6861



View Profile
July 09, 2020, 04:53:53 AM
 #7

Huh Really?
That is certainly my impression.
Cubic Earth Two
Newbie
*
Offline Offline

Activity: 2
Merit: 0


View Profile
July 09, 2020, 06:14:37 AM
 #8


I assumed there is more development activity now than ever before, but that lots of it is just occurring via non-public channels. You've seemed to have taken somewhat of a break yourself, but perhaps you as well are more active behind the scenes? And are you feeling that things should be being improved that aren't? Or that it's good that protocol development has slowed down?
gmaxwell
Moderator
Legendary
*
expert
Offline Offline

Activity: 3752
Merit: 6861



View Profile
July 09, 2020, 07:10:13 AM
 #9

I assumed there is more development activity now than ever before, but that lots of it is just occurring via non-public channels.
Not possible-- of course it's always possible for people to talk in private but things have to be made public to have an effect.

I don't think it's good but I don't think it's particularly bad either.  There are a lot of improvements that would be useful, but at the same time Bitcoin works okay without them, and it's better things not be done than done poorly or thoughtlessly.
harding
Jr. Member
*
Offline Offline

Activity: 50
Merit: 46


View Profile WWW
July 09, 2020, 05:37:39 PM
Merited by suchmoon (4)
 #10

[...] there isn't much activity in improving the Bitcoin protocol anymore [...]

This statement runs counter to my intuition, so I made a quick comparison.  First, I assumed we were talking about the P2P network protocol ("the Bitcoin protocol" could also refer to the consensus protocol, but this conversion seemed to be about networking).  Second, I looked at all merges to Bitcoin Core during the six months up to today and the same period five years earlier.  Completely subjectively, I made a list of all the commits I thought affected how Bitcoin Core interacted with the protocol in a non-trivial way.  Here's the list for the the past six months (2020):

Code:
$ git log --since=2020-01-09 --until=2020-07-09 --merges --oneline | wc -l
768

abdfd2d0e Merge #19219: Replace automatic bans with discouragement filter
8edfc1715 Merge #19204: p2p: Reduce inv traffic during IBD
28ce05d06 Merge #19260: p2p: disconnect peers that send filterclear + update existing filter msg disconnect logic
73407ff65 Merge #16939: p2p: Delay querying DNS seeds
7d32cce3e Merge #19010: net processing: Add support for getcfheaders
4479eb04d Merge #18960: indexes: Add compact block filter headers cache
c73bd004a Merge #18861: Do not answer GETDATA for to-be-announced tx
553bb3fc3 Merge #18962: net processing: Only send a getheaders for one block in an INV
e45fb7e0d Merge #18877: Serve cfcheckpt requests
7a5767423 Merge #18808: [net processing] Drop unknown types in getdata
0ef0d33f7 Merge #18038: P2P: Mempool tracks locally submitted transactions to improve wallet privacy
67dfd18f4 Merge #16902: O(1) OP_IF/NOTIF/ELSE/ENDIF script implementation
d104aa0ac Merge #17951: Use rolling bloom filter of recent block txs for AlreadyHave() check
01fc5891f Merge #16702: p2p: supplying and using asmap to improve IP bucketing in addrman

Here's the list for the same six-month period five years ago (2015).  Note, the merge messages back then didn't include a title, so I placed the subject from one of the commits in that merge in brackets:

Code:
$ git log --since=2015-01-09 --until=2015-07-09 --merges --oneline | wc -l
321

66e546577 Merge pull request #6310 [f581d3d banlist.dat: store banlist on disk]
41076aad0 Merge pull request #6124 [ffd75ad Enable CHECKLOCKTIMEVERIFY as a standard script verify flag]
e34a8acd9 Merge pull request #6233 [3e91433 Advance pindexLastCommonBlock for blocks in chainActive]
c1fb0e107 Merge pull request #6274 [02a6702 Add option `-alerts` to opt out of alert system]
3a1d3e8f5 Merge pull request #5985 [14d4eef Fix removing of orphan transactions]
8d9f0a606 Merge pull request #5927 [dce8360 Reduce checkpoints' effect on consensus.]
9d6060244 Merge pull request #5875 [9be0e68 Be stricter in processing unrequested blocks]
88a7ead5d Merge pull request #6172 [a1ba077 Ignore getheaders requests when not synced.]
9f7809f6c Merge pull request #5976 [8ba7f84 Reduce download timeouts as blocks arrive]
e9af4e65b Merge pull request #5947 [36cba8f Alert if it is very likely we are getting a bad chain]
7bf5d5efa Merge pull request #5918 [f7303f9 Use equivalent PoW for non-main-chain requests]
5048465fc Merge pull request #5662 [00dcaf4 Change download logic to allow calling getheaders/getdata on inbound peers]
71900b442 Merge pull request #6029 [a784f90 Cap nAttempts penalty at 8 and switch to pow instead of a division loop.]
27e8d224e Merge pull request #5945 [ad9e86d Keep mempool consistent during block-reorgs]
f7dea1cba Merge pull request #5941 [1d21ba2 Scale up addrman]
29fef0b90 Merge pull request #5360 [9c27379 Reduce fingerprinting through timestamps in 'addr' messages.]
c1b723c30 Merge pull request #5442 [dca799e Ignore getaddr messages on Outbound connections.]
51377c2db Merge pull request #5843 [ba04c4a Limit message sizes before transfer]
fcf646c9b Merge pull request #5286 [a930658 Change the default maximum OP_RETURN size to 80 bytes]
6ee87f9bc Merge pull request #5647 [3ff735c Increase block download timeout base from 10 to 20 minutes.]
41e6e4cab Merge pull request #5713 [5a47811 BIP66 changeover logic (Pieter Wuille)]
de8b9ab75 Merge pull request #5608 [9161303 Introduce 10 minute block download timeout]

So there were 22 (subjectively) notable merges in 2015 and 14 notable merges in 2020, a 35% decrease in the number of merges. In both cases, an overwhelming majority of the changes are tweaks to the existing logic mean to improve security, privacy, or reliability rather than more significant extensions of the protocol.  Looking at the titles of the merges, and again very subjectively, I think I'd consider the aggregate of the changes made in 2015 to be even more than 35% useful than the 2020 changes, so the gap between the years is arguably larger than suggested by just the numbers.

However, based on these data, I don't personally agree that "there isn't much activity in improving the Bitcoin protocol anymore."  The rate of change does seem slower, but I think it's far from stagnant.
gmaxwell
Moderator
Legendary
*
expert
Offline Offline

Activity: 3752
Merit: 6861



View Profile
July 10, 2020, 06:51:33 PM
Last edit: July 11, 2020, 03:02:03 AM by gmaxwell
Merited by ETFbitcoin (6)
 #11

However, based on these data, I don't personally agree that "there isn't much activity in improving the Bitcoin protocol anymore."  The rate of change does seem slower, but I think it's far from stagnant.

I think your post supports my position.

A change in the P2P protocol would require or at least enable a behaviour change by a remote peer. Only one of these PRs do that. (Ideally they'd enable new functionality, though not always)

Almost all those commits in the recent change you listed are implementation optimizations, local behaviour changes, or refactorings and don't change the P2P protocol-- many don't change the externally visible behaviour at all.

I would generally characterize more recent activity as being extremely biased towards making changes that appear have generally no effect on behaviour  (if not absolutely no effect), which allows people to feel more confident that they don't break anything and makes the changes simpler because they don't require rewriting any unrelated tests.  This isn't to say that I don't think the changes are good, I think almost all of them are, but I would also say that only a couple of them accomplish anything a user might care about.

Code:
#19219  -- Optimization. Corrects an exploitable vulnerability reported by a bcasher, took over a month to get merged. Doesn't change externally visible behaviour except for the vulnerability (and extremely rare hash collisions).

#19204 -- Local behaviour change that doesn't change the protocol. (proposed in a slightly different form in 2016, but never implemented in that form)

#19260 -- Straight up revert of #8709  which would mess up compatibility for nodes which implemented the 2016 alternative to #19204 (which presumably no one did, so it's not a problem)

#16939 -- Local behaviour change.  If addrman isn't empty wait 5 minutes before querying DNSseeds instead of 11 seconds. Only a behaviour change at all when getting a connection takes longer than 11 seconds but less than 5 minutes for a previously run node.

#19010 -- This is part of an actual protocol change, though one that is optional, disabled by default, and somewhat resource expensive and described in a 2017 (three year old) BIP. (also, not obviously useful-- though that's my opinion).

#18960 -- implementation optimization to facilitate #19010, no externally visible behaviour change.

#18861 -- local behaviour change.  I ran a node for the 2 months with a modified version of the PR that logged whenever it was triggered. It was never triggered, so in practice it's not even a behaviour change currently just theoretically. (I also got sipa to run my patch on his node, with the same results)  It's a fine enough change and I was happy to see it, though the leak it prevents still exists in other forms.

#18962 -- local behaviour change that only matters against a non-existing conjectural alternative implementations.

#18877  -- part of #19010

#18808  -- local behaviour change, doesn't make any difference with existing implementations. (Also I don't understand how it philosophically meshes with other changes from the last year or two which increase the nodes aggressiveness in disconnecting on unexpected messages, such as #19260, #15759, etc.)

#18038 -- local behaviour change to be much less aggressive in rebroadcasting transactions that haven't been mined in a long time and won't get mined soon. Adds separate tracking so that it's still aggressive about resending things which have never been getdataed at all, so that the reduction in aggression doesn't result in failing to broadcast entirely.  (Yippie! I hadn't seen this had gone in, I'm particularly happy about it.)

#16902 -- implementation optimization, no externally visible behaviour change

#17951 -- Extremely narrow local behaviour change, more of an implementation optimization.  Uses a cheaper but differently imprecise mechanism to avoid requesting broadcasts of already confirmed transactions (e.g. this one won't ignore long-ago-txn-with-still unspent outputs and has (rare) false positives).  Though it isn't mentioned in the PR anywhere that I can see, I believe that this change is significantly motivated by a future plant to change to witness-ids for relay: there is no index on nodes of past witness IDs, so the older check wouldn't work at all in a post-witness-relay world.  If that's correct, then this could be seen as a facilitating change for a future protocol change.

#16702 -- Experimental feature, off by default, requires a 'map file' that was difficult for me to produce as a subject matter expert. :)  Basically in places where the software treats /16 peers as being in the same network this replaces that with an ASN database. Perhaps when it becomes non-experimental and the project figures out how to ship a map file I could upgrade this one to local behaviour change. (It's a pretty cool local behaviour change, but still just a local behaviour change).

So there is only one thing that I would consider an actual change to the P2P protocol, and it's the realization of a protocol written in 2017.

In the context of this thread-- the question was observing stale blocks.  The idea of adding a P2P message to allow relaying their headers has been suggested many times probably going back to 2012. It came up more often post 2017 or so when it became clear that block relay improvements were making it extremely hard to observe stale blocks.  It would require a new P2P announcement since the existing headers message is a statement that you have the block-in-question available for getdata.

But no one is working on that. Now, if it had happened and been in your list I would have described it as an unimportant feature that wouldn't improve life for end users (though it would significantly improve developers ability to gauge the health of the network)-- given that and as you sample there are have been few recent changes that change the protocol, I really don't expect it to happen soon unless something changes.

Things aren't completely stagnant, e.g.

I've seen some activity towards revising ADDR messages. The current addr message are incompatible with v3 TOR hidden services, which have existed since 2017 and have much better properties. If this work is not completed and deployed within a year hidden services will stop working for Bitcoin users. Given the normal release and deployment timelines as well as the fact that the revised version is not merged or apparently merge-ready, I think it is now extremely likely that onion-peer support will be disrupted for users, in practice. Work on this began two years ago but didn't make progress for a long time.

There has also been some activity towards wtxid based transaction relay and the related erlay change.  There was some noise again on p2p encryption but it seems to not be particularly active.

But these changes are all moving extremely slowly, to the point where-- personally-- I forget everything about them between updates to them and don't find it interesting to contribute to them. Especially as I've found the testing framework makes it extremely tedious to work on any P2P behaviour change, because behaviour changes tend to break a wide assortment of unrelated tests, that people are slow to review or comment on them, and that their patches are continually broken by changes which are fast moving because they don't actually change behaviour.

Given that there isn't (yet) a fire behind p2p protocol updates required to keep onion peers working. I don't think it's unfair to characterize things as stagnant from the perspective of the p2p protocol itself and I really can't see purely instrumentation changes like stale-header-broadcast happening any time soon. (And I wouldn't recommend it: I'd recommend efforts be focused on addrv2 and other user-impacting changes like taproot!)

Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!