High orphan rate and long confirmation time discussion

[jetmine]

Another coding optimization would be if you receive a second block solution, to immediately discard the first and push out the newer one if it has a significantly smaller hash, of course still keeping track of both as valid work from miners.

Chain proof-of-work is calculated based on the hash target, so if you get another block at the same height there is no benefit to keeping the one with "the smaller hash".

Maybe if you receive a second block solution, keeping the block that removes the most transactions from the memory pool would be the right "good for the entire ecosystem" policy. 

Using "the smaller hash" has a benefit:  It is unambiguous!

Orphans are the result of a network split (different portions of the network consider a different block as the best current block).  If a client knows two blocks and has to decide, using the smaller hash will always lead to the same decision.  This works towards the goal of reducing orphans (reducing the occurrance of differing decisions on distinct clients).

Your proposal of using the "most beneficious block" does not necessarily fulfill this goal.  Since transactions themselves suffer similar network split problems, two distinct clients can make differing decisions about the same set of blocks.  Thus leading to the same result: orphans.  In fact, your solution may possibly produce more orphans than before.

In the past I have thought about the propagation problem as well, but refrained from posting a formal proposal so far.  However, I came to a similar result.  The single best solution is to use the "smaller hash" (as in "more work" or better: "more luck") which is universally testable for everyone who knows the same particular set of competing blocks.

Also, in my thoughts, it makes sense to consider a maximum expected propagation time.  A block will typically take no more than N seconds to propagate to most clients.  I made calculations, extrapolated from number of hops, current network size, but dont remember the exact outcome.  Whatever the best value of N is does not matter here.  What matters is that when a client first receives a block of new height, it should open a "propagation curtesy window" of N seconds.  When a competing block is received within that window, then it can be reasonably considered potential victim of network skew and a resolution can be made by prefering the "lower hash" block.

If a competing block is received outside of the N seconds window, it can be disregarded(like now).  It can't be victim of just network skew (unless N was poorly chosen).

The advantage is obvious.  Mostly everyone will work on the same block, even in situations that currently produce an orphan.  AND, the proposal is completely voluntary and 100% compatible.  Nobody would be able to tell whether or not you have implemented it (unless he is monitoring your network traffic and sees in which order you have really received the blocks).  This is the best proof for compatibility.  There is no disadvantage to the current system.  The more miners implement it, the lower the orphan rate goes.

And last not least, this approach doesn't open a security vulnerability either.  It is more difficult to create a block with "lower hash" than to create a "just enough" block with better height.  Therefore there is no benefit of trying to create "lower hash" blocks for already existing blocks.  Since this method relies on nothing else but the block hash, is is "unhackable".  Your proposal however does not have this property.  It also depends on the transaction memory pool, which can easily be poisoned.

I have thought about it since about 3 months, and I am absolutely convinced that it is the universally best solution.

[Luke-Jr]

Using "the smaller hash" has a benefit:  It is unambiguous!

Orphans are the result of a network split (different portions of the network consider a different block as the best current block).  If a client knows two blocks and has to decide, using the smaller hash will always lead to the same decision.  This works towards the goal of reducing orphans (reducing the occurrance of differing decisions on distinct clients).

No, we're talking about a scenario where there is already an orphan. It cannot be avoided. Even if the whole network focusses on extending only one of those immediately, your next block still has the same risk of an orphan. So it doesn't gain anything. On the other hand, using the lower hash opens the door to new attacks: if I find a block with a super-low hash, I can just wait to broadcast it until someone else finds it with a higher hash, and instant-orphan them!

[jetmine]

On the other hand, using the lower hash opens the door to new attacks: if I find a block with a super-low hash, I can just wait to broadcast it until someone else finds it with a higher hash, and instant-orphan them!

If you find a block with a super-low hash, you can just broadcast it right away and you GET THE REWARD.  There is no incentive to hold the block back, and play chances.  The chances would be to have to compete against a later block, which could turn out to be an even lower hash.  You can calculate those chances (based on difficulty and your super-low hash), but they never turn out favorably against the guaranteed reward.

Also, how do you know that similar schemes are not already active?  I assume your pool doesnt do it, based on what you write in the post I am replying to.  But there is plenty of evidence that blocks are not just elected by arrival time (standard client).  The two blocks mentioned here in this thread can serve as example.  Blockchain.info RX timestamps were half an hour apart, yet based on same clock source.  There was no 3rd block at this height, so we can safely assume that Deepbit has either not known, or voluntarily ignored the new block for half an hour.  Do you really think that network propagation has caused this phenomena?  Or rather it suggests that Deepbit used a different election criteria than arrival time ...

Think about it.

EDIT: The block in question is 183255.  If you look at the preceding block 183254, you will notice that blockchain.info tags it as Deepbit too.  The same goes for the superceding block 183256.  We are looking at a chain of blocks from Deepbit, with a silenced attempt of interruption by someone else.  But Deepbit "pokered" during half an hour, and eventually killed the competitor.  What was the algorithm used?  It isnt the standard client, so tell me.

[Graet]

On the other hand, using the lower hash opens the door to new attacks: if I find a block with a super-low hash, I can just wait to broadcast it until someone else finds it with a higher hash, and instant-orphan them!

If you find a block with a super-low hash, you can just broadcast it right away and you GET THE REWARD.  There is no incentive to hold the block back, and play chances.  The chances would be to have to compete against a later block, which could turn out to be an even lower hash.  You can calculate those chances (based on difficulty and your super-low hash), but they never turn out favorably against the guaranteed reward.

Also, how do you know that similar schemes are not already active?  I assume your pool doesnt do it, based on what you write in the post I am replying to.  But there is plenty of evidence that blocks are not just elected by arrival time (standard client).  The two blocks mentioned here in this thread can serve as example.  Blockchain.info RX timestamps were half an hour apart, yet based on same clock source.  There was no 3rd block at this height, so we can safely assume that Deepbit has either not known, or voluntarily ignored the new block for half an hour.  Do you really think that network propagation has caused this phenomena?  Or rather it suggests that Deepbit used a different election criteria than arrival time ...

Think about it.

EDIT: The block in question is 183255.  If you look at the preceding block 183254, you will notice that blockchain.info tags it as Deepbit too.  The same goes for the superceding block 183256.  We are looking at a chain of blocks from Deepbit, with a silenced attempt of interruption by someone else.  But Deepbit "pokered" during half an hour, and eventually killed the competitor.  What was the algorithm used?  It isnt the standard client, so tell me.

If only blockchain.info reported block finders correctly. it hasn't for a long time. and is a well known issue.
lots of conspiracy theories come from it
carry on

[organofcorti]

On the other hand, using the lower hash opens the door to new attacks: if I find a block with a super-low hash, I can just wait to broadcast it until someone else finds it with a higher hash, and instant-orphan them!

If you find a block with a super-low hash, you can just broadcast it right away and you GET THE REWARD.  There is no incentive to hold the block back, and play chances.  The chances would be to have to compete against a later block, which could turn out to be an even lower hash.  You can calculate those chances (based on difficulty and your super-low hash), but they never turn out favorably against the guaranteed reward.

Also, how do you know that similar schemes are not already active?  I assume your pool doesnt do it, based on what you write in the post I am replying to.  But there is plenty of evidence that blocks are not just elected by arrival time (standard client).  The two blocks mentioned here in this thread can serve as example.  Blockchain.info RX timestamps were half an hour apart, yet based on same clock source.  There was no 3rd block at this height, so we can safely assume that Deepbit has either not known, or voluntarily ignored the new block for half an hour.  Do you really think that network propagation has caused this phenomena?  Or rather it suggests that Deepbit used a different election criteria than arrival time ...

Think about it.

EDIT: The block in question is 183255.  If you look at the preceding block 183254, you will notice that blockchain.info tags it as Deepbit too.  The same goes for the superceding block 183256.  We are looking at a chain of blocks from Deepbit, with a silenced attempt of interruption by someone else.  But Deepbit "pokered" during half an hour, and eventually killed the competitor.  What was the algorithm used?  It isnt the standard client, so tell me.

If only blockchain.info reported block finders correctly. it hasn't for a long time. and is a well known issue.
lots of conspiracy theories come from it
carry on

Timestamps aren't accurate either.

I've thought about it, jetmine, and I simply cannot see any way that a pool can convince a majority of nodes to accept the block it has produced over someone else's. That would require network coercion on a massive scale.

A chain of blocks from deepbit isn't surpising. Deepbit has about 28% of the network hash rate. They will on average solve 28% of the blocks. Each block solved has a 28% chance of coming from them.

This tends to be more accurate with regards to block ownership: http://blockorigin.pfoe.be/blocklist.php

[kano]

...
Timestamps aren't accurate either.

I've thought about it, jetmine, and I simply cannot see any way that a pool can convince a majority of nodes to accept the block it has produced over someone else's. That would require network coercion on a massive scale.
...

Actually, I thought of what might be a reasonably simple way to coerce the network a week or so ago ...
Run lots of bitcoind's ...
If you look at blockchain.info it is connected to what? around 250 nodes - would it be that hard to run 250 optimised on their actual block databases and setup appropriately, and point them at a big pool?
However, Luke-jr ignored it when I suggested it ...

[organofcorti]

tl;dr:
1. Orphan rates for the network have been increasing since May, not February.
2. Block size may have an effect on the likelihood of a particular block being confirmed or orphaned.


A few days ago I finally got network orphan rate and size data. I got it from Blockchain.info (couldn't get bitcoin-tools to work on my mac even with the updated BerkeleyDB module - I apologise for hammering your servers there guys) - since then I've been analysing it and trying the best way to show it, and here it is.


                       


The first chart shows the cumulative sum of orphans since August last year. The big peak in April I assume is due to the BIP16 change, but since then the rate of orphaned block creation has increased somewhat, although since Block height 180000 the number of orphans has increased in an approximately linearly, slope ~ 0.02, or 2 orphans for every one hundred blocks.

The second chart shows the same data as number of orphans per difficulty period. After settling down just after April, the number of orphans per difficulty period has been increasing, as seen in the cumulative orphans chart from block height 175000. This actually tallies well with the blockchain.info chart I posted a page back which shows a spike in number of transactions since May, and may indicate a relationship after all.

The last chart shows boxplots of the orphaned block size minus the related confirmed block size in kB (1e03). For the rare cases of more than one orphaned sibling, the mean was taken. This shows a significant change in the variance of the difference since BIP16 -  lots of small blocks competing against larger ones, sometimes winning and sometimes losing, although the median of the last difficulty period shows a skew toward orphans being larger than the related confirmed blocks.

My hypothesis was that larger blocks with more transactions would propagate more slowly and be more likely to be orphaned. The boxplot for the last difficulty period does lend some credence to the idea, but we'll have to wait another difficulty period or two to be sure.

Another explanation may be that some pools are making bocks with fewer transactions, and other pools aren't. This in itself may be creating random variance and not be related to the orphan production.

Any other ideas?

[kano]

organofcorti - yes some pools have varying rules about ignoring transactions (or putting in no transactions)
Some of course are very good with putting lots of txn's in their blocks.
This is a grounded well known fact - not a postulation.

[organofcorti]

organofcorti - yes some pools have varying rules about ignoring transactions (or putting in no transactions)
Some of course are very good with putting lots of txn's in their blocks.
This is a grounded well known fact - not a postulation.

I'm aware of that. The postulation was that it may be causing the variance in block size.

[deepceleron]

The last chart shows boxplots of the orphaned block size minus the related confirmed block size in kB (1e03). For the rare cases of more than one orphaned sibling, the mean was taken. This shows a significant change in the variance of the difference since BIP16 -  lots of small blocks competing against larger ones, sometimes winning and sometimes losing, although the median of the last difficulty period shows a skew toward orphans being larger than the related confirmed blocks.

My hypothesis was that larger blocks with more transactions would propagate more slowly and be more likely to be orphaned. The boxplot for the last difficulty period does lend some credence to the idea, but we'll have to wait another difficulty period or two to be sure.

Another explanation may be that some pools are making bocks with fewer transactions, and other pools aren't. This in itself may be creating random variance and not be related to the orphan production.

Any other ideas?

BIP16 caused a huge amount of orphans. All it took was for one person to push out a transaction to split the network, and some miners have still tried to include it this month:

http://blockchain.info/tx-index/3618498/4005d6bea3a93fb72f006d23e2685b85069d270cb57d15f0c057ef2d5e3f78d2

[organofcorti]

There was a huge spike of orphans in April (around 10% of network blocks had orphan races, one orphan actually had several siblings), but BIP16 is not causing orphans anymore. Are you saying that left over tx from the BIP16 orphans are still yet to be completed and adding to the current average tx size increase?

[deepceleron]

There was a huge spike of orphans in April (around 10% of network blocks had orphan races, one orphan actually had several siblings), but BIP16 is not causing orphans anymore. Are you saying that left over tx from the BIP16 orphans are still yet to be completed and adding to the current average tx size increase?

If a miner tries to include the transaction above because they have not upgraded to BIP16, they will create a block that is not accepted by the network, other miners won't build on it. That transaction is still floating around on the network, the last time it was included in a block (getting the miner an orphan instead of 50 BTC) was a week ago.

[organofcorti]

There was a huge spike of orphans in April (around 10% of network blocks had orphan races, one orphan actually had several siblings), but BIP16 is not causing orphans anymore. Are you saying that left over tx from the BIP16 orphans are still yet to be completed and adding to the current average tx size increase?

If a miner tries to include the transaction above because they have not upgraded to BIP16, they will create a block that is not accepted by the network, other miners won't build on it. That transaction is still floating around on the network, the last time it was included in a block (getting the miner an orphan instead of 50 BTC) was a week ago.

So the these non BIP16 tx's will continue to pop up and cause orphans?

[eleuthria]

There was a huge spike of orphans in April (around 10% of network blocks had orphan races, one orphan actually had several siblings), but BIP16 is not causing orphans anymore. Are you saying that left over tx from the BIP16 orphans are still yet to be completed and adding to the current average tx size increase?

If a miner tries to include the transaction above because they have not upgraded to BIP16, they will create a block that is not accepted by the network, other miners won't build on it. That transaction is still floating around on the network, the last time it was included in a block (getting the miner an orphan instead of 50 BTC) was a week ago.

So the these non BIP16 tx's will continue to pop up and cause orphans?

Transactions like the one linked will only cause an orphan to the miner/pool that make a block containing it.  The block they make is invalid per the majority of the network's rules (BIP16), and all properly updated clients will continue building the chain off the previous block, ignoring the invalid one.

[organofcorti]

There was a huge spike of orphans in April (around 10% of network blocks had orphan races, one orphan actually had several siblings), but BIP16 is not causing orphans anymore. Are you saying that left over tx from the BIP16 orphans are still yet to be completed and adding to the current average tx size increase?

If a miner tries to include the transaction above because they have not upgraded to BIP16, they will create a block that is not accepted by the network, other miners won't build on it. That transaction is still floating around on the network, the last time it was included in a block (getting the miner an orphan instead of 50 BTC) was a week ago.

So the these non BIP16 tx's will continue to pop up and cause orphans?

Transactions like the one linked will only cause an orphan to the miner/pool that make a block containing it.  The block they make is invalid per the majority of the network's rules (BIP16), and all properly updated clients will continue building the chain off the previous block, ignoring the invalid one.

So it's up to the pool to prevent the old transactions from being included. It is likely in your opinion that at least a good part of the orphans atm are being caused by old and non-BIP16 transactions?

[eleuthria]

So it's up to the pool to prevent the old transactions from being included. It is likely in your opinion that at least a good part of the orphans atm are being caused by old and non-BIP16 transactions?

No, very few of them are caused by this anymore.  The transaction linked earlier (http://blockchain.info/tx-index/3618498/4005d6bea3a93fb72f006d23e2685b85069d270cb57d15f0c057ef2d5e3f78d2) is the only one I'm aware of.  As far as I'm aware, that transaction was not created with a standard client, it was purposely crafted in a way to be valid under non-BIP16 bitcoind clients and invalid under BIP16 bitcoind clients.

You can follow on that page the number of orphans that have been caused by it (4 this month).

It is up to the miner to prevent the transaction from being included, yes.  Any bitcoind released in the last few months will automatically reject the transaction and not include it in a block.  There is nothing that can be done about people who refuse to update months after the change was made.  BIP16 did not appear overnight, there was a lot of debate and warning about its implementation.

[organofcorti]

It is up to the miner to prevent the transaction from being included, yes.  Any bitcoind released in the last few months will automatically reject the transaction and not include it in a block.  There is nothing that can be done about people who refuse to update months after the change was made.  BIP16 did not appear overnight, there was a lot of debate and warning about its implementation.

Miner or pool? I thought the pools decided on tx's to include. I could be wrong, this really isn't an area I know well.


EDIT: I think you were referring to solo miners and pooled mining there?

[eleuthria]

It is up to the miner to prevent the transaction from being included, yes.  Any bitcoind released in the last few months will automatically reject the transaction and not include it in a block.  There is nothing that can be done about people who refuse to update months after the change was made.  BIP16 did not appear overnight, there was a lot of debate and warning about its implementation.

Miner or pool? I thought the pools decided on tx's to include. I could be wrong, this really isn't an area I know well.


EDIT: I think you were referring to solo miners and pooled mining there?

Either solo miners or the pool.  Whichever entity is putting the block together and including the transactions.

[kano]

It is up to the miner to prevent the transaction from being included, yes.  Any bitcoind released in the last few months will automatically reject the transaction and not include it in a block.  There is nothing that can be done about people who refuse to update months after the change was made.  BIP16 did not appear overnight, there was a lot of debate and warning about its implementation.

Miner or pool? I thought the pools decided on tx's to include. I could be wrong, this really isn't an area I know well.


EDIT: I think you were referring to solo miners and pooled mining there?

Either solo miners or the pool.  Whichever entity is putting the block together and including the transactions.

Well - the fix for higher network TH/s related to this statement, is to move it to the miner ... rather than increase the nonce size ...

https://bitcointalk.org/index.php?topic=89278.0

I guess since Satoshi decided a 32bit nonce will be big enough originally, it must be big enough for eternity ...