High orphan rate and long confirmation time discussion

[Schleicher]

Now it would be interesting to see if the orphaned blocks have more transactions included than the blocks in the main chain.

[DeathAndTaxes]

Well I am less concerned about block prop times as doesn't the new (0.7) of bitcoin eliminate the double verification of block txs.  That should dramatically help with prop times.

Also correct me if i am wrong but could block prop be spit into a two step process.

1) Broadcast block header.  Nodes can verify the validity of the pow w/ just header.
2) Nodes request block txs from headers they have confirmed are valid.

This would result in full block taking no longer to verify and propogate than a 0 tx bock.  It would also provide other useful benefits.


I am more concerned with some unexplained factor causing a rise in avg confirmation time.  The again if I was running a major pool I might not feel that way.

[gmaxwell]

This can be seen here:

Confirmations can't take zero minutes on average, that graphic is broken. Presumably the data doesn't start until the point where it first ticks above zero.

Can you update your post to reflect this? The implication that delays started so early is causing people down thread to jump to erroneous conclusions.

[Mike Hearn]

That solution is indeed possible and already being worked on.

There are some other things being looked at as well, for improving block propagation times.

[kano]

An increase in the number of transactions on the network is of course what EVERYONE wants.

What everyone may or may not want is how that happens.

But seriously, the BTC network is a tiny ant in the monetary world and certainly needs to be able to deal with something a LOT more than it can (without people complaining about it) today.

Attempting to reduce the transactions (as I've seen some suggest) is, IMO, completely against the BTC ideal.

IMO, anyone thinking that anything but a few rules should be encoded into the basic bitcoind spec and base code is missing the point.

Slowing down or ignoring txns/addresses/fee levels is up to pools and solo miners.
As long as pools aren't lying about what they do (or hiding the facts) there is no technical issue there, since people should mine at pools they are happy with the way the pool handles transactions.
The base client should not have much if any of this at all.

The problem that comes with this is, however, reality.
Some pool OPs may lie or hide what they are doing.
A lot of miners don't really care if their pool is fucking up the network.

Still, it's a BTC free market, not a BTC dev fascist state (even if some don't see it that way


Definitely the software needs improving, not more rules and restrictions added to it, to stunt the growth of BTC.


OT: Just a point about block times - yes if you look through the block-chain you will regularly find blocks that have a timestamp before the previous block.
If you look at the previous block you will often see that it is Eligius (at there, a 2000% E: is considered possible and reasonable)
That's life ...

[gmaxwell]

Some say that bitcoin was never meant to be a system for micro-transactions but that actually appears completely wrong to me if we're to assume widespread acceptance of btc.

You should probably avoid the use of the word microtransaction: Commonly used meanings span at least four orders of magnitude in value.

Is bitcoin a system suitable for $1 scale value transactions.  Perhaps.  Is it a system for sub-cent transactions, very very likely not.  At the end of the day bitcoin is a worldwide broadcast medium. The broadcastness provides very high confidence that it can't be artificially inflated and that old txn can't be reversed. But the fact that no information is hidden does ultimately limit the scaling— though I don't think that its much of a problem. You don't need that kind of security for very tiny transactions. So you can use something that isn't broadcast based to trade small values and frequently settle against bitcoin. ::shrugs::

Of course, none of this has anything to do with improving the performance for what we have today, of course that has to be done.   But the solution to that is "shut up and code" and "shut up and test", not more navel gazing on the forums.

[guruvan]

Just to make it clear what we're talking about:

... [images]

In case anyone can think of anything that occurred in conjunction with the increase in tx conf times, the start of the current tx conf times is 2012-02-26, with previous spikes on 2012-02-06 and 2012-02-15 to 2012-02-16.

In all my years of network management, this kind of chart clearly has indicated that I've made a change to my system that's resulted in the unanticipated increased use of a resource.

I'm certainly not an expert on bitcoin or it's code, but this definitely suggests that a change in the code has caused increased confirmation times.

[Luke-Jr]

Well I am less concerned about block prop times as doesn't the new (0.7) of bitcoin eliminate the double verification of block txs.  That should dramatically help with prop times.

Also correct me if i am wrong but could block prop be spit into a two step process.

1) Broadcast block header.  Nodes can verify the validity of the pow w/ just header.
2) Nodes request block txs from headers they have confirmed are valid.

This would result in full block taking no longer to verify and propogate than a 0 tx bock.  It would also provide other useful benefits.

No; since you now don't have the block until you've requested every txn you're missing, it would probably take longer. And you can't accept the block until you have all its transactions, or you create a security risk. That being said, it might still help to cut down on bandwidth use; my suggestion in this area is a compromise: send blocks as header+txnlist initially, but keep track of which transactions you didn't have when you received it, and automatically send those to peers you relay the block to before they ask for them. This way, nodes only have to explicitly ask for transactions they didn't have, but the peer relaying the block to them did have.

To address the block propagation problem itself, however, my first recommendation is to relay the block before checking transactions (but after checking proof-of-work and other fixed-cost checks, so DoS attacks are still impractical). The blocks with more transactions still have the cost of transaction verification delay (that cannot be fixed without changes to the Bitcoin system itself), but that delay shouldn't affect orphaning of the block itself.

[Schleicher]

In case anyone can think of anything that occurred in conjunction with the increase in tx conf times, the start of the current tx conf times is 2012-02-26, with previous spikes on 2012-02-06 and 2012-02-15 to 2012-02-16.

The 0.6.0 RC2 client was released around that time, I think.

[DeathAndTaxes]

No; since you now don't have the block until you've requested every txn you're missing, it would probably take longer. And you can't accept the block until you have all its transactions, or you create a security risk.

What security risk?  The proof of work can't be faked.  From the Pow point of view the tx in the block are immaterial.  The merkle root hash is merely a 256bit input.  Also i didn't mean download tx one at a time.  

More like:
broadcast block header -> node verifies and relays -> node verifies and relays -> node verifies and relays

Then nodes request the "block body" from peers and propagate.  If there is no valid matching merkle tree for the header the block is discarded as invalid and network revert to the prior longest chain.

[Luke-Jr]

No; since you now don't have the block until you've requested every txn you're missing, it would probably take longer. And you can't accept the block until you have all its transactions, or you create a security risk.

What security risk?  The proof of work can't be faked.  From the Pow point of view the tx in the block are immaterial.  The merkle root hash is merely a 256bit input.  Also i didn't mean download tx one at a time.  

More like:
broadcast block header -> node verifies and relays -> node verifies and relays -> node verifies and relays

Then nodes request the "block body" from peers and propagate.  If there is no valid matching merkle tree for the header the block is discarded as invalid and network revert to the prior longest chain.

And in the meantime, all the miners are working on an invalid blockchain...

[kano]

No; since you now don't have the block until you've requested every txn you're missing, it would probably take longer. And you can't accept the block until you have all its transactions, or you create a security risk.

What security risk?  The proof of work can't be faked.  From the Pow point of view the tx in the block are immaterial.  The merkle root hash is merely a 256bit input.  Also i didn't mean download tx one at a time.  

More like:
broadcast block header -> node verifies and relays -> node verifies and relays -> node verifies and relays

Then nodes request the "block body" from peers and propagate.  If there is no valid matching merkle tree for the header the block is discarded as invalid and network revert to the prior longest chain.

In fact as I told Luke-jr in IRC over a day ago - most bitcoind's already have every txn except the coinbase txn in their mempool.

[kano]

No; since you now don't have the block until you've requested every txn you're missing, it would probably take longer. And you can't accept the block until you have all its transactions, or you create a security risk.

What security risk?  The proof of work can't be faked.  From the Pow point of view the tx in the block are immaterial.  The merkle root hash is merely a 256bit input.  Also i didn't mean download tx one at a time.  

More like:
broadcast block header -> node verifies and relays -> node verifies and relays -> node verifies and relays

Then nodes request the "block body" from peers and propagate.  If there is no valid matching merkle tree for the header the block is discarded as invalid and network revert to the prior longest chain.

And in the meantime, all the miners are working on an invalid blockchain...

Only when there's an orphan issue ...

[organofcorti]

This can be seen here:

Confirmations can't take zero minutes on average, that graphic is broken. Presumably the data doesn't start until the point where it first ticks above zero.

Can you update your post to reflect this? The implication that delays started so early is causing people down thread to jump to erroneous conclusions.

Done.

[DeathAndTaxes]

And in the meantime, all the miners are working on an invalid blockchain...

Really that is the security risk?

Propogation should take at most a few seconds so miners would be wasting work on a bad block for a few seconds and only on the few time (<1% ?) that a bad block w/ valid header is propagated.  Current miners are wasting hashing power on every block from the time it is found until the time it is propagated to them.

Remember the header can't be faked a valid hash on a block with a random nonce as the merkle root hash requires the same amount of work as a valid hash on block header with a merkle root hash which corresponds to an actual set of valid txs.    Not much attack there (attack would by definition needs to waste 100x to 300x as much resources as he causes good miners to waste).  Maybe in very isolate incidences a miner may produce an invalid block and that error isn't detected for a few seconds but miners that do that aren't getting compensated so they have a vested interest in fixing that flaw.

[Realpra]

I developed a theoretical concept for a swarm client, that would likely solve this issue.

Basically blocks are split up into branches so that no clients have to propagate, store nor confirm a full block.

It's over in the development discussions.

[imsaguy]

subbb

[Gladamas]

subbb

[Raoul Duke]

Maybe they should fix the uPnP not working on at least Ubuntu(can't talk about other OS's as I have no experience with them).
I have uPnP enabled on the client, uPnP enabled on my router, yet, if I don't forward the port on the router I never get more than 8 connections.
Maybe an increase in better connected nodes will solve the block propagation time?
Transactions get propagated fairly quick even with 8 connections but maybe full blocks don't as they are bigger in size.

[Gladamas]

It seems like part of the problem with blocks not including transactions is the maximum block size of 500kB. Doesn't this pose a scalability problem?

There are also some odd rules mentioned in the Wiki that increase fees when the queue of transactions gets too deep. I haven't reviewed the Bitcoin code to comment further how these are enforced by miners and when sending in the client:
"If the blocksize (size of all transactions currently waiting to be included in a block) is less than 27 kB, transactions are free.
If the blocksize is more than 250 kB, transactions get increasingly more expensive as the blocksize approaches the limit of 500 kB."

The size of the queue of unconfirmed transactions waiting for inclusion is known by your client but can't be viewed. It can be seen here: http://bitcoincharts.com/bitcoin/txlist/. At this immediate moment, there are 2763 unconfirmed transactions (1454847 bytes). That would seem to me to be larger than the maximum blocksize indicated above, so the above rule about higher transaction fees may be kicking in on the miner's side for inclusion of transactions (but the client isn't prompting).