High orphan rate and long confirmation time discussion

[-ck]

So it appears pretty clear that most pools are suffering from a significant rise in orphan rates and many pool owners believe the issue appears to be related to slow block solve propagation due to the massive increase in transaction volume (mostly due to satoshidice).  Now it's clear that while only a small number of us are gamblers and care about satoshidice's success, the fact is most users of sd are obeying the transaction fees as determined by bitcoin (which are rather small now, but still present). So we're hitting some scalability issue with the network sooner than imagined. Some say that bitcoin was never meant to be a system for micro-transactions but that actually appears completely wrong to me if we're to assume widespread acceptance of btc.

I've seen massively delayed longpolls from various pools now and some just accept the shares long after the block has changed before they're sending out the longpoll, while other pools have a deathly pause, not returning a response about shares for up to 25 seconds, then sending out a longpoll and 25 seconds worth of rejected work. Some say that deepbit's transaction limits means they're avoiding this problem, but I'm not informed enough to know if this is true, but it would seem to me this must incur significant transaction delays. Some solution needs to be worked on that benefits all - pools, miners, and bitcoin at large. I don't want to see transactions delayed for up to 6 hours when even transaction fees are paid, but nor do I want my pools to become a massive orphanage.

Please discuss.

[1714103568]

1714103568

[1714103568]

1714103568

[1714103568]

1714103568

[1714103568]

1714103568

[1714103568]

1714103568

[Graet]

Well said.

We are working on this issue for Ozcoin today, if it is suitable and works we will share our solution, I know others are trying different things too

[Shadow383]

It is a major issue - if you look at the sort of transaction volume most stock exchanges or major payment processors get, it seems doubtful that the network as it stands could cope.

[organofcorti]

This is a very interesting question and I'm glad you made a thread for it. From the little investigation I've done it seems that there's been an increase in orphans since about March, which coincides with a sudden increase in confirmation times. The strange thing about this is the sudden increase in confirmation times does not coincide with the increase in transactions per block which started to increase from May, two months later.

This can be seen here:

http://blockchain.info/charts/avg-confirmation-time
http://blockchain.info/charts/n-transactions-per-block

So the increase in tx per block does not seem to have caused the increase in confirmation times. Satoshidice doesn't look like the culprit to me.

[-ck]

This is a very interesting question and I'm glad you made a thread for it. From the little investigation I've done it seems that there's been an increase in orphans since about March, which coincides with a sudden increase in confirmation times. The strange thing about this is the sudden increase in confirmation times does not coincide with the increase in transactions per block which started to increase from May, two months later.

This can be seen here:

http://blockchain.info/charts/avg-confirmation-time
http://blockchain.info/charts/n-transactions-per-block

So the increase in tx per block does not seem to have caused the increase in confirmation times. Satoshidice doesn't look like the culprit to me.

Interesting... is this BIP16 then?

[organofcorti]

Interesting... is this BIP16 then?

Don't think so - BIP16 was introduced 1st April, wasn't it? Increased confirmation times began in March. Can you think of anything else that changed then?

[-ck]

Interesting... is this BIP16 then?

Don't think so - BIP16 was introduced 1st April, wasn't it? Increased confirmation times began in March. Can you think of anything else that changed then?

It was accepted April 1, but there were pools that had already adopted it if I recall correctly.

[organofcorti]

Interesting... is this BIP16 then?

Don't think so - BIP16 was introduced 1st April, wasn't it? Increased confirmation times began in March. Can you think of anything else that changed then?

It was accepted April 1, but there were pools that had already adopted it if I recall correctly.

Do you know of any way to find out when pools started to apply BIP16, without having to trawl the threads? I could look at every pool and try to correlate orphan rate with BIP16 take up.

Why would BIP16 cause increased confirmation times?

[-ck]

Do you know of any way to find out when pools started to apply BIP16, without having to trawl the threads? I could look at every pool and try to correlate orphan rate with BIP16 take up.

Why would BIP16 cause increased confirmation times?

I have no idea but it is the one most significant change to bitcoin that correlates very strongly with that time frame, unlike satoshidice's increase in transaction volume. Is BIP16 the same time we started accepting smaller transaction fees? They're allegedly 1/20th of what they were, but I don't know when that change hit. Maybe they really did need to be bigger?

[doublec]

Why would BIP16 cause increased confirmation times?

IIRC a bitcoin release candidate was used by some pools that had a BIP 16 switch on time 2 weeks earlier than what the official time eventually became. Pools and individual miners using this RC would have produced a lot of orphans until they noticed. This could cause increased confirmation times for a brief period when a chunk of the network was effectively offline. I don't think this was very long, nor was it a big chunk.

[-ck]

Why would BIP16 cause increased confirmation times?

IIRC a bitcoin release candidate was used by some pools that had a BIP 16 switch on time 2 weeks earlier than what the official time eventually became. Pools and individual miners using this RC would have produced a lot of orphans until they noticed. This could cause increased confirmation times for a brief period when a chunk of the network was effectively offline. I don't think this was very long, nor was it a big chunk.

Interesting, but the increased confirmation times and orphans persisted, so perhaps it's something about the protocol itself rather than it interacting with the non-BIP16 world? The timing of 2 weeks before correlates almost exactly with the graph showing increased transaction time.

[-ck]

The number of BIP16 transactions in the blockchain is very small.
I doubt BIP16 is the root cause.

I'm not implying it's actual BIP16 transactions that are responsible, but whatever code changes or protocol changes were required to support them since the correlation with rising transaction time is surprisingly strong.

[doublec]

Interesting, but the increased confirmation times and orphans persisted, so perhaps it's something about the protocol itself rather than it interacting with the non-BIP16 world? The timing of 2 weeks before correlates almost exactly with the graph showing increased transaction time.

After the two weeks you get the miners who hadn't switched to the BIP 16 enabled code getting orphans. Maybe there's a bunch of standalone miners running non-upgraded code that are unattended (remains of botnets, unattended servers, etc?)

[organofcorti]

Edit: as gmaxwell suggested I contacted Blockchain.info and indeed they only started recording Avg tx conformation time In February. That being the case, it doesn't seem that there's a sudden large increase after all.


Just to make it clear what we're talking about:





In case anyone can think of anything that occurred in conjunction with the increase in tx conf times, the start of the current tx conf times is 2012-02-26, with previous spikes on 2012-02-06 and 2012-02-15 to 2012-02-16.

[organofcorti]

Is BIP16 the same time we started accepting smaller transaction fees? They're allegedly 1/20th of what they were, but I don't know when that change hit. Maybe they really did need to be bigger?

Smaller tx fees started with Bitcoin vers 0.3.23, which was available from June 2011.

[Syke]

I'm looking over some of the orphan races at:

http://blockchain.info/orphaned-blocks

Races where 2 blocks are found with timestamps a few seconds apart is to be expected. But I'm seeing quite a few races where block are minutes apart. If it's taking blocks minutes to propate across the network, we've got a problem.

Like here's a particularly nasty one.

http://blockchain.info/block-index/232898/0000000000000873fda1001c4986936dfa3697ec84ff74e48c11fe9d355be07f

vs.

http://blockchain.info/block-index/232896/000000000000069c27242a928e33e45ddd27d6fd014c77946ba4d3d2891f5398

Some poor miner finds a block, and 30 minutes later DeepBit still hasn't received that block and thus orphans it when they find a block. 30 minutes!

[-ck]

I'm looking over some of the orphan races at:

http://blockchain.info/orphaned-blocks

Races where 2 blocks are found with timestamps a few seconds apart is to be expected. But I'm seeing quite a few races where block are minutes apart. If it's taking blocks minutes to propate across the network, we've got a problem.

Like here's a particularly nasty one.

http://blockchain.info/block-index/232898/0000000000000873fda1001c4986936dfa3697ec84ff74e48c11fe9d355be07f

vs.

http://blockchain.info/block-index/232896/000000000000069c27242a928e33e45ddd27d6fd014c77946ba4d3d2891f5398

Some poor miner finds a block, and 30 minutes later DeepBit still hasn't received that block and thus orphans it when they find a block. 30 minutes!

Bear in mind that block generation times can be not remotely accurate with rolltime enabled. cgminer limits rolling to about 10 minutes, so the time could be 10 minutes out from the real local time, but apparently luke-jr has removed all limits on his rolltime on eligius meaning work from there could appear to be from any time in the future.

[organofcorti]

Full history of orphaned blocks at p2Pool, ordered by numbers of blocks solved:



Although it looks like this could just be because more blocks are being solved, there's no obvious correlation between orphans and hashrate:



I'd like to get data from a much more established pool though.

[DeathAndTaxes]

Just to make it clear what we're talking about:

(snip)

Thanks for the charts it illustrates the symptoms clearly.  That sharp rise in avg confirmation time can't be explained by increased tx volume.

[Luke-Jr]

Just to make it clear what we're talking about:

(snip)

Thanks for the charts it illustrates the symptoms clearly.  That sharp rise in avg confirmation time can't be explained by increased tx volume.

The avg confirmation time isn't the (immediate) problem. The problem is block propagation time. If that isn't solved, you will likely see another sharp rise in avg confirmation time soon.