High orphan rate and long confirmation time discussion

[organofcorti]

The most recent bitcoin/bitcoin.git has "signature verification cache" which should speed up block validation quite a bit.

That's very interesting and if it makes a big difference should be a fairly urgent update for pool owners if the problem is as big as it appears to be.

It's mostly the rest of the network that needs to upgrade for this to be of any benefit.

I think he meant that should be a fairly urgent update in order to reduce the problem for pool owners.

[1714022784]

1714022784

[1714022784]

1714022784

[1714022784]

1714022784

[1714022784]

1714022784

[kano]

There are two parts to slowing down network propagation:
1) Sending crap repeatedly
2) Wasting time before sending it

It does seem rather odd that (even after I mentioned it) people are still debating what is actually the problem and looking for answers ...
Is it really that hard to put some debug in bitcoind and send a big block from one place to another and see the results at both ends?
Repeat and rinse.
Is this sort of testing beyond the level of bitcoin devs?

[deepceleron]

The first poster discovering the client-block-receive to miner-long-poll-receive delay time reveals another factor unrelated to the Bitcoin network block propagation speed - the time it takes for a pool to transmit a new block to all miners. In my past experience, depending on the number of pool miners, the number of pool servers, and the pool software and it's delays communicating with Bitcoin, going through the list of miners and sending each a LP response can take a significant amount of time, generally four, but up to ten seconds from the first to the last miner being notified. This from my own research profiling pool LP times as a predictor of who found the block.

If large block propagation was a significant factor, you would expect to see pools, more highly connected and on faster connections, sending your miner new block notification before your Bitcoin client gets it through the P2P network. That it is the opposite indicates instead the internal pool latency in pushing new work to all miners.

We have had 12 orphans since we went DGM 30/12/2011
2 were our fault when we didnt get the BIP16 updates right
so 10 orphans for over 6.5 months
The last 2 were 15th and 8th of this month, whilst either or both of these may be attributed to current issues, we are not showing an unusually high orphan rate.

We spent quite some time over the last 24hours looking to see what we could do poolside to help the longpolls, stales and big block issues.
We came to the conclusion that apart from optimising some logging and databases some more to reduce load there isnt really much we can do, this is a network issue and the pools will need help from the devs to resolve.

Ozcoin will continue to accept all standard transactions into our blocks, we do not see any advantage to restricting the number of transactions in our blocks.
....

One could make the argument that if your pool and a 0-tx mystery miner both generated a block at the same time, the much smaller block could propagate to miners first and be built upon first with another fast block find. The chief optimization, if you are dedicated to creating larger blocks than other miners may be creating, would be to ensure all your pool servers are highly connected to other pools, and communication of new blocks to other pools' known IPs takes priority. Pools should publish and share IPs of bitcoind's with each other (best it be a closed/trusted list to prevent DDOS facilitation), creating a high-priority sub-network to lower orphans for all.

Another coding optimization would be if you receive a second block solution, to immediately discard the first and push out the newer one if it has a significantly smaller hash, of course still keeping track of both as valid work from miners.

It seems like part of the problem with blocks not including transactions is the maximum block size of 500kB. Doesn't this pose a scalability problem?

There are also some odd rules mentioned in the Wiki that increase fees when the queue of transactions gets too deep. I haven't reviewed the Bitcoin code to comment further how these are enforced by miners and when sending in the client:
"If the blocksize (size of all transactions currently waiting to be included in a block) is less than 27 kB, transactions are free.
If the blocksize is more than 250 kB, transactions get increasingly more expensive as the blocksize approaches the limit of 500 kB."

The size of the queue of unconfirmed transactions waiting for inclusion is known by your client but can't be viewed. It can be seen here: http://bitcoincharts.com/bitcoin/txlist/. At this immediate moment, there are 2763 unconfirmed transactions (1454847 bytes). That would seem to me to be larger than the maximum blocksize indicated above, so the above rule about higher transaction fees may be kicking in on the miner's side for inclusion of transactions (but the client isn't prompting).

I would caution people to only check this page when you really need to - listing the current 2342781 byte backlog of transactions makes for 7.7MB of HTML.

[Gavin Andresen]

Another coding optimization would be if you receive a second block solution, to immediately discard the first and push out the newer one if it has a significantly smaller hash, of course still keeping track of both as valid work from miners.

Chain proof-of-work is calculated based on the hash target, so if you get another block at the same height there is no benefit to keeping the one with "the smaller hash".

Maybe if you receive a second block solution, keeping the block that removes the most transactions from the memory pool would be the right "good for the entire ecosystem" policy.  That way even if small blocks propagate slightly faster that might be offset if a larger, slower block was found.  (but making new-block-flooding independent of the size of the block is an even better solution, and that shouldn't be too hard to implement)

Creating a semi-trusted backbone of connections to other pools/miners so your new blocks propagate quickly is a good idea.

[doublec]

The first poster discovering the client-block-receive to miner-long-poll-receive delay time reveals another factor unrelated to the Bitcoin network block propagation speed - the time it takes for a pool to transmit a new block to all miners. In my past experience, depending on the number of pool miners, the number of pool servers, and the pool software and it's delays communicating with Bitcoin, going through the list of miners and sending each a LP response can take a significant amount of time, generally four, but up to ten seconds from the first to the last miner being notified. This from my own research profiling pool LP times as a predictor of who found the block.

I agree, longpoll notification is slow. I wonder if using a lightweight message queuing system like zeromq would do a better job instead of the http based long polling.

[kano]

The first poster discovering the client-block-receive to miner-long-poll-receive delay time reveals another factor unrelated to the Bitcoin network block propagation speed - the time it takes for a pool to transmit a new block to all miners. In my past experience, depending on the number of pool miners, the number of pool servers, and the pool software and it's delays communicating with Bitcoin, going through the list of miners and sending each a LP response can take a significant amount of time, generally four, but up to ten seconds from the first to the last miner being notified. This from my own research profiling pool LP times as a predictor of who found the block.

I agree, longpoll notification is slow. I wonder if using a lightweight message queuing system like zeromq would do a better job instead of the http based long polling.

... implement a protocol rather than use something designed for web servers ...

[rjk]

Creating a semi-trusted backbone of connections to other pools/miners so your new blocks propagate quickly is a good idea.

^ This +1. Any smart pool should be blasting its blocks out across as many nodes as it is able to as quickly as possible. One way to do this is to manually connect to several reliable super nodes.

[jgarzik]

^ This +1. Any smart pool should be blasting its blocks out across as many nodes as it is able to as quickly as possible. One way to do this is to manually connect to several reliable super nodes.

An effort to create a bitcoin backbone is now underway...

[-ck]

The first poster discovering the client-block-receive to miner-long-poll-receive delay time reveals another factor unrelated to the Bitcoin network block propagation speed - the time it takes for a pool to transmit a new block to all miners. In my past experience, depending on the number of pool miners, the number of pool servers, and the pool software and it's delays communicating with Bitcoin, going through the list of miners and sending each a LP response can take a significant amount of time, generally four, but up to ten seconds from the first to the last miner being notified. This from my own research profiling pool LP times as a predictor of who found the block.

I agree, longpoll notification is slow. I wonder if using a lightweight message queuing system like zeromq would do a better job instead of the http based long polling.

... implement a protocol rather than use something designed for web servers ...

longpoll is not the issue, but merely a symptom of other issues. Fix those and longpoll will still suffice... that and I'd be really pissed if I had to support something else after spending months getting longpoll working a treat in cgminer.

[deepceleron]

Another coding optimization would be if you receive a second block solution, to immediately discard the first and push out the newer one if it has a significantly smaller hash, of course still keeping track of both as valid work from miners.

Chain proof-of-work is calculated based on the hash target, so if you get another block at the same height there is no benefit to keeping the one with "the smaller hash".

Maybe if you receive a second block solution, keeping the block that removes the most transactions from the memory pool would be the right "good for the entire ecosystem" policy.  That way even if small blocks propagate slightly faster that might be offset if a larger, slower block was found.  (but making new-block-flooding independent of the size of the block is an even better solution, and that shouldn't be too hard to implement)

Creating a semi-trusted backbone of connections to other pools/miners so your new blocks propagate quickly is a good idea.

I had gotten the impression somewhere that "most difficult chain wins" meant the hash difficulties, not target difficulties. If that is not the case, the only time that there would be a more difficult chain would be on difficulty changes - i.e. if you had received two block 183456s (a block with a difficulty recalc), one may have a more difficult target calculated by its miner. A unique hack that might make your recalc block win an orphan race (once in a million) would be to round up the difficulty recalc within the limits of network acceptance. Outside of that, if the only policy Bitcoin miners use is "first new block received", orphan prevention would seem to be just a race to publication then, that you tell all other miners about your block before they find their own or receive a block from another pool.

[mrb]

sub

What is with all this "sub" spam? Is there some reason you guys can't just click to stupid "notify" link next to "reply"?

Yes, because the "notify" link sends an e-mail when there is a new post.  If you are watching 20 or so threads, that's 20 or so junk e-mails you get.  Where as if you make a post in the topic, "sub", then you can just click on the nice "Show new replies to your posts." at the top of the forums and get a list of all the threads you follow with new activity.

I concur with TheHarbinger. I wish forum admins would install the SMF Bookmark Mod, which provides the same user experience as "subscribing" to a thread.

[TheHarbinger]

1200 Unconfirmed transactions as I post this.   

[kano]

1200 Unconfirmed transactions as I post this.   

~1000 isn't rare at the moment.
I restarted my bitcoind 12 hours ago - but the highest I've got since then was only 1338 - so not as bad as some days.
... 1 less and it would have been 1337

[DeathAndTaxes]

I had gotten the impression somewhere that "most difficult chain wins" meant the hash difficulties, not target difficulties. If that is not the case, the only time that there would be a more difficult chain would be on difficulty changes - i.e. if you had received two block 183456s (a block with a difficulty recalc), one may have a more difficult target calculated by its miner. A unique hack that might make your recalc block win an orphan race (once in a million) would be to round up the difficulty recalc within the limits of network acceptance. Outside of that, if the only policy Bitcoin miners use is "first new block received", orphan prevention would seem to be just a race to publication then, that you tell all other miners about your block before they find their own or receive a block from another pool.

No it is target difficulty.  Generally speaking longest == more difficult.  The reason why the network checks difficulty and not raw length is because otherwise an attacker could exploit the dynamic difficulty and pull off a 51% attack with <51% of hashing power.  For most intents and purposes outside of possible attack longest == most difficult but the code only looks at the sum of the chain's block target difficulty to determine the best chain.

[TheHarbinger]

1200 Unconfirmed transactions as I post this.   

~1000 isn't rare at the moment.
I restarted my bitcoind 12 hours ago - but the highest I've got since then was only 1338 - so not as bad as some days.
... 1 less and it would have been 1337

We need Ozcoin to hit a block to clear the backlog.   

[Graet]

1200 Unconfirmed transactions as I post this.   

~1000 isn't rare at the moment.
I restarted my bitcoind 12 hours ago - but the highest I've got since then was only 1338 - so not as bad as some days.
... 1 less and it would have been 1337

We need Ozcoin to hit a block to clear the backlog.   

lol we did have a nice run while I slept
currently 704 Unconfirmed Transaction as I post, was ~300 when I woke up

[jgarzik]

1200 Unconfirmed transactions as I post this.   

That low?    It was over 9,000 unconfirmed just a few days ago...

[kano]

Hmm, no orphans for over 24 hours (since 185346) according to blockchain.
Well at least some good news even if it is unrelated to any code change

... jinx ...

[Graet]

1200 Unconfirmed transactions as I post this.   

That low?    It was over 9,000 unconfirmed just a few days ago...

https://blockchain.info/
slush just hoovered up over 900 txns

the unconfirmed number is very dynamic, sort of depends on whether pools accepting lots of txns get a run or those limiting txns get a run of blocks
looking every few days doesn't give a good representation

869 Unconfirmed Transactions now

[-ck]

For what it's worth, mtred have moved their bitcoind over to 0.6.3rc1 after I pointed it out to them and I have noticed a significant improvement in their block change speed (with much less lag across longpolls) already, even though a lot of the network has not yet changed over to the potentially faster protocol.