High orphan rate and long confirmation time discussion

[Tittiez]

subbb

[mdude77]

subbb

Sub'd.

M

[kano]

It seems like part of the problem with blocks not including transactions is the maximum block size of 500kB. Doesn't this pose a scalability problem?

...
The size of the queue of unconfirmed transactions waiting for inclusion is known by your client but can't be viewed. It can be seen here: http://bitcoincharts.com/bitcoin/txlist/. At this immediate moment, there are 2763 unconfirmed transactions (1454847 bytes). That would seem to me to be larger than the maximum blocksize indicated above, so the above rule about higher transaction fees may be kicking in on the miner's side for inclusion of transactions (but the client isn't prompting).

The size of the 0-confirm queue is shown in debug.log every time addUnchecked() is called just before a txn is stored in memory pool.
And that is called every time - so you can watch it count up with each txn or orphan and drop down with each block
e.g. just now: 06/18/12 21:13:21 addUnchecked(): size 1141
That 1141 is the number of txns in the memory pool - that is the number of 0-confirm txns

Edit: though it is only correct if your bitcoind has been running for a while since you wont know all 0-confirm txns if you just started your bitcoind

Edit2: see here's what happens in debug.log when a block is found - just now - Ozcoin (with other stuff around it removed):
06/18/12 21:24:40 addUnchecked(): size 1465
06/18/12 21:24:50 SetBestChain: new best=000000000000007b3cb3  height=185186  work=359073512741370072393
06/18/12 21:24:50 addUnchecked(): size 1018

So that block (185186) processed 448 txn's from the memory pool (+1 coinbase that wasn't in the memory pool as per expected)
244.80kB in size

Edit3: however the next block by Bitparking was (they ignored all 0-confirm txns):
06/18/12 21:30:13 addUnchecked(): size 1114
06/18/12 21:30:15 SetBestChain: new best=000000000000046b404d  height=185187  work=359080312542205322700
06/18/12 21:30:19 addUnchecked(): size 1115

That block (185187) processed 0 txn's from the memory pool
0.25kB in size

Edit4: the next block was again Ozcoin
06/18/12 21:46:40 addUnchecked(): size 1460
06/18/12 21:46:56 SetBestChain: new best=0000000000000732fdc3  height=185188  work=359087112343040573007
06/18/12 21:46:57 addUnchecked(): size 822

So that block (185188) processed 639 txn's from the memory pool (+1 coinbase ...)
243.31kB in size

Edit5: and for the last edit - the next block was Deepbit
06/18/12 21:49:04 addUnchecked(): size 862
06/18/12 21:49:11 SetBestChain: new best=000000000000085b53c3  height=185189  work=359093912143875823314
06/18/12 21:49:12 addUnchecked(): size 689

So that block (185189) processed 174 txn's from the memory pool (+1 coinbase ...)
49.80kB in size

[Raize]

What incentives are there toward propagating blocks? Does the code look for a different node if one node appears to be delaying sending a specific block?

The reason why I ask is this, it would seem that if you are a mining pool that has a reasonable idea of the hashing power of Deepbit, you could take a block they've solved in a faster timeframe than normal and ignore sending it on immediately, instead mining on it for some time simply to get an advantage over the other mining pools that don't yet have the block. Sure, you run the risk of orphaning the block and making your work irrelevant, and it is likely that other nodes connected to Deepbit would send it on fast enough anyway, but the less clients connected to the Bitcoin network, the greater advantage withholding the block nets you.

I'm not saying such a thing is happening here, but there would seem to be some advantage to such a delay.

[kano]

There is the risk that you don't propagate your block fast enough if some other pool finds a block you are witholding.
A 50 BTC risk.

[Gladamas]

It seems like part of the problem with blocks not including transactions is the maximum block size of 500kB. Doesn't this pose a scalability problem?

...
The size of the queue of unconfirmed transactions waiting for inclusion is known by your client but can't be viewed. It can be seen here: http://bitcoincharts.com/bitcoin/txlist/. At this immediate moment, there are 2763 unconfirmed transactions (1454847 bytes). That would seem to me to be larger than the maximum blocksize indicated above, so the above rule about higher transaction fees may be kicking in on the miner's side for inclusion of transactions (but the client isn't prompting).

The size of the 0-confirm queue is shown in debug.log every time addUnchecked() is called just before a txn is stored in memory pool.
And that is called every time - so you can watch it count up with each txn or orphan and drop down with each block
e.g. just now: 06/18/12 21:13:21 addUnchecked(): size 1141
That 1141 is the number of txns in the memory pool - that is the number of 0-confirm txns

Edit: though it is only correct if your bitcoind has been running for a while since you wont know all 0-confirm txns if you just started your bitcoind

Edit2: see here's what happens in debug.log when a block is found - just now - Ozcoin (with other stuff around it removed):
06/18/12 21:24:40 addUnchecked(): size 1465
06/18/12 21:24:50 SetBestChain: new best=000000000000007b3cb3  height=185186  work=359073512741370072393
06/18/12 21:24:50 addUnchecked(): size 1018

So that block (185186) processed 448 txn's from the memory pool (+1 coinbase that wasn't in the memory pool as per expected)
244.80kB in size

Edit3: however the next block by Bitparking was (they ignored all 0-confirm txns):
06/18/12 21:30:13 addUnchecked(): size 1114
06/18/12 21:30:15 SetBestChain: new best=000000000000046b404d  height=185187  work=359080312542205322700
06/18/12 21:30:19 addUnchecked(): size 1115

That block (185187) processed 0 txn's from the memory pool
0.25kB in size

Edit4: the next block was again Ozcoin
06/18/12 21:46:40 addUnchecked(): size 1460
06/18/12 21:46:56 SetBestChain: new best=0000000000000732fdc3  height=185188  work=359087112343040573007
06/18/12 21:46:57 addUnchecked(): size 822

So that block (185188) processed 639 txn's from the memory pool (+1 coinbase ...)
243.31kB in size

Edit5: and for the last edit - the next block was Deepbit
06/18/12 21:49:04 addUnchecked(): size 862
06/18/12 21:49:11 SetBestChain: new best=000000000000085b53c3  height=185189  work=359093912143875823314
06/18/12 21:49:12 addUnchecked(): size 689

So that block (185189) processed 174 txn's from the memory pool (+1 coinbase ...)
49.80kB in size

So this means that they are purposely not including some transactions?

[TheHarbinger]

It seems like part of the problem with blocks not including transactions is the maximum block size of 500kB. Doesn't this pose a scalability problem?

...
The size of the queue of unconfirmed transactions waiting for inclusion is known by your client but can't be viewed. It can be seen here: http://bitcoincharts.com/bitcoin/txlist/. At this immediate moment, there are 2763 unconfirmed transactions (1454847 bytes). That would seem to me to be larger than the maximum blocksize indicated above, so the above rule about higher transaction fees may be kicking in on the miner's side for inclusion of transactions (but the client isn't prompting).

Stuff

So this means that they are purposely not including some transactions?

It looks like Ozcoin is processing all transactions, Bitparking is passing along no transactions, and Deepbit has a transaction fee threshold that prunes lower fee and/or free transactions from their blocks.

[Luke-Jr]

sub

What is with all this "sub" spam? Is there some reason you guys can't just click to stupid "notify" link next to "reply"?

[TheHarbinger]

sub

What is with all this "sub" spam? Is there some reason you guys can't just click to stupid "notify" link next to "reply"?

Yes, because the "notify" link sends an e-mail when there is a new post.  If you are watching 20 or so threads, that's 20 or so junk e-mails you get.  Where as if you make a post in the topic, "sub", then you can just click on the nice "Show new replies to your posts." at the top of the forums and get a list of all the threads you follow with new activity.

So, there is your reason.

[Graet]

It seems like part of the problem with blocks not including transactions is the maximum block size of 500kB. Doesn't this pose a scalability problem?

...
The size of the queue of unconfirmed transactions waiting for inclusion is known by your client but can't be viewed. It can be seen here: http://bitcoincharts.com/bitcoin/txlist/. At this immediate moment, there are 2763 unconfirmed transactions (1454847 bytes). That would seem to me to be larger than the maximum blocksize indicated above, so the above rule about higher transaction fees may be kicking in on the miner's side for inclusion of transactions (but the client isn't prompting).

Stuff

So this means that they are purposely not including some transactions?

It looks like Ozcoin is processing all transactions, Bitparking is passing along no transactions, and Deepbit has a transaction fee threshold that prunes lower fee and/or free transactions from their blocks.

Just doing what we can to reduce the txn backlog
More helpers welcome

[-ck]

It looks like Ozcoin is processing all transactions, Bitparking is passing along no transactions, and Deepbit has a transaction fee threshold that prunes lower fee and/or free transactions from their blocks.

The question then, is, does this make a difference to them? Are deepbit (and to a lesser extent bitparking) keeping a low orphan rate or is this barking up the wrong tree? I still get the feeling we're missing the underlying cause for this since it predates the abrupt rise in the number of transactions.

[kano]

... and more to that, that is unexplained, is for me block 184660 (and some before that) showed up in my miner 45 seconds after my independent blockcoind had 'ACCEPTED' it.
i.e. block 184660 was VERY slow propagating about the net.

Around the time of that block (before) it happened twice more that I noticed.
The one I saw before 184660 (NFI what block it was) was almost 60 seconds and the 1st I didn't notice the exact amount of time but that was what made me pay attention to it.

But there was clearly a propagation problem for those blocks (in case anyone was thinking otherwise)

bitcoind:
got 184660 at 06/15/12 13:15:08 UTC
and 'ACCEPTED' it at 13:15:17

miner (different computer):
LP was 2012-06-15 13:16:02 UTC

This also shows that my bitcoind was slow (9 seconds) to accept the block, however that can be ignored since my bitcoind has 2 extra processing modules that can slow that down.
(clearly unrelated to the problem since my bitcoind accepted the block before the miner knew about it, not after)

And, yes my clocks are always correct
I noticed the difference itself while watching both logs on the screen, and I've been using ntpd for ... years ...

[Graet]

It looks like Ozcoin is processing all transactions, Bitparking is passing along no transactions, and Deepbit has a transaction fee threshold that prunes lower fee and/or free transactions from their blocks.

The question then, is, does this make a difference to them? Are deepbit (and to a lesser extent bitparking) keeping a low orphan rate or is this barking up the wrong tree? I still get the feeling we're missing the underlying cause for this since it predates the abrupt rise in the number of transactions.

went and dug through the DB
We have had 12 orphans since we went DGM 30/12/2011
2 were our fault when we didnt get the BIP16 updates right
so 10 orphans for over 6.5 months
The last 2 were 15th and 8th of this month, whilst either or both of these may be attributed to current issues, we are not showing an unusually high orphan rate.

We spent quite some time over the last 24hours looking to see what we could do poolside to help the longpolls, stales and big block issues.
We came to the conclusion that apart from optimising some logging and databases some more to reduce load there isnt really much we can do, this is a network issue and the pools will need help from the devs to resolve.

Ozcoin will continue to accept all standard transactions into our blocks, we do not see any advantage to restricting the number of transactions in our blocks.

On a side note, value of txns included with blocks has gone from 0.0x to 0.x with a lot around 0.3 and we had one block with more than 1BTC in txn fees.

Graeme

[realnowhereman]

I've always thought the protocol was missing a `getbody` command.  But what it really needs is four commands:

• getblockfull (got this, it's called 'getdata')
• getblockheader (got this, it's called 'headers')
• getblockheaderandtxids
• getblockbody

The `inv` broadcast after a block is found should result in many `getblockheaderandtxids` requests.  Most miners will have the vast majority of the transactions in any block already, so there is no need for the entire block to be rebroadcast just so they can get the coinbase.

The others `get` messages would be for light clients and previously-not-connected clients.


Edit: It also occurs to me that this isn't a huge problem (barring the waste of bandwidth); the difficulty will adjust to take account of the orphans, since they are effectively lowering the hashing power of the miners.

[doublec]

It looks like Ozcoin is processing all transactions, Bitparking is passing along no transactions, and Deepbit has a transaction fee threshold that prunes lower fee and/or free transactions from their blocks.

The no transactions in the recent bitparking blocks was due to a bug in my attempt to lower the cpu percentage that processing transactions was consuming. I've now fixed this and it should start showing transactions again.

The issue the bitparking pool has had is the priority ordering of transactions in CreateNewBlock in main.cpp was causing lag in the pool and high percentage cpu usage of the bitcoind process when the memory pool had a large number of transactions. I have yet to investigate why since that function is supposed to only be called every 60 seconds or so. Perhaps it takes a lock that causes timeouts for miners or something. At some point I'll investigate and see if that was the cause. Changing this function so that less transactions were processed in the queue seemed to result in a large drop in cpu % (from 80% to 5%).

[jgarzik]

The most recent bitcoin/bitcoin.git has "signature verification cache" which should speed up block validation quite a bit.

[Prattler]

P2pool now transmits the block header, so it's possible to see how much earlier
p2pool (150+ peers) saw the block compared to my local bitcoind. This gives
a decent estimation how fast blocks propagate.
 
http://blockchain.info/block-index/239691/000000000000035b015bc1b9cd6e5cbfa4e37371ebae7f444f6774428dcb36c8
block size: 396.08 kB
My bitcoind received it 29 seconds later than p2pool.
 
http://blockchain.info/block-index/239693/00000000000006a3bb36582136c46daad07991460b800534d0195ac99bed939b
block size: 223.01 kB
My bitcoind received it 16 seconds later than p2pool.
 
From what I've seen, the time varies from 0 s to more than a minute, mostly
depending on the block size, but also bitcoind connectivity.

I'd conclude that ATM bitcoin network cannot handle 250+ kB blocks reliably.
No sane pool operator should make bigger than 100 kB blocks with low
0.0005 BTC/kB fees.

Bitcoind:    "connections" : 54, -- so it's well connected, with 10
connections it would be much worse.

My PC is good with no load, internet connection good (10-20 Mbit/s). This is
a bitcoin network thing.

[spiccioli]

My PC is good with no load, internet connection good (10-20 Mbit/s). This is
a bitcoin network thing.

Well, a standard ADSL here in (a big city of) Italy is something like 10-20Mbit/256-512Kbit, so a 400-500Kb block requires 8-10 seconds just to be sent to the next peer.

If you are sending it to 10 peers at the same time it takes more than one minute, so it seems that sending a smaller header would help a lot.

spiccioli

[-ck]

The most recent bitcoin/bitcoin.git has "signature verification cache" which should speed up block validation quite a bit.

That's very interesting and if it makes a big difference should be a fairly urgent update for pool owners if the problem is as big as it appears to be.

[Luke-Jr]

The most recent bitcoin/bitcoin.git has "signature verification cache" which should speed up block validation quite a bit.

That's very interesting and if it makes a big difference should be a fairly urgent update for pool owners if the problem is as big as it appears to be.

It's mostly the rest of the network that needs to upgrade for this to be of any benefit.