Bitcoin Forum
December 11, 2016, 10:04:47 AM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: « 1 2 [3] 4 5 »  All
  Print  
Author Topic: High orphan rate and long confirmation time discussion  (Read 8774 times)
Tittiez
Hero Member
*****
Offline Offline

Activity: 686



View Profile
June 18, 2012, 05:47:41 PM
 #41

subbb
1481450687
Hero Member
*
Offline Offline

Posts: 1481450687

View Profile Personal Message (Offline)

Ignore
1481450687
Reply with quote  #2

1481450687
Report to moderator
1481450687
Hero Member
*
Offline Offline

Posts: 1481450687

View Profile Personal Message (Offline)

Ignore
1481450687
Reply with quote  #2

1481450687
Report to moderator
1481450687
Hero Member
*
Offline Offline

Posts: 1481450687

View Profile Personal Message (Offline)

Ignore
1481450687
Reply with quote  #2

1481450687
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481450687
Hero Member
*
Offline Offline

Posts: 1481450687

View Profile Personal Message (Offline)

Ignore
1481450687
Reply with quote  #2

1481450687
Report to moderator
1481450687
Hero Member
*
Offline Offline

Posts: 1481450687

View Profile Personal Message (Offline)

Ignore
1481450687
Reply with quote  #2

1481450687
Report to moderator
1481450687
Hero Member
*
Offline Offline

Posts: 1481450687

View Profile Personal Message (Offline)

Ignore
1481450687
Reply with quote  #2

1481450687
Report to moderator
mdude77
Legendary
*
Offline Offline

Activity: 1358


View Profile
June 18, 2012, 09:08:01 PM
 #42


Sub'd.

M

MMinerMonitor author, monitor/auto/schedule reboots/alerts/remote/MobileMiner for Ants and Spondoolies! Latest (5.2). MPoolMonitor author, monitor stats/workers for most pools, global BTC stats (current/nxt diff/USD val/hashrate/calc)! Latest (v4.2) 
Buyer beware of Bitmain hardware and services.
kano
Legendary
*
Offline Offline

Activity: 1932


Linux since 1997 RedHat 4


View Profile
June 18, 2012, 09:15:36 PM
 #43

It seems like part of the problem with blocks not including transactions is the maximum block size of 500kB. Doesn't this pose a scalability problem?

...
The size of the queue of unconfirmed transactions waiting for inclusion is known by your client but can't be viewed. It can be seen here: http://bitcoincharts.com/bitcoin/txlist/. At this immediate moment, there are 2763 unconfirmed transactions (1454847 bytes). That would seem to me to be larger than the maximum blocksize indicated above, so the above rule about higher transaction fees may be kicking in on the miner's side for inclusion of transactions (but the client isn't prompting).
The size of the 0-confirm queue is shown in debug.log every time addUnchecked() is called just before a txn is stored in memory pool.
And that is called every time - so you can watch it count up with each txn or orphan and drop down with each block
e.g. just now: 06/18/12 21:13:21 addUnchecked(): size 1141
That 1141 is the number of txns in the memory pool - that is the number of 0-confirm txns

Edit: though it is only correct if your bitcoind has been running for a while since you wont know all 0-confirm txns if you just started your bitcoind

Edit2: see here's what happens in debug.log when a block is found - just now - Ozcoin (with other stuff around it removed):
06/18/12 21:24:40 addUnchecked(): size 1465
06/18/12 21:24:50 SetBestChain: new best=000000000000007b3cb3  height=185186  work=359073512741370072393
06/18/12 21:24:50 addUnchecked(): size 1018

So that block (185186) processed 448 txn's from the memory pool (+1 coinbase that wasn't in the memory pool as per expected)
244.80kB in size

Edit3: however the next block by Bitparking was (they ignored all 0-confirm txns):
06/18/12 21:30:13 addUnchecked(): size 1114
06/18/12 21:30:15 SetBestChain: new best=000000000000046b404d  height=185187  work=359080312542205322700
06/18/12 21:30:19 addUnchecked(): size 1115

That block (185187) processed 0 txn's from the memory pool
0.25kB in size

Edit4: the next block was again Ozcoin
06/18/12 21:46:40 addUnchecked(): size 1460
06/18/12 21:46:56 SetBestChain: new best=0000000000000732fdc3  height=185188  work=359087112343040573007
06/18/12 21:46:57 addUnchecked(): size 822

So that block (185188) processed 639 txn's from the memory pool (+1 coinbase ...)
243.31kB in size

Edit5: and for the last edit Smiley - the next block was Deepbit
06/18/12 21:49:04 addUnchecked(): size 862
06/18/12 21:49:11 SetBestChain: new best=000000000000085b53c3  height=185189  work=359093912143875823314
06/18/12 21:49:12 addUnchecked(): size 689

So that block (185189) processed 174 txn's from the memory pool (+1 coinbase ...)
49.80kB in size

Pool: https://kano.is BTC: 1KanoiBupPiZfkwqB7rfLXAzPnoTshAVmb
CKPool and CGMiner developer, IRC FreeNode #ckpool and #cgminer kanoi
Help keep Bitcoin secure by mining on pools with Stratum, the best protocol to mine Bitcoins with ASIC hardware
Raize
Donator
Legendary
*
Offline Offline

Activity: 1375


View Profile
June 18, 2012, 09:34:36 PM
 #44

What incentives are there toward propagating blocks? Does the code look for a different node if one node appears to be delaying sending a specific block?

The reason why I ask is this, it would seem that if you are a mining pool that has a reasonable idea of the hashing power of Deepbit, you could take a block they've solved in a faster timeframe than normal and ignore sending it on immediately, instead mining on it for some time simply to get an advantage over the other mining pools that don't yet have the block. Sure, you run the risk of orphaning the block and making your work irrelevant, and it is likely that other nodes connected to Deepbit would send it on fast enough anyway, but the less clients connected to the Bitcoin network, the greater advantage withholding the block nets you.

I'm not saying such a thing is happening here, but there would seem to be some advantage to such a delay.

OrganofCorti's Neighbourhood Pool Watch - The most informative website on blockchain health
kano
Legendary
*
Offline Offline

Activity: 1932


Linux since 1997 RedHat 4


View Profile
June 18, 2012, 09:38:54 PM
 #45

There is the risk that you don't propagate your block fast enough if some other pool finds a block you are witholding.
A 50 BTC risk.

Pool: https://kano.is BTC: 1KanoiBupPiZfkwqB7rfLXAzPnoTshAVmb
CKPool and CGMiner developer, IRC FreeNode #ckpool and #cgminer kanoi
Help keep Bitcoin secure by mining on pools with Stratum, the best protocol to mine Bitcoins with ASIC hardware
Gladamas
Sr. Member
****
Offline Offline

Activity: 294


Bitcoin today is what the internet was in 1998.


View Profile
June 18, 2012, 10:31:27 PM
 #46

It seems like part of the problem with blocks not including transactions is the maximum block size of 500kB. Doesn't this pose a scalability problem?

...
The size of the queue of unconfirmed transactions waiting for inclusion is known by your client but can't be viewed. It can be seen here: http://bitcoincharts.com/bitcoin/txlist/. At this immediate moment, there are 2763 unconfirmed transactions (1454847 bytes). That would seem to me to be larger than the maximum blocksize indicated above, so the above rule about higher transaction fees may be kicking in on the miner's side for inclusion of transactions (but the client isn't prompting).
The size of the 0-confirm queue is shown in debug.log every time addUnchecked() is called just before a txn is stored in memory pool.
And that is called every time - so you can watch it count up with each txn or orphan and drop down with each block
e.g. just now: 06/18/12 21:13:21 addUnchecked(): size 1141
That 1141 is the number of txns in the memory pool - that is the number of 0-confirm txns

Edit: though it is only correct if your bitcoind has been running for a while since you wont know all 0-confirm txns if you just started your bitcoind

Edit2: see here's what happens in debug.log when a block is found - just now - Ozcoin (with other stuff around it removed):
06/18/12 21:24:40 addUnchecked(): size 1465
06/18/12 21:24:50 SetBestChain: new best=000000000000007b3cb3  height=185186  work=359073512741370072393
06/18/12 21:24:50 addUnchecked(): size 1018

So that block (185186) processed 448 txn's from the memory pool (+1 coinbase that wasn't in the memory pool as per expected)
244.80kB in size

Edit3: however the next block by Bitparking was (they ignored all 0-confirm txns):
06/18/12 21:30:13 addUnchecked(): size 1114
06/18/12 21:30:15 SetBestChain: new best=000000000000046b404d  height=185187  work=359080312542205322700
06/18/12 21:30:19 addUnchecked(): size 1115

That block (185187) processed 0 txn's from the memory pool
0.25kB in size

Edit4: the next block was again Ozcoin
06/18/12 21:46:40 addUnchecked(): size 1460
06/18/12 21:46:56 SetBestChain: new best=0000000000000732fdc3  height=185188  work=359087112343040573007
06/18/12 21:46:57 addUnchecked(): size 822

So that block (185188) processed 639 txn's from the memory pool (+1 coinbase ...)
243.31kB in size

Edit5: and for the last edit Smiley - the next block was Deepbit
06/18/12 21:49:04 addUnchecked(): size 862
06/18/12 21:49:11 SetBestChain: new best=000000000000085b53c3  height=185189  work=359093912143875823314
06/18/12 21:49:12 addUnchecked(): size 689

So that block (185189) processed 174 txn's from the memory pool (+1 coinbase ...)
49.80kB in size

So this means that they are purposely not including some transactions?

1GLADMZ5tL4HkS6BAWPfJLeZJCDHAd9Fr3 - LQ6Zx8v7fHVBiDX5Lmhbp6oEDB7dUFjANu
GPG 0xF219D5BB3C467E12 - Litecoin Forum
TheHarbinger
Sr. Member
****
Offline Offline

Activity: 378


Why is it so damn hot in here?


View Profile
June 18, 2012, 10:35:30 PM
 #47

It seems like part of the problem with blocks not including transactions is the maximum block size of 500kB. Doesn't this pose a scalability problem?

...
The size of the queue of unconfirmed transactions waiting for inclusion is known by your client but can't be viewed. It can be seen here: http://bitcoincharts.com/bitcoin/txlist/. At this immediate moment, there are 2763 unconfirmed transactions (1454847 bytes). That would seem to me to be larger than the maximum blocksize indicated above, so the above rule about higher transaction fees may be kicking in on the miner's side for inclusion of transactions (but the client isn't prompting).
Stuff

So this means that they are purposely not including some transactions?

It looks like Ozcoin is processing all transactions, Bitparking is passing along no transactions, and Deepbit has a transaction fee threshold that prunes lower fee and/or free transactions from their blocks.

12Um6jfDE7q6crm1s6tSksMvda8s1hZ3Vj
Luke-Jr
Legendary
*
Offline Offline

Activity: 2100



View Profile
June 18, 2012, 11:17:28 PM
 #48

sub
What is with all this "sub" spam? Is there some reason you guys can't just click to stupid "notify" link next to "reply"?

TheHarbinger
Sr. Member
****
Offline Offline

Activity: 378


Why is it so damn hot in here?


View Profile
June 18, 2012, 11:21:10 PM
 #49

sub
What is with all this "sub" spam? Is there some reason you guys can't just click to stupid "notify" link next to "reply"?

Yes, because the "notify" link sends an e-mail when there is a new post.  If you are watching 20 or so threads, that's 20 or so junk e-mails you get.  Where as if you make a post in the topic, "sub", then you can just click on the nice "Show new replies to your posts." at the top of the forums and get a list of all the threads you follow with new activity.

So, there is your reason.

12Um6jfDE7q6crm1s6tSksMvda8s1hZ3Vj
Graet
VIP
Legendary
*
Offline Offline

Activity: 980



View Profile WWW
June 18, 2012, 11:26:24 PM
 #50

It seems like part of the problem with blocks not including transactions is the maximum block size of 500kB. Doesn't this pose a scalability problem?

...
The size of the queue of unconfirmed transactions waiting for inclusion is known by your client but can't be viewed. It can be seen here: http://bitcoincharts.com/bitcoin/txlist/. At this immediate moment, there are 2763 unconfirmed transactions (1454847 bytes). That would seem to me to be larger than the maximum blocksize indicated above, so the above rule about higher transaction fees may be kicking in on the miner's side for inclusion of transactions (but the client isn't prompting).
Stuff

So this means that they are purposely not including some transactions?

It looks like Ozcoin is processing all transactions, Bitparking is passing along no transactions, and Deepbit has a transaction fee threshold that prunes lower fee and/or free transactions from their blocks.
Just doing what we can to reduce the txn backlog Cheesy
More helpers welcome Smiley




| Ozcoin Pooled Mining Pty Ltd https://ozcoin.net Double Geometric Reward System https://lc.ozcoin.net for Litecoin mining DGM| https://crowncloud.net VPS and Dedicated Servers for the BTC community
-ck
Moderator
Legendary
*
Offline Offline

Activity: 2002


Ruu \o/


View Profile WWW
June 19, 2012, 09:25:38 AM
 #51

It looks like Ozcoin is processing all transactions, Bitparking is passing along no transactions, and Deepbit has a transaction fee threshold that prunes lower fee and/or free transactions from their blocks.
The question then, is, does this make a difference to them? Are deepbit (and to a lesser extent bitparking) keeping a low orphan rate or is this barking up the wrong tree? I still get the feeling we're missing the underlying cause for this since it predates the abrupt rise in the number of transactions.

Primary developer/maintainer for cgminer and ckpool/ckproxy.
Pooled mine at kano.is, solo mine at solo.ckpool.org
-ck
kano
Legendary
*
Offline Offline

Activity: 1932


Linux since 1997 RedHat 4


View Profile
June 19, 2012, 09:53:57 AM
 #52

... and more to that, that is unexplained, is for me block 184660 (and some before that) showed up in my miner 45 seconds after my independent blockcoind had 'ACCEPTED' it.
i.e. block 184660 was VERY slow propagating about the net.

Around the time of that block (before) it happened twice more that I noticed.
The one I saw before 184660 (NFI what block it was) was almost 60 seconds and the 1st I didn't notice the exact amount of time but that was what made me pay attention to it.

But there was clearly a propagation problem for those blocks (in case anyone was thinking otherwise)

bitcoind:
got 184660 at 06/15/12 13:15:08 UTC
and 'ACCEPTED' it at 13:15:17

miner (different computer):
LP was 2012-06-15 13:16:02 UTC

This also shows that my bitcoind was slow (9 seconds) to accept the block, however that can be ignored since my bitcoind has 2 extra processing modules that can slow that down.
(clearly unrelated to the problem since my bitcoind accepted the block before the miner knew about it, not after)

And, yes my clocks are always correct Smiley
I noticed the difference itself while watching both logs on the screen, and I've been using ntpd for ... years ...

Pool: https://kano.is BTC: 1KanoiBupPiZfkwqB7rfLXAzPnoTshAVmb
CKPool and CGMiner developer, IRC FreeNode #ckpool and #cgminer kanoi
Help keep Bitcoin secure by mining on pools with Stratum, the best protocol to mine Bitcoins with ASIC hardware
Graet
VIP
Legendary
*
Offline Offline

Activity: 980



View Profile WWW
June 19, 2012, 10:00:11 AM
 #53

It looks like Ozcoin is processing all transactions, Bitparking is passing along no transactions, and Deepbit has a transaction fee threshold that prunes lower fee and/or free transactions from their blocks.
The question then, is, does this make a difference to them? Are deepbit (and to a lesser extent bitparking) keeping a low orphan rate or is this barking up the wrong tree? I still get the feeling we're missing the underlying cause for this since it predates the abrupt rise in the number of transactions.

went and dug through the DB
We have had 12 orphans since we went DGM 30/12/2011
2 were our fault when we didnt get the BIP16 updates right
so 10 orphans for over 6.5 months
The last 2 were 15th and 8th of this month, whilst either or both of these may be attributed to current issues, we are not showing an unusually high orphan rate.

We spent quite some time over the last 24hours looking to see what we could do poolside to help the longpolls, stales and big block issues.
We came to the conclusion that apart from optimising some logging and databases some more to reduce load there isnt really much we can do, this is a network issue and the pools will need help from the devs to resolve.

Ozcoin will continue to accept all standard transactions into our blocks, we do not see any advantage to restricting the number of transactions in our blocks.

On a side note, value of txns included with blocks has gone from 0.0x to 0.x with a lot around 0.3 and we had one block with more than 1BTC in txn fees.

Graeme


| Ozcoin Pooled Mining Pty Ltd https://ozcoin.net Double Geometric Reward System https://lc.ozcoin.net for Litecoin mining DGM| https://crowncloud.net VPS and Dedicated Servers for the BTC community
realnowhereman
Hero Member
*****
Offline Offline

Activity: 504



View Profile
June 19, 2012, 11:54:17 AM
 #54

I've always thought the protocol was missing a `getbody` command.  But what it really needs is four commands:

  • getblockfull (got this, it's called 'getdata')
  • getblockheader (got this, it's called 'headers')
  • getblockheaderandtxids
  • getblockbody

The `inv` broadcast after a block is found should result in many `getblockheaderandtxids` requests.  Most miners will have the vast majority of the transactions in any block already, so there is no need for the entire block to be rebroadcast just so they can get the coinbase.

The others `get` messages would be for light clients and previously-not-connected clients.



Edit: It also occurs to me that this isn't a huge problem (barring the waste of bandwidth); the difficulty will adjust to take account of the orphans, since they are effectively lowering the hashing power of the miners.

1AAZ4xBHbiCr96nsZJ8jtPkSzsg1CqhwDa
doublec
Legendary
*
Offline Offline

Activity: 1078


View Profile
June 19, 2012, 11:54:41 AM
 #55

It looks like Ozcoin is processing all transactions, Bitparking is passing along no transactions, and Deepbit has a transaction fee threshold that prunes lower fee and/or free transactions from their blocks.

The no transactions in the recent bitparking blocks was due to a bug in my attempt to lower the cpu percentage that processing transactions was consuming. I've now fixed this and it should start showing transactions again.

The issue the bitparking pool has had is the priority ordering of transactions in CreateNewBlock in main.cpp was causing lag in the pool and high percentage cpu usage of the bitcoind process when the memory pool had a large number of transactions. I have yet to investigate why since that function is supposed to only be called every 60 seconds or so. Perhaps it takes a lock that causes timeouts for miners or something. At some point I'll investigate and see if that was the cause. Changing this function so that less transactions were processed in the queue seemed to result in a large drop in cpu % (from 80% to 5%).
jgarzik
Legendary
*
Offline Offline

Activity: 1470


View Profile
June 19, 2012, 12:12:08 PM
 #56


The most recent bitcoin/bitcoin.git has "signature verification cache" which should speed up block validation quite a bit.


Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
Prattler
Full Member
***
Offline Offline

Activity: 192


View Profile
June 19, 2012, 07:45:07 PM
 #57

P2pool now transmits the block header, so it's possible to see how much earlier
p2pool (150+ peers) saw the block compared to my local bitcoind. This gives
a decent estimation how fast blocks propagate.
 
http://blockchain.info/block-index/239691/000000000000035b015bc1b9cd6e5cbfa4e37371ebae7f444f6774428dcb36c8
block size: 396.08 kB
My bitcoind received it 29 seconds later than p2pool.
 
http://blockchain.info/block-index/239693/00000000000006a3bb36582136c46daad07991460b800534d0195ac99bed939b
block size: 223.01 kB
My bitcoind received it 16 seconds later than p2pool.
 
From what I've seen, the time varies from 0 s to more than a minute, mostly
depending on the block size, but also bitcoind connectivity.

I'd conclude that ATM bitcoin network cannot handle 250+ kB blocks reliably.
No sane pool operator should make bigger than 100 kB blocks with low
0.0005 BTC/kB fees.

Bitcoind:    "connections" : 54, -- so it's well connected, with 10
connections it would be much worse.

My PC is good with no load, internet connection good (10-20 Mbit/s). This is
a bitcoin network thing.
spiccioli
Legendary
*
Offline Offline

Activity: 1376

nec sine labore


View Profile
June 19, 2012, 08:21:05 PM
 #58

My PC is good with no load, internet connection good (10-20 Mbit/s). This is
a bitcoin network thing.

Well, a standard ADSL here in (a big city of) Italy is something like 10-20Mbit/256-512Kbit, so a 400-500Kb block requires 8-10 seconds just to be sent to the next peer.

If you are sending it to 10 peers at the same time it takes more than one minute, so it seems that sending a smaller header would help a lot.

spiccioli
-ck
Moderator
Legendary
*
Offline Offline

Activity: 2002


Ruu \o/


View Profile WWW
June 19, 2012, 09:53:01 PM
 #59


The most recent bitcoin/bitcoin.git has "signature verification cache" which should speed up block validation quite a bit.


That's very interesting and if it makes a big difference should be a fairly urgent update for pool owners if the problem is as big as it appears to be.

Primary developer/maintainer for cgminer and ckpool/ckproxy.
Pooled mine at kano.is, solo mine at solo.ckpool.org
-ck
Luke-Jr
Legendary
*
Offline Offline

Activity: 2100



View Profile
June 19, 2012, 10:17:49 PM
 #60


The most recent bitcoin/bitcoin.git has "signature verification cache" which should speed up block validation quite a bit.


That's very interesting and if it makes a big difference should be a fairly urgent update for pool owners if the problem is as big as it appears to be.
It's mostly the rest of the network that needs to upgrade for this to be of any benefit.

Pages: « 1 2 [3] 4 5 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!