Bitcoin Forum
December 03, 2016, 05:56:57 PM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: « 1 2 3 [4] 5 »  All
  Print  
Author Topic: High orphan rate and long confirmation time discussion  (Read 8754 times)
organofcorti
Donator
Legendary
*
Offline Offline

Activity: 1946


Poor impulse control.


View Profile WWW
June 19, 2012, 10:20:04 PM
 #61


The most recent bitcoin/bitcoin.git has "signature verification cache" which should speed up block validation quite a bit.


That's very interesting and if it makes a big difference should be a fairly urgent update for pool owners if the problem is as big as it appears to be.
It's mostly the rest of the network that needs to upgrade for this to be of any benefit.

I think he meant that should be a fairly urgent update in order to reduce the problem for pool owners.

Bitcoin network and pool analysis 12QxPHEuxDrs7mCyGSx1iVSozTwtquDB3r
follow @oocBlog for new post notifications
1480787817
Hero Member
*
Offline Offline

Posts: 1480787817

View Profile Personal Message (Offline)

Ignore
1480787817
Reply with quote  #2

1480787817
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1480787817
Hero Member
*
Offline Offline

Posts: 1480787817

View Profile Personal Message (Offline)

Ignore
1480787817
Reply with quote  #2

1480787817
Report to moderator
1480787817
Hero Member
*
Offline Offline

Posts: 1480787817

View Profile Personal Message (Offline)

Ignore
1480787817
Reply with quote  #2

1480787817
Report to moderator
kano
Legendary
*
Offline Offline

Activity: 1918


Linux since 1997 RedHat 4


View Profile
June 19, 2012, 10:32:00 PM
 #62

There are two parts to slowing down network propagation:
1) Sending crap repeatedly
2) Wasting time before sending it

It does seem rather odd that (even after I mentioned it) people are still debating what is actually the problem and looking for answers ...
Is it really that hard to put some debug in bitcoind and send a big block from one place to another and see the results at both ends?
Repeat and rinse.
Is this sort of testing beyond the level of bitcoin devs?

Pool: https://kano.is BTC: 1KanoiBupPiZfkwqB7rfLXAzPnoTshAVmb
CKPool and CGMiner developer, IRC FreeNode #ckpool and #cgminer kanoi
Help keep Bitcoin secure by mining on pools with Stratum, the best protocol to mine Bitcoins with ASIC hardware
deepceleron
Legendary
*
Offline Offline

Activity: 1470



View Profile WWW
June 19, 2012, 10:49:37 PM
 #63

The first poster discovering the client-block-receive to miner-long-poll-receive delay time reveals another factor unrelated to the Bitcoin network block propagation speed - the time it takes for a pool to transmit a new block to all miners. In my past experience, depending on the number of pool miners, the number of pool servers, and the pool software and it's delays communicating with Bitcoin, going through the list of miners and sending each a LP response can take a significant amount of time, generally four, but up to ten seconds from the first to the last miner being notified. This from my own research profiling pool LP times as a predictor of who found the block.

If large block propagation was a significant factor, you would expect to see pools, more highly connected and on faster connections, sending your miner new block notification before your Bitcoin client gets it through the P2P network. That it is the opposite indicates instead the internal pool latency in pushing new work to all miners.

We have had 12 orphans since we went DGM 30/12/2011
2 were our fault when we didnt get the BIP16 updates right
so 10 orphans for over 6.5 months
The last 2 were 15th and 8th of this month, whilst either or both of these may be attributed to current issues, we are not showing an unusually high orphan rate.

We spent quite some time over the last 24hours looking to see what we could do poolside to help the longpolls, stales and big block issues.
We came to the conclusion that apart from optimising some logging and databases some more to reduce load there isnt really much we can do, this is a network issue and the pools will need help from the devs to resolve.

Ozcoin will continue to accept all standard transactions into our blocks, we do not see any advantage to restricting the number of transactions in our blocks.
....

One could make the argument that if your pool and a 0-tx mystery miner both generated a block at the same time, the much smaller block could propagate to miners first and be built upon first with another fast block find. The chief optimization, if you are dedicated to creating larger blocks than other miners may be creating, would be to ensure all your pool servers are highly connected to other pools, and communication of new blocks to other pools' known IPs takes priority. Pools should publish and share IPs of bitcoind's with each other (best it be a closed/trusted list to prevent DDOS facilitation), creating a high-priority sub-network to lower orphans for all.

Another coding optimization would be if you receive a second block solution, to immediately discard the first and push out the newer one if it has a significantly smaller hash, of course still keeping track of both as valid work from miners.

It seems like part of the problem with blocks not including transactions is the maximum block size of 500kB. Doesn't this pose a scalability problem?

There are also some odd rules mentioned in the Wiki that increase fees when the queue of transactions gets too deep. I haven't reviewed the Bitcoin code to comment further how these are enforced by miners and when sending in the client:
"If the blocksize (size of all transactions currently waiting to be included in a block) is less than 27 kB, transactions are free.
If the blocksize is more than 250 kB, transactions get increasingly more expensive as the blocksize approaches the limit of 500 kB."


The size of the queue of unconfirmed transactions waiting for inclusion is known by your client but can't be viewed. It can be seen here: http://bitcoincharts.com/bitcoin/txlist/. At this immediate moment, there are 2763 unconfirmed transactions (1454847 bytes). That would seem to me to be larger than the maximum blocksize indicated above, so the above rule about higher transaction fees may be kicking in on the miner's side for inclusion of transactions (but the client isn't prompting).
I would caution people to only check this page when you really need to - listing the current 2342781 byte backlog of transactions makes for 7.7MB of HTML.

Gavin Andresen
Legendary
*
Offline Offline

Activity: 1652


Chief Scientist


View Profile WWW
June 20, 2012, 12:14:56 AM
 #64

Another coding optimization would be if you receive a second block solution, to immediately discard the first and push out the newer one if it has a significantly smaller hash, of course still keeping track of both as valid work from miners.
Chain proof-of-work is calculated based on the hash target, so if you get another block at the same height there is no benefit to keeping the one with "the smaller hash".

Maybe if you receive a second block solution, keeping the block that removes the most transactions from the memory pool would be the right "good for the entire ecosystem" policy.  That way even if small blocks propagate slightly faster that might be offset if a larger, slower block was found.  (but making new-block-flooding independent of the size of the block is an even better solution, and that shouldn't be too hard to implement)

Creating a semi-trusted backbone of connections to other pools/miners so your new blocks propagate quickly is a good idea.

How often do you get the chance to work on a potentially world-changing project?
doublec
Legendary
*
Offline Offline

Activity: 1078


View Profile
June 20, 2012, 12:35:00 AM
 #65

The first poster discovering the client-block-receive to miner-long-poll-receive delay time reveals another factor unrelated to the Bitcoin network block propagation speed - the time it takes for a pool to transmit a new block to all miners. In my past experience, depending on the number of pool miners, the number of pool servers, and the pool software and it's delays communicating with Bitcoin, going through the list of miners and sending each a LP response can take a significant amount of time, generally four, but up to ten seconds from the first to the last miner being notified. This from my own research profiling pool LP times as a predictor of who found the block.
I agree, longpoll notification is slow. I wonder if using a lightweight message queuing system like zeromq would do a better job instead of the http based long polling.
kano
Legendary
*
Offline Offline

Activity: 1918


Linux since 1997 RedHat 4


View Profile
June 20, 2012, 12:38:56 AM
 #66

The first poster discovering the client-block-receive to miner-long-poll-receive delay time reveals another factor unrelated to the Bitcoin network block propagation speed - the time it takes for a pool to transmit a new block to all miners. In my past experience, depending on the number of pool miners, the number of pool servers, and the pool software and it's delays communicating with Bitcoin, going through the list of miners and sending each a LP response can take a significant amount of time, generally four, but up to ten seconds from the first to the last miner being notified. This from my own research profiling pool LP times as a predictor of who found the block.
I agree, longpoll notification is slow. I wonder if using a lightweight message queuing system like zeromq would do a better job instead of the http based long polling.
... implement a protocol rather than use something designed for web servers ...

Pool: https://kano.is BTC: 1KanoiBupPiZfkwqB7rfLXAzPnoTshAVmb
CKPool and CGMiner developer, IRC FreeNode #ckpool and #cgminer kanoi
Help keep Bitcoin secure by mining on pools with Stratum, the best protocol to mine Bitcoins with ASIC hardware
rjk
Sr. Member
****
Offline Offline

Activity: 420


1ngldh


View Profile
June 20, 2012, 12:42:06 AM
 #67

Creating a semi-trusted backbone of connections to other pools/miners so your new blocks propagate quickly is a good idea.
^ This +1. Any smart pool should be blasting its blocks out across as many nodes as it is able to as quickly as possible. One way to do this is to manually connect to several reliable super nodes.

Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
jgarzik
Legendary
*
Offline Offline

Activity: 1470


View Profile
June 20, 2012, 12:49:37 AM
 #68

^ This +1. Any smart pool should be blasting its blocks out across as many nodes as it is able to as quickly as possible. One way to do this is to manually connect to several reliable super nodes.

An effort to create a bitcoin backbone is now underway...


Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
-ck
Moderator
Legendary
*
Offline Offline

Activity: 1988


Ruu \o/


View Profile WWW
June 20, 2012, 01:02:26 AM
 #69

The first poster discovering the client-block-receive to miner-long-poll-receive delay time reveals another factor unrelated to the Bitcoin network block propagation speed - the time it takes for a pool to transmit a new block to all miners. In my past experience, depending on the number of pool miners, the number of pool servers, and the pool software and it's delays communicating with Bitcoin, going through the list of miners and sending each a LP response can take a significant amount of time, generally four, but up to ten seconds from the first to the last miner being notified. This from my own research profiling pool LP times as a predictor of who found the block.
I agree, longpoll notification is slow. I wonder if using a lightweight message queuing system like zeromq would do a better job instead of the http based long polling.
... implement a protocol rather than use something designed for web servers ...
longpoll is not the issue, but merely a symptom of other issues. Fix those and longpoll will still suffice... that and I'd be really pissed if I had to support something else after spending months getting longpoll working a treat in cgminer.

Primary developer/maintainer for cgminer and ckpool/ckproxy.
Pooled mine at kano.is, solo mine at solo.ckpool.org
-ck
deepceleron
Legendary
*
Offline Offline

Activity: 1470



View Profile WWW
June 20, 2012, 02:03:27 AM
 #70

Another coding optimization would be if you receive a second block solution, to immediately discard the first and push out the newer one if it has a significantly smaller hash, of course still keeping track of both as valid work from miners.
Chain proof-of-work is calculated based on the hash target, so if you get another block at the same height there is no benefit to keeping the one with "the smaller hash".

Maybe if you receive a second block solution, keeping the block that removes the most transactions from the memory pool would be the right "good for the entire ecosystem" policy.  That way even if small blocks propagate slightly faster that might be offset if a larger, slower block was found.  (but making new-block-flooding independent of the size of the block is an even better solution, and that shouldn't be too hard to implement)

Creating a semi-trusted backbone of connections to other pools/miners so your new blocks propagate quickly is a good idea.
I had gotten the impression somewhere that "most difficult chain wins" meant the hash difficulties, not target difficulties. If that is not the case, the only time that there would be a more difficult chain would be on difficulty changes - i.e. if you had received two block 183456s (a block with a difficulty recalc), one may have a more difficult target calculated by its miner. A unique hack that might make your recalc block win an orphan race (once in a million) would be to round up the difficulty recalc within the limits of network acceptance. Outside of that, if the only policy Bitcoin miners use is "first new block received", orphan prevention would seem to be just a race to publication then, that you tell all other miners about your block before they find their own or receive a block from another pool.

mrb
Legendary
*
Offline Offline

Activity: 1106


View Profile WWW
June 20, 2012, 09:07:23 AM
 #71

sub
What is with all this "sub" spam? Is there some reason you guys can't just click to stupid "notify" link next to "reply"?

Yes, because the "notify" link sends an e-mail when there is a new post.  If you are watching 20 or so threads, that's 20 or so junk e-mails you get.  Where as if you make a post in the topic, "sub", then you can just click on the nice "Show new replies to your posts." at the top of the forums and get a list of all the threads you follow with new activity.

I concur with TheHarbinger. I wish forum admins would install the SMF Bookmark Mod, which provides the same user experience as "subscribing" to a thread.
TheHarbinger
Sr. Member
****
Offline Offline

Activity: 378


Why is it so damn hot in here?


View Profile
June 21, 2012, 12:33:25 AM
 #72

1200 Unconfirmed transactions as I post this.   Undecided

12Um6jfDE7q6crm1s6tSksMvda8s1hZ3Vj
kano
Legendary
*
Offline Offline

Activity: 1918


Linux since 1997 RedHat 4


View Profile
June 21, 2012, 12:41:04 AM
 #73

1200 Unconfirmed transactions as I post this.   Undecided
~1000 isn't rare at the moment.
I restarted my bitcoind 12 hours ago - but the highest I've got since then was only 1338 - so not as bad as some days.
... 1 less and it would have been 1337 Smiley

Pool: https://kano.is BTC: 1KanoiBupPiZfkwqB7rfLXAzPnoTshAVmb
CKPool and CGMiner developer, IRC FreeNode #ckpool and #cgminer kanoi
Help keep Bitcoin secure by mining on pools with Stratum, the best protocol to mine Bitcoins with ASIC hardware
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
June 21, 2012, 12:48:54 AM
 #74

I had gotten the impression somewhere that "most difficult chain wins" meant the hash difficulties, not target difficulties. If that is not the case, the only time that there would be a more difficult chain would be on difficulty changes - i.e. if you had received two block 183456s (a block with a difficulty recalc), one may have a more difficult target calculated by its miner. A unique hack that might make your recalc block win an orphan race (once in a million) would be to round up the difficulty recalc within the limits of network acceptance. Outside of that, if the only policy Bitcoin miners use is "first new block received", orphan prevention would seem to be just a race to publication then, that you tell all other miners about your block before they find their own or receive a block from another pool.

No it is target difficulty.  Generally speaking longest == more difficult.  The reason why the network checks difficulty and not raw length is because otherwise an attacker could exploit the dynamic difficulty and pull off a 51% attack with <51% of hashing power.  For most intents and purposes outside of possible attack longest == most difficult but the code only looks at the sum of the chain's block target difficulty to determine the best chain.
TheHarbinger
Sr. Member
****
Offline Offline

Activity: 378


Why is it so damn hot in here?


View Profile
June 21, 2012, 01:08:44 AM
 #75

1200 Unconfirmed transactions as I post this.   Undecided
~1000 isn't rare at the moment.
I restarted my bitcoind 12 hours ago - but the highest I've got since then was only 1338 - so not as bad as some days.
... 1 less and it would have been 1337 Smiley

We need Ozcoin to hit a block to clear the backlog.   Grin

12Um6jfDE7q6crm1s6tSksMvda8s1hZ3Vj
Graet
VIP
Legendary
*
Offline Offline

Activity: 980



View Profile WWW
June 21, 2012, 01:24:17 AM
 #76

1200 Unconfirmed transactions as I post this.   Undecided
~1000 isn't rare at the moment.
I restarted my bitcoind 12 hours ago - but the highest I've got since then was only 1338 - so not as bad as some days.
... 1 less and it would have been 1337 Smiley

We need Ozcoin to hit a block to clear the backlog.   Grin
lol we did have a nice run while I slept
currently 704 Unconfirmed Transaction as I post, was ~300 when I woke up Tongue

| Ozcoin Pooled Mining Pty Ltd https://ozcoin.net Double Geometric Reward System https://lc.ozcoin.net for Litecoin mining DGM| https://crowncloud.net VPS and Dedicated Servers for the BTC community
jgarzik
Legendary
*
Offline Offline

Activity: 1470


View Profile
June 21, 2012, 01:32:23 AM
 #77

1200 Unconfirmed transactions as I post this.   Undecided

That low?  Smiley  It was over 9,000 unconfirmed just a few days ago...


Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
kano
Legendary
*
Offline Offline

Activity: 1918


Linux since 1997 RedHat 4


View Profile
June 21, 2012, 01:57:58 AM
 #78

Hmm, no orphans for over 24 hours (since 185346) according to blockchain.
Well at least some good news even if it is unrelated to any code change Smiley

... jinx ...

Pool: https://kano.is BTC: 1KanoiBupPiZfkwqB7rfLXAzPnoTshAVmb
CKPool and CGMiner developer, IRC FreeNode #ckpool and #cgminer kanoi
Help keep Bitcoin secure by mining on pools with Stratum, the best protocol to mine Bitcoins with ASIC hardware
Graet
VIP
Legendary
*
Offline Offline

Activity: 980



View Profile WWW
June 21, 2012, 02:01:24 AM
 #79

1200 Unconfirmed transactions as I post this.   Undecided

That low?  Smiley  It was over 9,000 unconfirmed just a few days ago...


https://blockchain.info/
slush just hoovered up over 900 txns

the unconfirmed number is very dynamic, sort of depends on whether pools accepting lots of txns get a run or those limiting txns get a run of blocks
looking every few days doesn't give a good representation Smiley

869 Unconfirmed Transactions now

| Ozcoin Pooled Mining Pty Ltd https://ozcoin.net Double Geometric Reward System https://lc.ozcoin.net for Litecoin mining DGM| https://crowncloud.net VPS and Dedicated Servers for the BTC community
-ck
Moderator
Legendary
*
Offline Offline

Activity: 1988


Ruu \o/


View Profile WWW
June 21, 2012, 10:05:26 AM
 #80

For what it's worth, mtred have moved their bitcoind over to 0.6.3rc1 after I pointed it out to them and I have noticed a significant improvement in their block change speed (with much less lag across longpolls) already, even though a lot of the network has not yet changed over to the potentially faster protocol.

Primary developer/maintainer for cgminer and ckpool/ckproxy.
Pooled mine at kano.is, solo mine at solo.ckpool.org
-ck
Pages: « 1 2 3 [4] 5 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!