Bitcoin Forum
March 28, 2017, 02:03:20 PM *
News: Latest stable version of Bitcoin Core: 0.14.0  [Torrent]. (New!)
 
   Home   Help Search Donate Login Register  
Pages: [1] 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 »  All
  Print  
Author Topic: How (and why) pools (and all miners) should use the Relay Network  (Read 201177 times)
Matt Corallo
Hero Member
*****
Offline Offline

Activity: 754


View Profile
September 02, 2014, 08:52:00 AM
 #1

UPDATE: The relay network has been updated to use Bitcoin FIBRE - you can find th enew instructions to peer with it at http://bitcoinfibre.org/public-network.html.

Many of you have likely seen the recent discussions on the P2Pool thread and the post on the foundation blog about the relay network.

It exists as a way for pool operators (and all miners, though not hashers) to get their blocks relayed quickly across a separate network both as a backup to the P2P network and as a quicker way to get the latest blocks as it skips relaying transactions which have already been seen. Thus, if you're a miner anywhere from a small p2pool miner to a large pool, you should be running one of the relay network clients.

You can see more information about its original goals and its original announcement here

You can find the latest information at http://bitcoinrelaynetwork.org/

Bitcoin Ubuntu PPA maintainer - donate to me personally: 1JBMattRztKDF2KRS3vhjJXA7h47NEsn2c
http://bitcoinrelaynetwork.org maintainer
PGP ID: 07DF 3E57 A548 CCFB 7530  7091 89BB B866 3E2E65CE
1490709800
Hero Member
*
Offline Offline

Posts: 1490709800

View Profile Personal Message (Offline)

Ignore
1490709800
Reply with quote  #2

1490709800
Report to moderator
1490709800
Hero Member
*
Offline Offline

Posts: 1490709800

View Profile Personal Message (Offline)

Ignore
1490709800
Reply with quote  #2

1490709800
Report to moderator
1490709800
Hero Member
*
Offline Offline

Posts: 1490709800

View Profile Personal Message (Offline)

Ignore
1490709800
Reply with quote  #2

1490709800
Report to moderator
If you want to be a moderator, report many posts with accuracy. You will be noticed.
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1490709800
Hero Member
*
Offline Offline

Posts: 1490709800

View Profile Personal Message (Offline)

Ignore
1490709800
Reply with quote  #2

1490709800
Report to moderator
1490709800
Hero Member
*
Offline Offline

Posts: 1490709800

View Profile Personal Message (Offline)

Ignore
1490709800
Reply with quote  #2

1490709800
Report to moderator
Matt Corallo
Hero Member
*****
Offline Offline

Activity: 754


View Profile
September 02, 2014, 08:52:11 AM
 #2

I used to have a list of pools here, but it had gone unupdated in a long time and was very incomplete...essentially, most the major pools and large miners are connecting to the relay network in some capacity.

Bitcoin Ubuntu PPA maintainer - donate to me personally: 1JBMattRztKDF2KRS3vhjJXA7h47NEsn2c
http://bitcoinrelaynetwork.org maintainer
PGP ID: 07DF 3E57 A548 CCFB 7530  7091 89BB B866 3E2E65CE
gmaxwell
Moderator
Legendary
*
Offline Offline

Activity: 2128



View Profile
September 02, 2014, 08:54:11 AM
 #3

I've been running the relay node client since Matt wrote it and it appears save a considerable amount of bandwidth (e.g. avoiding resending >95% of all transactions) which means faster block relaying and fewer orphans.

Plus, alternative transports make the network more robust e.g. some crazy firewall starts blocking the Bitcoin P2P network, the relay client may get through and prevent partitioning.

Great stuff.
-ck
Moderator
Legendary
*
Offline Offline

Activity: 2100


Ruu \o/


View Profile WWW
September 02, 2014, 09:54:44 AM
 #4

Thanks Matt, good work. The ckpool instances out there (public and the private one yet to become public) already use this and we're grateful.

Primary developer/maintainer for cgminer and ckpool/ckproxy.
Solo mine at solo.ckpool.org
-ck
zvs
Legendary
*
Offline Offline

Activity: 1428



View Profile WWW
September 02, 2014, 09:59:38 AM
 #5

does it request blocks/transactions from multiple sources as well?

re:

   "addr" : "public.eu.relay.mattcorallo.com:8335",
        "bytessent" : 21548126,
        "bytesrecv" : 5476607,

or bitcoind i reset maybe an hr ago

        "addr" : "146.185.173.241:8335",
        "bytessent" : 3910519,
        "bytesrecv" : 281180,



Dacentec, best deals for US dedicated servers. They regularly restock $20-$25 Opterons with 8-16GB RAM & 2x1-2TB HDD's (ofc, usually lots of other good stuff to choose from).  I did a Serverbear benchmark of one of my $20/mo Opteron (June last year), it's here.  Have had about a half dozen different servers with Dacentec, & none have failed to sustain at least 40MB/s (burst higher). My favorite is a 12-month rent-to-own ZT Systems 2XL5520 16GB 2x2TB SATA for $40/month (got lucky with the 'off-brand', haven't seen a RTO 2xL5520 for under $50/mo since -- at least for monthly contracts).  wholesaleinternet.com has some ancient 2-core intel CPUs @ $10/mo sometimes (I got an Intel Core 2 6300 @ 1.86GHz, with a 250GB HDD with 46000 hours on it, LOL. $20 @ Dacentec is much better, if you can grab one). joesdatacenter.com (same location as Wholesale Internet) also occasionally has specials (or if you don't want to wait, it has an AMD Opteron 170 @ $16/mo).
mdude77
Legendary
*
Offline Offline

Activity: 1358


View Profile
September 02, 2014, 11:08:59 AM
 #6

Watching.

M

MMinerMonitor author, monitor/auto/schedule reboots/alerts/remote/MobileMiner for Ants and Spondoolies! Latest (5.2). MPoolMonitor author, monitor stats/workers for most pools, global BTC stats (current/nxt diff/USD val/hashrate/calc)! Latest (v4.2) 
Buyer beware of Bitmain hardware and services.
linuxforyou
Jr. Member
*
Offline Offline

Activity: 37


View Profile WWW
September 02, 2014, 07:42:03 PM
 #7

Thanks Matt for the great work, Like the python version more.

RTFM
DrHaribo
Legendary
*
Offline Offline

Activity: 2044


Bitminter.com Operator


View Profile WWW
September 02, 2014, 08:33:48 PM
 #8

Thanks for this great service, Matt.

I've been using the relay network since it was first announced. The only problem I can report is that I began getting the blocks so quickly that blockchain.info marked a lot of blocks as coming from my pool when they did not.  Cheesy

▶▶▶ Bitminter.com - Your trusted mining pool since 2011.
Matt Corallo
Hero Member
*****
Offline Offline

Activity: 754


View Profile
September 02, 2014, 09:23:48 PM
 #9

does it request blocks/transactions from multiple sources as well?

re:

   "addr" : "public.eu.relay.mattcorallo.com:8335",
        "bytessent" : 21548126,
        "bytesrecv" : 5476607,

or bitcoind i reset maybe an hr ago

        "addr" : "146.185.173.241:8335",
        "bytessent" : 3910519,
        "bytesrecv" : 281180,
Hmm? Note sure what you're asking here. I know several pools are running one of the clients, and Ive manually made outbound peers to a few large pools and a small set of other random nodes, all of which are received blocks from. I dont think I ever fixed the outbound manual peering to send blocks over those connections, but all other connections should both have blocks and transactions pulled from them and receive blocks (and probably transactions).

Bitcoin Ubuntu PPA maintainer - donate to me personally: 1JBMattRztKDF2KRS3vhjJXA7h47NEsn2c
http://bitcoinrelaynetwork.org maintainer
PGP ID: 07DF 3E57 A548 CCFB 7530  7091 89BB B866 3E2E65CE
PatMan
Hero Member
*****
Offline Offline

Activity: 924


Watch out for the "Neg-Rep-Dogie-Police".....


View Profile WWW
September 02, 2014, 09:25:33 PM
 #10

I've been using this neat bit of kit on my p2pool setup for the last week or two, integrates seamlessly with p2pool  - great stuff indeed Matt, many thanks  Smiley

"When one person is deluded it is called insanity - when many people are deluded it is called religion" - Robert M. Pirsig.  I don't want your coins, I want change.
Amazon UK BTC payment service - https://bitcointalk.org/index.php?topic=301229.0 - with FREE delivery!
http://www.ae911truth.org/ - http://rethink911.org/ - http://rememberbuilding7.org/
Matt Corallo
Hero Member
*****
Offline Offline

Activity: 754


View Profile
September 02, 2014, 09:28:46 PM
 #11

Thanks for this great service, Matt.

I've been using the relay network since it was first announced. The only problem I can report is that I began getting the blocks so quickly that blockchain.info marked a lot of blocks as coming from my pool when they did not.  Cheesy
Do you use standard bitcoin addnode or have you upgraded to one of the relay network clients? Are there any features you want form the relay clients?

Bitcoin Ubuntu PPA maintainer - donate to me personally: 1JBMattRztKDF2KRS3vhjJXA7h47NEsn2c
http://bitcoinrelaynetwork.org maintainer
PGP ID: 07DF 3E57 A548 CCFB 7530  7091 89BB B866 3E2E65CE
DrHaribo
Legendary
*
Offline Offline

Activity: 2044


Bitminter.com Operator


View Profile WWW
September 02, 2014, 10:53:21 PM
 #12

Do you use standard bitcoin addnode or have you upgraded to one of the relay network clients? Are there any features you want form the relay clients?

I'm still just using addnode. I was planning to build the client into my mining server code. But since it is taking me a while to get around to it, I should probably just set up the standard client in the meantime.

Here's how I think a last bit of performance can be squeezed out of this:

Identify blocks coming from the biggest pools, by coinbase address and/or signature. If not recognized consider the source unknown and untrusted.

For known sources, keep track of how many times your local bitcoind accepts and rejects blocks from this source. Sources with 10+ accepted blocks and zero rejected are considered trusted. Other sources are untrusted.

When you receive a block from a trusted source, start building on top of it immediately. Still submit the block to local bitcoind for full verification. If bitcoind rejects the block, push new work data to miners to immediately go back to building on the previous parent block.

Possible issue: some getwork clients will get confused about what is old vs. new work when you send work data it has previously seen. They will get stuck DoSing the pool requesting new work over and over as fast as they can until there is a new block change. I'm not sure if any getblocktemplate clients have issues with going back to "old" work. Maybe they can get stuck mining on the invalid block, thinking it is the latest work? They'll probably be fine, but I need to find out for sure. Stratum clients, using only a single TCP connection, should be fine.

Thoughts?

▶▶▶ Bitminter.com - Your trusted mining pool since 2011.
DrHaribo
Legendary
*
Offline Offline

Activity: 2044


Bitminter.com Operator


View Profile WWW
September 02, 2014, 10:54:32 PM
 #13

Would something similar to this relay network be useful for the p2pool share chain?

▶▶▶ Bitminter.com - Your trusted mining pool since 2011.
Matt Corallo
Hero Member
*****
Offline Offline

Activity: 754


View Profile
September 02, 2014, 11:28:17 PM
 #14

I'm still just using addnode. I was planning to build the client into my mining server code. But since it is taking me a while to get around to it, I should probably just set up the standard client in the meantime.

Here's how I think a last bit of performance can be squeezed out of this:

Identify blocks coming from the biggest pools, by coinbase address and/or signature. If not recognized consider the source unknown and untrusted.

For known sources, keep track of how many times your local bitcoind accepts and rejects blocks from this source. Sources with 10+ accepted blocks and zero rejected are considered trusted. Other sources are untrusted.

When you receive a block from a trusted source, start building on top of it immediately. Still submit the block to local bitcoind for full verification. If bitcoind rejects the block, push new work data to miners to immediately go back to building on the previous parent block.

Possible issue: some getwork clients will get confused about what is old vs. new work when you send work data it has previously seen. They will get stuck DoSing the pool requesting new work over and over as fast as they can until there is a new block change. I'm not sure if any getblocktemplate clients have issues with going back to "old" work. Maybe they can get stuck mining on the invalid block, thinking it is the latest work? They'll probably be fine, but I need to find out for sure. Stratum clients, using only a single TCP connection, should be fine.

Thoughts?

The relay networlk already doesnt do full verification. It does simple SPV validation simply as a DoS precaution, but thats already rather quick. Building blocks on top of another miner without validation is still not possible no matter what precautions you take...Imagine a pool op of a smaller pool who wants to have some fun...they break into a large, "trusted" miner and change it to include some invalid transaction in its blocks, now it finds a block and everyone else goes off and mines on it, giving the smaller miner a chance to mine blocks for free while the rest of the network is off building invalid crap.

Bitcoin Ubuntu PPA maintainer - donate to me personally: 1JBMattRztKDF2KRS3vhjJXA7h47NEsn2c
http://bitcoinrelaynetwork.org maintainer
PGP ID: 07DF 3E57 A548 CCFB 7530  7091 89BB B866 3E2E65CE
Matt Corallo
Hero Member
*****
Offline Offline

Activity: 754


View Profile
September 02, 2014, 11:28:40 PM
 #15

Would something similar to this relay network be useful for the p2pool share chain?
P2Pool already does something similar within its own network for the share chain.

Bitcoin Ubuntu PPA maintainer - donate to me personally: 1JBMattRztKDF2KRS3vhjJXA7h47NEsn2c
http://bitcoinrelaynetwork.org maintainer
PGP ID: 07DF 3E57 A548 CCFB 7530  7091 89BB B866 3E2E65CE
Matt Corallo
Hero Member
*****
Offline Offline

Activity: 754


View Profile
September 02, 2014, 11:29:16 PM
 #16

Thanks Matt, good work. The ckpool instances out there (public and the private one yet to become public) already use this and we're grateful.
Are you using one of the clients or just a simple -addnode?

Bitcoin Ubuntu PPA maintainer - donate to me personally: 1JBMattRztKDF2KRS3vhjJXA7h47NEsn2c
http://bitcoinrelaynetwork.org maintainer
PGP ID: 07DF 3E57 A548 CCFB 7530  7091 89BB B866 3E2E65CE
-ck
Moderator
Legendary
*
Offline Offline

Activity: 2100


Ruu \o/


View Profile WWW
September 02, 2014, 11:40:09 PM
 #17

Thanks Matt, good work. The ckpool instances out there (public and the private one yet to become public) already use this and we're grateful.
Are you using one of the clients or just a simple -addnode?
Using the full java client.

Primary developer/maintainer for cgminer and ckpool/ckproxy.
Solo mine at solo.ckpool.org
-ck
gmaxwell
Moderator
Legendary
*
Offline Offline

Activity: 2128



View Profile
September 03, 2014, 01:15:36 AM
 #18

Would something similar to this relay network be useful for the p2pool share chain?
P2Pool already does something similar within its own network for the share chain.
Yup, has for some time— it's is part of why p2pool's observed orphan rate is less than 1/10th several of the larger pools.

Identify blocks coming from the biggest pools, by coinbase address and/or signature. If not recognized consider the source unknown and untrusted.
For known sources, keep track of how many times your local bitcoind accepts and rejects blocks from this source. Sources with 10+ accepted blocks and zero rejected are considered trusted. Other sources are untrusted.
Please don't do this. It undermines the security model of Bitcoin, and at most saves you a few _milliseconds_ (and even there it isn't lost work because you could theoretically find a block and win the block race) since virtually all the transactions are already verified, and cached in memory and the node doesn't check them again.

SPV clients are counting the blocks you produce to be valid. Setting up that kind of verification also produces extreme fragility in that if a software error makes a good node produce a bad block you could have a whole series of bad blocks created by mutually trusting hosts, creating a large fork in the network. This kind of approach is also vulnerable to attack since without a PKI infrastructure you cannot determine who the source of a block is— for example it could just be a sybil who is relaying good blocks to you from other parties but later starts feeding you trash. (and, of course, imposing identities on mining has a multitude of problems and risks that go beyond the simply technical)
DrHaribo
Legendary
*
Offline Offline

Activity: 2044


Bitminter.com Operator


View Profile WWW
September 03, 2014, 07:10:12 AM
 #19

The relay networlk already doesnt do full verification. It does simple SPV validation simply as a DoS precaution, but thats already rather quick. Building blocks on top of another miner without validation is still not possible no matter what precautions you take...Imagine a pool op of a smaller pool who wants to have some fun...they break into a large, "trusted" miner and change it to include some invalid transaction in its blocks, now it finds a block and everyone else goes off and mines on it, giving the smaller miner a chance to mine blocks for free while the rest of the network is off building invalid crap.

If a big pool got hacked you might be mining on invalid data from the time you submit their block to your local bitcoind and until you get the response back that it rejects the block. After that you switch to building on the previous parent block and mark this pool as untrusted so it won't be repeated.

About 100 times per day you get a sub-second advantage to avoid creating an orphan race. And very rarely, probably less often than once a year or maybe never, you mine invalid data for a sub-second. For every invalid block you mine, you have avoided 36 500 orphan races, of which you would have lost more than half because the other guy had a head start. Sacrifice one block (or likely zero) to save 20 000.

SPV clients are counting the blocks you produce to be valid. Setting up that kind of verification also produces extreme fragility in that if a software error makes a good node produce a bad block you could have a whole series of bad blocks created by mutually trusting hosts, creating a large fork in the network. This kind of approach is also vulnerable to attack since without a PKI infrastructure you cannot determine who the source of a block is— for example it could just be a sybil who is relaying good blocks to you from other parties but later starts feeding you trash. (and, of course, imposing identities on mining has a multitude of problems and risks that go beyond the simply technical)

In the rare event that an invalid block might be created, it's not going to reach SPV clients.

Verification doesn't need to be 100%. You can check that the difficulty (bits) is correct, that the generation transaction pays to a known address used by a "trusted" pool, and that the block hash is valid at this difficulty. So an attacker could mine an invalid block and cause you to mine invalid data for less than a second. That's a cost of over 25 BTC to them. They could repeat it for every trusted payment address you have, but no more than that.

Say you trust 4 sources. An attacker could spend over 100 BTC to make you mine invalid data for maybe one second in total. A smarter attacker would use blockwithholding instead, making you lose not 1 second of mining but 4 blocks (average) worth of mining effort, which is much much more than any pool does in 1 second.

Please don't do this. It undermines the security model of Bitcoin, and at most saves you a few _milliseconds_ (and even there it isn't lost work because you could theoretically find a block and win the block race) since virtually all the transactions are already verified, and cached in memory and the node doesn't check them again.

I don't see how it undermines the security model of Bitcoin. If you mine a block before the parent is verified by your own bitcoind you could wait until that happens before sending your new block to the relay network. That way an invalid block never leaves your pool. This way others on the relay network will only see the first invalid block from someone else. You will never add another invalid that they will see.

Say you get blocks from trusted sources 100 times per day and it saves you a total of 25 seconds of mining where you would have entered an orphan race and the other guy has a head start. Maybe 15 seconds where you would have lost an orphan race if you had gotten a block. That's an hour and a half of worthless mining saved per year.

You are right there isn't much to save. For most pools 1.5 hours of mining per year isn't a lot of blocks. While I'm pretty sure this would be of help and not do any harm, the benefit probably doesn't outweigh the effort of implementing this, which is why it has just stayed an idea.

▶▶▶ Bitminter.com - Your trusted mining pool since 2011.
Matt Corallo
Hero Member
*****
Offline Offline

Activity: 754


View Profile
September 03, 2014, 08:22:11 AM
 #20

This discussion has been hashed out time and time again so there is no reason to continue it here. Suffice to say, you cannot build on blocks relayed to you by the relay network any more than you can on blocks sent to you from a random peer on the p2p network.

Bitcoin Ubuntu PPA maintainer - donate to me personally: 1JBMattRztKDF2KRS3vhjJXA7h47NEsn2c
http://bitcoinrelaynetwork.org maintainer
PGP ID: 07DF 3E57 A548 CCFB 7530  7091 89BB B866 3E2E65CE
Pages: [1] 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!