p2pool - Decentralized, Absolutely DoS-Proof, Pool Hopping-Proof Pool [archival]

[forrestv]

Not quite complete ... but that problem's been fixed. As of rev 1379, it at least generates.

It's not ready yet, I'll post here when it's near usability.

[1714010232]

1714010232

[1714010232]

1714010232

[1714010232]

1714010232

[1714010232]

1714010232

[1714010232]

1714010232

[forrestv]

Nearing completeness ... as of rev 1383, generation and sharing of shares between nodes works.

TODO:

• expiring of old shares - currently they build up in the client's memory
• various limits - maximum block size, nonce size, merkle branch length
• more review of timestamping, chain selection

Can I get a list of hostnames (or IP addresses if they don't have a hostname) of people interested in supporting this by running a node? These addresses would be included in the source to be used as bootstrap nodes. Right now there is a hardcoded list of nodes and an option to provide additional nodes on the command line. I believe that this is sufficient, though peer exchange through IRC could be implemented if need be.

[Shevek]

Nearing completeness ... as of rev 1383, generation and sharing of shares between nodes works.

TODO:

• expiring of old shares - currently they build up in the client's memory
• various limits - maximum block size, nonce size, merkle branch length
• more review of timestamping, chain selection

Nice!

But some questions: how finally rewarding and share-tracking will be? Will be a score of best N shares? Will the reward go only to shares in the solved block?

I think, N=600 is too low. A single share is worth 0,083 BTC, which is a pretty high reward for a single match; so, large variability is expected for those whose processing power is below 100 MH/s. Ideally, N=5000 makes 1 share = 0,01 BTC which is low enough to attend smooth incomes.

Also, if a list of best shares will be kept by nodes, it doesn't matter wheather the shares belong to the round the block was solved or not.

Can I get a list of hostnames (or IP addresses if they don't have a hostname) of people interested in supporting this by running a node?

I can run a node. I'll send the IP in a PM.

[forrestv]

Things have been changed up quite a bit - there is no longer a fixed difficulty divider:

Shares are now generated with variable difficulty to keep the rate of shares at about one every 5 seconds. Rewards are split over the last 3*n shares. n = the number of shares that have an equivalent work to the generation of one block.

With this, share generation should be much more often. Shares are now about 200 bytes, making the per-share overhead much smaller.

Since rewards are split over the last 3 block-equivalences of shares, every share should, on average, get a payout three times. This will reduce variance that comes from the pool's block generation.

(All the magic numbers here are subject to change!)

[nanotube]

Things have been changed up quite a bit - there is no longer a fixed difficulty divider:

Shares are now generated with variable difficulty to keep the rate of shares at about one every 5 seconds. Rewards are split over the last 3*n shares. n = the number of shares that have an equivalent work to the generation of one block.

With this, share generation should be much more often. Shares are now about 200 bytes, making the per-share overhead much smaller.

Since rewards are split over the last 3 block-equivalences of shares, every share should, on average, get a payout three times. This will reduce variance that comes from the pool's block generation.

(All the magic numbers here are subject to change!)

Is there a nice (however brief) text description of your latest framework somewhere?

[forrestv]

Is there a nice (however brief) text description of your latest framework somewhere?

Currently the only description is the initial release post combined with posts I've made about the changes. I realize that that's not really enough to characterize the system - many technical details were omitted. I'll write up a detailed description soon so that it can be reviewed. In the mean time, I'm available on IRC on Freenode for any questions/discussion (freenode, nick: forrestv, channel: #bitcoin-dev or #p2pool).


Current status: Working on making algorithms more efficient (share validation mainly)

[nanotube]

Is there a nice (however brief) text description of your latest framework somewhere?

Currently the only description is the initial release post combined with posts I've made about the changes. I realize that that's not really enough to characterize the system - many technical details were omitted. I'll write up a detailed description soon so that it can be reviewed. In the mean time, I'm available on IRC on Freenode for any questions/discussion (freenode, nick: forrestv, channel: #bitcoin-dev or #p2pool).

looking forward to it.

[CydeWeys]

This will be awesome.  Hopefully we can kill Deepbit.  The #1 thing about using something like this?  Protection against DDOS.  Oh, and no fees.  Yup ...

[forrestv]

I'm ready for people to help test on the testnet. To use p2pool:

EDIT: Now using git - https://github.com/forrestv/p2pool

• Get a bitcoind running on the testnet: bitcoind -testnet
• Download p2pool from git or get Windows version below: git clone git://github.com/forrestv/p2pool.git
• Run p2pool: python run_p2pool.py --testnet
• Run a miner daemon with long polling connecting to 127.0.0.1 on port 9332 with any username and password

p2pool will start and take a lot of CPU for a little while downloading and validating all the shares from the P2P network and all the block headers from the local bitcoind (this usually takes about a minute). Once it's ready, you'll see output like this:

Code:

GOT SHARE! 6f20cce78cddd90b111a5be23f4b37e3c9a4bf27d4ff5966150d8054d40
Generating!
Generating!
Accepted share, new highest, will pass to peers! Hash: 6f20cce78cddd90b111a5be23f4b37e3c9a4bf27d4ff5966150d8054d40
Generating!
Pool rate: 454 mhash/s 228 shares Contribution: 99.84% >453 mhash/s

The testnet dynamically adjusts its difficulty to keep share generation at 1 share per second (while on the mainnet share generation is one every five seconds). With 9354 shares it was running without much load on the computer. On the mainnet it'll have at most ~17 thousand shares - one every 5 seconds from the past day.

Start mining, even if it's just with a CPU miner! My GPU miner should make the pool generate a block within the next two hours, and that'll show how payouts work.

Last, join #p2pool on freenode for discussion/announcements of how the pool's doing!

[Meni Rosenfeld]

Shares are now generated with variable difficulty to keep the rate of shares at about one every 5 seconds. Rewards are split over the last 3*n shares. n = the number of shares that have an equivalent work to the generation of one block.

Can you explain more about this point?

Do I understand correctly that if a share (or several shares in a row) takes a long time to be found, the difficulty for the next share will decrease? If so, will the payout per share be proportional to the difficulty? How will this synergize with the 3*n rule? Also, how will you handle updates of the global difficulty mid-round? If not done correctly these matters can make the pool prone to hopping.

Also, won't communication latency be a major problem with a share per 5 seconds?

[forrestv]

Do I understand correctly that if a share (or several shares in a row) takes a long time to be found, the difficulty for the next share will decrease? If so, will the payout per share be proportional to the difficulty? How will this synergize with the 3*n rule?

Yes, difficulty for the next share is computed from the time elapsed between the last share and the 200th ago share. Payouts are proportional to difficulty. Difficulty of individual shares is factored in when the chain is traversed backwards to find which shares to payout to - shares are included until the total difficulty of the shares is equal to 3*(current difficulty), which the last share being partially included to get it exact.

Also, how will you handle updates of the global difficulty mid-round? If not done correctly these matters can make the pool prone to hopping.

The '3*n rule' was chosen to minimize the effect of global difficulty changes. Using it makes the expected value of shares approximately follow the expected value of mining independently, with a period where there is higher-than-solo return followed by lower-than-solo return.


larger: http://u.forre.st/u/snuqtcri/return.png

Blue: Expected return on solo mining
Green: Expected return if p2pool were to use a sliding window of a fixed number of shares
Purple: Expected return using p2pool's '3*n rule'

The higher-than-solo region will encourage miners to keep mining afterwards in order to preserve their value. If everyone stopped mining after the return went under solo, the region of shares that pay higher than solo mining would shift backwards.

I put a lot of thought into this issue, and this was the best solution I could find. The increase/decrease in expected yield is proportional to the change in difficulty and is much lower than what can be obtained from pool hopping on a proportional pool, so I believe that effects will be minimal.

Also, won't communication latency be a major problem with a share per 5 seconds?

Five seconds is a bit higher than I originally thought would work. Share validation and passing is fast with the p2pool client - with a well connected network of p2p nodes the effect of latency should be minimal. If latency is assumed to plague all users evenly, there is no net loss because orphan shares don't actually 'lose' anything. If an orphan share qualifies as a block, it will still be accepted by the bitcoin network and paid to the shares' owners. If it isn't, its 'value' is evenly distributed to the rest of people using p2pool.

[forrestv]

First block generated! (on the testnet, sadly) http://blockexplorer.com/testnet/block/000000000060b9542ac78c6c44f273c1a6df2ca4806b5e19d234576b1aee7a64

[Shevek]

I have some problems...

First, is not possible to run testnet and real net at the same time. So, I must sacrifice the "real" bitcoind to test p2pool

When I do it, I must say p2pool not only --testnet, but rpcport also.

Once the starting occurs without problem, it can't get work from bitcoind 

(using bitcoind 0.3.24 for Linux).

[forrestv]

You can start a bitcoind in parallel with the mainnet one with:

Code:

bitcoind -testnet -rpcport=18332

Then you can do:

Code:

python run_p2pool.py USER PASS --testnet --bitcoind-rpc-port=18332

If this doesn't work, pastebin the error .. because it works for me.

EDIT: Perhaps you used --bitcoind-p2p-port instead of --bitcoind-rpc-port? That would explain your problem.

[Shevek]

You can start a bitcoind in parallel with the mainnet one with:

Code:

bitcoind -testnet -rpcport=18332

Success! It was a problem with the number of dashes 

Then you can do:

Code:

python run_p2pool.py USER PASS --testnet --bitcoind-rpc-port=18332

If this doesn't work, pastebin the error .. because it works for me.

EDIT: Perhaps you used --bitcoind-p2p-port instead of --bitcoind-p2p-port? That would explain your problem.

rpc-port is ok. After handsake OK ("Testing bitcoind RPC connection to 'http://127.0.0.1:18832/' with authorization....", then this error is shown repeatedly:

Code:

Error getting work from bitcoind:
Unhandled Error
Traceback (most recent call last):
Failure: twisted.web.error.Error: 500 Internal Server Error

[forrestv]

Bitcoind gives that error if it's downloading blocks or if it doesn't have any connections. Try

Code:

bitcoind -rpcport=18332 getwork

to see the error. I suppose I should print the resulting message instead of just showing the error code ... fixing that now.

EDIT: Rev 1412 does that:

p2pool.util.jsonrpc.Error: -9 Bitcoin is not connected!

[Shevek]

Bitcoind gives that error if it's downloading blocks or if it doesn't have any connections. Try

Code:

bitcoind -rpcport=18332 getwork

to see the error.

error: {"code":-9,"message":"Bitcoin is not connected!"}

So, I suppose your diagnostic is ok.

[Shevek]

About 5 second rate...

If a block is solved in an average of 10 minutes, at the rate of 5 second per share, it yields 120 shares per block. Counting 3*n -> 360 shares to be rewarded on average.

Too low for those who are pooling to avoid uncertainty 

[Meni Rosenfeld]

About 5 second rate...

If a block is solved in an average of 10 minutes, at the rate of 5 second per share, it yields 120 shares per block. Counting 3*n -> 360 shares to be rewarded on average.

Too low for those who are pooling to avoid uncertainty  

No. The pool finds 120 pool-shares in the time it takes the entire network to find 1 block. But if p2pool is 1% of the network, then the total difficulty of these shares is just 1% of the global difficulty, so 3*difficulty means 36000 shares.

With the constant-ratio method, a p2pool too large means share intervals are too short. With the constant-interval method, a p2pool too large means less variance reduction. It's a tradeoff but it should balance out - if p2pool is large, it will cater mostly to larger miners/pools for which a variance reduction north of X360 is sufficient.

[forrestv]

Now we're ready to begin testing on the main Bitcoin network. To use p2pool:

• Run bitcoin with the RPC interface enabled: put rpcuser=USER, rpcpassword=PASS, and server=1 in bitcoin.conf
• Download p2pool:
  
  • Windows py2exe: See first post
  • git: git clone git://github.com/forrestv/p2pool.git
• Run p2pool:
  
  • Windows py2exe: run_p2pool.exe USER PASS
  • git: python run_p2pool.py USER PASS
• Run a miner daemon with long polling connecting to 127.0.0.1 on port 9332 with any username and password