Bitcoin Forum

Centralization at large mining pools is a long-term risk to the safety and security of the Bitcoin network.  p2pool-like mining is vitally important as it does not create systemic risk in the form of centralized pools.  p2pool also benefits the network by substantially boosting the quantity of listening relay nodes across the world.  On November 8th, 2013, the Litecoin Dev Team awarded a ~$2,000 grant (http://ltc.block-explorer.com/tx/436f384b5412498cf3f3e90579954f4428e06fbd36a1c5268b77d639d49f366f) (and $1,000 more (https://bitcointalk.org/index.php?topic=329860.msg4199208#msg4199208) on December 29th) to Forrest Voight in support of development to improve the scalability of the p2pool decentralized mining network.


Our previous support of forrestv enabled the release of ASIC-capable p2pool-13.x for p2pool BTC along with minor improvements to scalability and dust reduction that were of benefit to both the Bitcoin and Litecoin networks.  This new grant is to encourage further research & development that would allow p2pool to comfortably scale to a much larger size.

p2pool Research & Development Ideas Document
https://docs.google.com/document/d/1fbc6yfMJMFAZzVG6zYOwZJvYU0AhM4cvd4bUShL-ScU/edit
Several different proposals are being brainstormed in this document.  Please add comments to the document or reply in this thread if you have ideas.

$2,000 is not enough for this important cause.  Join us in support of Decentralized Mining!
Decentralized mining is vitally important to the long-term security of the Bitcoin network.  $2,000 is too small for this important purpose, so I urge you to please join us in donating to support these important development goals.

forrestv's GPG signed statement: http://im.forre.st/pb/50065825.txt (http://im.forre.st/pb/50065825.txt)

To be clear, forrestv is not a Litecoin Dev.  We just consider his work to be vitally important so we encourage the entire community to support his efforts to protect the Bitcoin network.  If someone else has a feasible implementation even better than p2pool then we may be interested in supporting that too.

Why is Litecoin supporting this cause?
Litecoin Dev takes part in several substantive efforts in support of Bitcoin because we are in the same boat in terms of technology development and the needs to protect the network.  Litecoin gives back to Bitcoin and will increasingly do more in the coming year.

• Litecoin 0.8 contains some patches that differ from Bitcoin 0.8, backported fixes from Bitcoin 0.9, and other things not yet merged in Bitcoin 0.9.  Our testing on a smaller scale with real users and real money on the line has enabled finding bugs and further verifying patches before code shipped in Bitcoin 0.8.x and 0.9.
• Those improvements to 0.8 are also published in Bitcoin 0.8.5 OMG (https://bitcointalk.org/index.php?topic=320695), a well-tested branch of Bitcoin 0.8.5 plus many bug fixes and mature features beyond the standard 0.8.5.  This branch has also helped to find bugs before they reach Bitcoin 0.9.  p2pool miners may like the disablewallet feature.
• Litecoin Dev also supports various Bitcoin devs including forrestv in support of work important to both features and network security.
  

Keep Bitcoin and Litecoin decentralized!

Great work Warren!

thanks warren :)

1. c. this is a must as not everyone is bothered with high variance, so there must be an option to not pay this fee.

2. a. good idea, this will lower the amount of new miners leaving p2pool.

5. we need this since quite some time.

6. is 6 needed if 5 is done properly?

7. this is probably one of the main goals as it will reduce lost (in terms of p2pool blocks, not btc blocks) work alot and would finally alow to create "super-nodes"

#2, this is a great idea in tandem with #3 or #4  (didn't read the full specifics on those)

#7, yeah, not so much anymore (since I put a 1% fee on it), but before I would have up to 10 work requests per cycle, so had to set maxblocksize to 1000...   even w/ a very good computer, at 100,000 size, it would take something around 100-150ms to get all the new work out

#8, i already do this for my own nodes  =p     ...   though i can't monitor the incoming connections constantly.. so some automated disconnect for peers that aren't helpful at all would be nice

ed: oh, on #7 + #8, since the majority of people never bother to alter the defaults, some easy method of choosing the type of node you want to run might be beneficial.  i've always felt 6 outgoing connections is far too few, and a lot of people have incoming firewalled.  

maybe when you initially run p2pool, it generates a config file & asks you how many outgoing connections you want, w/ suggestions based on bandwidth.  similar problem, most people never touch --p2pool-node

Thanks for the donation warren and Litecoin devs!

I like the sub-pool idea.  I've already contributed a patch (mostly taken from generalfault's stratum-mining fork) to log shares to a MySQL database.  It would be great if this could be standardized and also offer clients the option of connecting using getblocktemplate as well as stratum or getwork.  I've also written a custom difficulty patch that I can share.  I like the idea of encouraging miners to use higher difficulty, but I'm not sure how well that would work with a sub-pool.  There would need to be a way to pass that cost on to miners, especially if allowing them to set their own difficulty.

The DB share logging patch I submitted could easily be extended using the same stratum-mining code (the two licenses are compatible) to support PGSQL and SQLite.

You might also want to take a look at how stratum-mining distributes new work to miners as it seems much more lightweight than the way P2Pool does it.

https://blockchain.info/address/1KxvX5Hx8nh36ig2gT5bpeEcqLQcwJsZGB
http://explorer.litecoin.net/address/LPfkfi2tMuGSc64PZTsP9Amt367hwAUQzY

Further donations raised in support of decentralized mining is now approaching $1,000.  Please give to further incentivize these important development goals.

P2Pool all time "luck" is now at 117.6%— It would be interesting if we had a way of telling how much of that is actual luck vs just p2pool having a block relaying advantage due to its highly distributed nature.

Wonder if it makes sense to prioritize the transaction of volunteer miners as an incentive.

It wouldn't be hard for p2pool to at least prioritize it's users spending their generated coin— but P2Pool has never had a large enough share of the global hashrate to make it seem worth doing.

Stopping existing miners from leaving sounds like the most important thing for sure, but after that is stabilised I think basic stuff like a proper website and more professional documentation/help/installers could go a long way.

I'm sure one of the reasons centralised pools are more popular is simply that the profit motive pushes them to a higher level of professionalism, and miners respond to that.

Otherwise known as the "I like the interface" position.


The main issue to me is the competitivity of the p2pool node hardware. I believe that increased usage could be easier to encourage if something can be done to slim down the resource overheads on p2pool, but it's difficult to see how the disk space and disk access performance could be improved (at least with the present sharechain design). The memory and CPU requirements could be improved within the current design, but the obvious solution would sacrifice the ease of platform portability that is currently enjoyed with using the python runtime. And obviously a C or C++ re-write would be a massive job, particularly if it were to embrace a range of platforms, a problem that's already solved.

I guess there is some respite in the form of improvements to bitcoind's memory usage with the -disablewallet configuration option coming soon/available now in a testing branch, but I can't help wondering that time passing and the progress it brings might be most decisive. Did Pieter Wuille's custom ECDSA re-implementation ever get merged? That sort of thing will be more important when the upper limit to the block size inevitably changes, in whatever way is eventually decided upon.

I'm thinking along the lines of being able to easily adapt low-cost computing devices into p2pool nodes. A change to the block size may force the disk space and performance requirements out of the feasibility zone, but every other requirement is unlikely to become so unwieldy. In just a few years, the typical low cost computing device in the Arduino/RasPi mold may be more than capable of all the performance characteristics that good operation of the p2pool/bitcoind setup requires (taking into account the balance of increasing transaction frequency, downward revision in processing requirements per transaction and the upward direction of the processing capabilities of the latest ARM designs). At the present time though, some form of high performance desktop machine just has to be used as a p2pool node, there is no real alternative if you want to make the most of your hashrate.

The reason for the low-cost device angle is obvious: all new mining devices either include or rely on a computing device with networking controller, ethernet or Wifi, and at least enough performance to run unoptimised builds of the mining software (until the developers can get a device to test and code with). To make the stage where unoptimised miner code/drivers with as much comfort margin as possible will become the norm, and so the manufacturers may begin to choose over-specified devices as the more prudent option (it could even begin to help drive down the unit cost of these sorts of low cost computing devices in itself). Once the mining code for a given device is optimised, the now vacated headroom could be leveraged for running p2pool. Can it be done now with the current version of python and it's memory management, on our current generation of mining ASICs? No is the answer. Can native p2pool (and bitcoind) builds be practicably produced for the processor architecture of every possible low-cost computer used as a mining controller? I expect no is the answer to that question too.

But there must be some opportunity to leverage the processing controllers that inevitably form a part of nearly every typical miner that rolls out of the manufacturers doors. Maybe then someone might be (more necessarily) motivated to work on a shiny-tastic web interface  :D

Just sent 20mBTC
With decentralized system we are our own bank and I believe anyone involve has a responsibility of keeping the network safe. Im doing my part with the donation, hope it will help your effort.

A nice feature would be a distributed memory pool.

There could be a field in the links on the share chain for validated transactions.  The entry for each transaction would include;

- the transaction
- the path to the merkle root for that transaction
- the input transactions

Providing all the input transactions is expensive.  With 2 input transactions at 250 bytes, that means you need to provide 750 bytes worth of transactions.  If the merkle tree is 12 levels deep that is an additional 32 * 12 bytes = 384.  That is around 450% larger than just the transaction.

With large multi-input transactions, it would be bigger.  If the transactions per share chain link were limited in size, then naturally those transactions would be discouraged.

Double spending could be protected against by having a system where you can claim a share by showing a transaction included in the share was double spent. 

Nodes can't add the transaction to the share chain's memory pool without proving the transaction exists (and providing the inputs so that all verification information is provided).

The owner of the share which added a transactions might get the tx fees (or a percentage of them).  This would encourage nodes to add transactions.  If an illegal transaction is added, then that share goes to the address which submitted the notification of the error.

If it became popular, then even SPV wallets might store all that info for coins held in the wallet, and transmit it when trying to spend the coin.

Combined with something like the "Ultimate Blockchain Compression", maybe even double spending could be done locally.

There could be a network rule for how to pick transactions from the pool.  This would mean that all nodes mine against the same block.

Transactions that have been added would not be included for at least a few shares.  For example, if there was 10 second shares and transactions in the 12 most recent shares were not used, then there would be 2 minute for illegal transactions to be detected.

Fantastic initiative Warren. Donation on the way...

Just read the Brainstorming document. Some great plans.


Fee on each share is a smart idea, I didn't realise the payout per share is value/difficulty weighted as things are now. I now see the wisdom in larger miners upping their share diff threshold.

Trustless Accumulator (both variants) would be vital, infrequent dusty payouts to those with relatively small hashing power is a real barrier.

Multi-threaded share propagation is potentially good, and could work very well with the per-peer Statistical Tracking (as mentioned). Even though it does little in practical over-time terms to alter the payouts, it would increase user perception of "less wasted work" in the system as a whole. Although I'm not sure whether this won't just have the effect of increasing the effective granularity of stales, as there will still be the same number of shares in the chain.

Per-peer Statistical Tracking in it's own right is great for encouraging the kind of miner who uses professional hosting, and so contributes to a perception of a professionalism. Gives those miners a perceived high worth status, even if it's only in terms of the stratification of connections. And of course orphans could be less prevalent.

Glad to see someone clearly understands the issues involved. =)

SUPPORT DECENTRALIZED MINING TECH

https://bitcointalk.org/index.php?topic=329860.0
If the community donates in excess of $1,000 to Forrest's donation addresses before noon UTC December 31st, the Litecoin Dev Team will contribute an additional $1,000 in support of  research and development of p2pool.

https://blockchain.info/address/1KxvX5Hx8nh36ig2gT5bpeEcqLQcwJsZGB
0.114995 BTC
http://ltc.block-explorer.com/address/LPfkfi2tMuGSc64PZTsP9Amt367hwAUQzY
204.00221964 LTC

The donation addresses must increase in value by > $1,000 above the current received totals.

Why are we doing this?
Litecoin Dev already donated $2,600 to Forrest earlier this year.  We strongly believe that p2pool improvement is critical to the future of Bitcoin so we want to encourage others to join us in supporting this cause.

Anyone concerned about the growth of silent pools (Ghash.io, discus fish etc) should support this initiative.

Adding a trustless accumulator helps with dust payouts.  It doesn't affect the minimum share difficulty.

Even without a bootstrap, it could be tweaked to reward larger miners.  This acts as a direct incentive for miners to push up their share difficulty during the transition.

The lower hashing power miners don't lose out directly, since their debt is held in the block chain.  They do have a lower expected payout though.

Minimum share difficulty requires increasing the share rate.  However, ASICs have difficulty with quickly updating shares.

One way would be to update the share chain in groups of shares.

Share Group Header:

int version
hash prev: points to the previous share group header
hash[32]: points to 32 valid shares
long: timestamp
int: difficulty

Shares would be valid if they paid out based on the previous share group.

Nodes would broadcast their shares.  Once a node has 32 shares that build on the previous share group, then it would construct a new header.

The most likely outcome is that the miner which hits the 32nd share would broadcast a sharegroup containing his share and the 31 others.

A miner who finds a share has an incentive to broadcast it so that it is included in any sharegroup.

If the share rate was 1 second, the share group rate would be 32 seconds.  ASICs would only need to update once per share-group rather than once per share.  

So, you get a 32X drop in the minimum share difficulty, but ASICs still only have to update once every 32 seconds.

This would create more dust, due to the higher share rate.  So, it would need to be combined with an accumulator of some kind.

[Edit]
If shares that pointed to any of the previous 3-4 share groups were considered valid, then the orphan rate would be even lower.  ASICS could run for 2-3 minutes on the same base.
[/edit]

I wish people would donate so we can finally get some proper updates to P2Pool. By not supporting P2Pool mining, people are only helping the operators of the standard pools.

You can set up your own P2Pool node (even in Windows) and mine without a fee. It's idiotic to keep dumping your coins into the pockets of the operators of standard pools.

Of course, this is ignoring the native security of mining P2Pool, where you can mine on any node and there is zero risk of your coins being stolen (since you are mining directly into your wallet).

Im bumping this thread, this chart is looking ugly

http://i40.tinypic.com/x5eed3.jpg

ghash.io is showing 33% with 4 days chart and 36% with 24hr chart.
their cex.io scheme is getting them a lot of hash.

bump

Recent post on reddit describes frustration with p2pool

"I tried to switch to p2pool, but it's simply not working for me. The computer I use to control my ASIC's is an old one that can not run bitcoind and p2pool at the same time. I tried, but it's just too slow. p2pool always warns about not being able to access bitcoind and CPU is constantly at 100% whenever bitcoind receives new transactions.
Next I tried to mine using other p2pool nodes that I chose from here: http://p2pool-nodes.info/ I tried to select nodes that are close, low latency, and have no fee. Unfortunately about 10% of my work seems to get rejected, not sure why. So that did not work either.
Any ideas what I could do? I love the concept of p2pool, but currently the centralized mining pools are just much more usable."

http://www.reddit.com/r/Bitcoin/comments/1usb72/p2pool_i_really_tried_but_its_not_working_for_me/

great goal! this will help BTC and LTC!

bitcoin and litecoin need this now , more then ever.

Push it, push it real good.

How much do you estimate is needed to implement these changes?

Dont worry about rejects on p2pool. 10% of rejects is normal.
Rejects are on centralized pools too, simple they hide it as they are centralised and they work on pps so that doesn't matter for them.

Join us on http://elizium.name

10% is better than the current p2pool average, in fact. I think people just need to fully absorb the way the concept works (including me  ;D).

For the whole mining world, not just those trying out p2pool, too many miners seem to have the impression that you simply "plug it in, then watch the money fly out". You really need to get a grasp of everything that affects your mining as best you can. For instance, there are still people that think buying with fiat and making more than that amount back at a future cryptocurrency exchange rate constitutes profitability. Even more hilariously, some buy coins to pay for the equipment immediately before they make their order. It seems that money printing machines are a little too tempting a business to become involved with, strange isn't it!

is this still active?

Is #5 just the creation of one sub-pool? the description of the idea is very shallow...

IMHO p2pool should behave like gnutella (or any other decent DHT with supernodes), where each user says if he can be a supernode. If he can, he becomes a sub-pool of p2pool. He has no control over who joins him.
When a regular miner connects to p2pool, p2pool chooses a sub-pool to connect to, balancing the miners. With enough sub-pools, no/very small double spend probability.

This would also solve the ease of use problem discussed in the reddit thread http://www.reddit.com/r/Bitcoin/comments/1usb72/p2pool_i_really_tried_but_its_not_working_for_me/

Maybe this has democratization implications? We want democracy, not anarchy though.
I'm sure this has been proposed ages ago since it is so simple, but another push won't hurt. I'd donate decently for it.

What if the pool owner rips off the people who connect to him?

Putting everything onto a chain is what keeps everything honest.  Connecting to a public p2pool node has the same variance as connecting to your own personal one.

For public nodes to work, there has to be reputation of some kind, you can't just have hidden super-nodes.

why do supernodes have to be hidden for what i said to work? they can be public, just let p2pool handle the connection.
Obviously an hacked up miner could connect to a node of his choice, but what incentive would he have to do so in a balanced network?

For a node to handle some of the load for the network, it would have to manage shares and payouts.  This requires trust of some kind.

The p2pool system is to use a sharechain, but to move some of the load off the sharechain means you have to replace that trust.

Is this initiative and the suggestions put forth still active?

The current hashrate of P2Pool for Bitcoin seems to have dropped or remained flat as the difficulty has sky-rocketed in the last few months.

Are we going to see another push here and on Reddit to get more hashrate onto P2Pool nodes?

With regards to an incentive for large miners to increase their share difficulty, like (1 - Fee per share) in the document,

If a miner on p2pool sets a higher difficulty, his block effectively 'weighs' more than a normal share, and so is his share LESS lightly to be turned into a stale share ?


 

No difference AFAIK

Stales happen because a share is based on the identifiable "current share". Everyone works on that. When someone submits a solution to the current share, it propagates across p2pool nodes, and it takes time to do that. This means that more than one person can have a solution to the current share competing to hit the majority of p2pool nodes. If your share looses the race, it's an orphan. There is no question of which share was the higher difficulty, just whose share gets majority acceptance first.

BTCest thread I've seen in months.

Yes. Yes. I will be pledging more than just $/BTC to this cause. I am working towards working on this most important project, too!

P2P Mining Decentralization is the CORNERSTONE of the bitcoin protocol ;D

That can't be right.. When working out which chain is the one with the most work, the longest, and so which one to extend, a large miner's share with higher difficulty must be more important than a normal share.. Otherwise there is wasted work being done and the chain is weaker?

I'm not sure whether a chain of proofs (for this purpose) is made stronger for any practical purpose if the more difficult solution is favoured. It makes little sense to favour more difficult solutions anyway, there would have to be some time limit or majority chain committal cutoff to define when a more difficult solution couldn't unseat a solution that was in the process of being accepted by the network. Then you'd have to reset the cutoff, to make it "fair", and another unseating could take place. This could create occasional chains of multiple usurped shares, according to the random distribution of miners on p2pool trumping a committed share.

Say there are 3 miners in all.

2 have a hash power of 1. 1 has hash power 2. Making a total of 4.

Without changing his difficulty the powerful miner would get 50% of the blocks. And the total hash rate of the network would be 4.

The Big miner then sets his difficulty to x2, effectively making him mine at the same speed as the other 2 smaller miners.

Now he gets 33% of the blocks, the same as the other 2 miners. BUT - if his block is not considered more 'worthy' of extension because of it's higher difficulty, the total network hash rate is now only 3.. We have just lost 25% of the power..

I have looked at the p2pool documentation but have not found reference to the actual way the longest chain is calculated,.. and whether the higher difficulty is used in the calculation.. if someone knows I'm all ears!

IF it is the case that your custom difficulty affects your shares 'weight', then a higher difficulty would mean your share is more likely to be the one that is extended in any chain race, and so less lightly to be a stale.. (Unless the total of all miners average each other out in some way) ?

Stales have no effect on network hash power in terms of generating coins because stale shares that meet the block target are submitted to the coin daemon. If it is accepted, then the block pays out to the network even if it was a stale share for the share chain.


This wouldn't be healthy for the network. All shares should have the same chance of being stale. (Well, some nodes are more efficient than others, but it isn't because of the contents of the share.)

Miners using higher diff targets to help out smaller miners aren't hurt more by stales than anyone else. Yes when they have a stale share it is a larger loss than a smaller share would have been, but the smaller shares they are finding, the more stale shares they will have because it's all a %.

If you have a target of 1000 and a 1% stale rate, each stale share loses you 1000 diff 1 shares worth of work.

If you instead set target to /1 and have a 1% stale rate, then out of every 1000 diff 1 shares worth of work you lose 10 shares to stale and have 990 accepted.

In other words:

1000 per share * 100 shares = 100,000 shares of diff 1 work * 1% = 1000 diff 1 work (1 share) lost as stale.
1 per share * 100,000 shares = 100,000 shares of diff 1 work * 1% = 1000 diff 1 work (1000 shares) lost as stale

All in averages, of course. Total credit lost exactly the same. No matter what target you set, you'll lose stale % amount of it. The target will only change the variance of the stale cost.

In addition to roy7's reply, also remember that higher difficulty shares get a higher reward as a proportion of blocks found.

anyone able to help me with how to make this work after editing the networks.py files?
says it cant import network


.Traceback (most recent call last):
  File "run_p2pool.py", line 3, in <module>
    from p2pool import main
  File "/home/regtapool/p2pool-rav/p2pool/main.py", line 25, in <module>
    from . import networks, web, work
  File "/home/regtapool/p2pool-rav/p2pool/networks.py", line 1, in <module>
    from p2pool.bitcoin import networks
  File "/home/regtapool/p2pool-rav/p2pool/bitcoin/networks.py", line 1, in <module>
    from p2pool.bitcoin import networks
ImportError: cannot import name networks

Another bump for this thread.  So has any of the proposed donations and p2pool suggestions made any progress?

There have been no visible checkins on gitHub for the main p2pool branch for a long while now and barely a peep from forrestv. Others have tweaked here and there on their own forks or personal copies, but nothing has come in by pull-request lately either.

So LTC and BTC devs that have pushed for this and contributed... Has there been any movement?  With the recent uproar about GHash.io and centralized pools, and a push for p2pool again, perhaps it's time for an accounting and see if we're getting anything for our time and donations and suggestions or are we throwing BTC and LTC into development that's just not happening?

Is it possible to distinguish a block mined by a P2Pool from one mined by a centralized pool? are any traces left?

Would it be possible to (at some time) accept only blocks mined by p2pools? or least give some incentive?

Hi, IM new to P2Pool and working out if it is worth using or not, I only have 20ghs at the moment and soon to be 100ghs is it worth using p2pool or should I stick to traditional pools? Also atthe moment I have a time to share of 4.5 days is this normal ?

This isn't the correct place to ask the question. :) 20GHs isn't worth mining in p2pool node but 100GHs would be good. The minimum hash rate recommended is 40GHs. Remember you will have to detect the electricity & maintenance costs when you calculate estimate profit.
Kindly,
      MZ