An enhanced alternative p2p “pool” – a Think Tank.

[PatMan]

Greetings miners & members of the Bitcoin community,

I have been active in Bitcoin circles for over 2 years now and started mining shortly after that. I am a great fan of p2p, I believe in what it stands for as well as the protocol. I am also a great fan & believer in Bitcoin, and believe that Bitcoin & p2p are the perfect partnership for taking Bitcoin to the next level. The advantages to p2p mining are many, but the most appealing to miners are that it is almost completely DDOS proof – meaning no “down” time for anyone, as any miner or pool op will tell you, this is the scourge of Bitcoin mining.

I find it strange that, although there are many “standard” pools to choose from, there only seems to be one p2p version, leaving p2p fans no alternative but to use that “pool”.  I would like to see this change, so am writing this post in an effort to see what kind of feedback/demand/ideas there are for developing a viable alternative to the single p2p “pool”. As we all know, competition breeds innovation – if there is no competition then innovation can be slow or even cease completely, leaving many miners including myself a little frustrated. If Bitcoin is to continue to grow as rapidly as it is doing, then miners & pools must do the same, it’s integral – one needs the other. This is becoming even more apparent with the arrival of the next generation of mining hardware, if the pool software is not compatible with, or constantly updated to deal with these advancements then miners will choose a pool that is. Simple. Just as important, maybe more so, is the ability of pool ops/devs to listen to what their users/miners requests are and deal with them in a fast, efficient and timely manner. This is the nature of p2p, is why it’s so successful & can only improve everything for everybody. After talking to many miners, I feel that there is not only a demand for another p2p “pool”, but there is need for it, and there is much to be gained in having a viable alternative p2p “pool”, especially in regard to efficiency, compatibility & stability.
So, with regards to this, I am inviting anyone who thinks they can help contribute in any way to setting up the new software. I’m no programmer or coder, so would be especially interested to hear from you geeks out there! I have listed below some of what I believe to be of importance, in no particular order (or maybe there is?) and welcome suggestions/ideas and additions will be added accordingly:

1)   Software should be written in C or C++.  I believe this to be the best, most compatible & most widely used language, but is changeable of course.

2)   Ease of use.  With the growing popularity of Bitcoin and new users it is important that the software will work out of the box – with tinkering available to advanced users.

3)   Cross platform compatibility.  Obvious really.

4)   FULL Stratum support for ALL devices.  This is essential for its future success, plug & play. No patches or half baked workarounds.

5)   Maximum efficiency.  No more valuable lost work due to stales, rejects, DOA etc. I see no reason why p2p should be any different to other pools in this respect. 99.8% or more.

6)   Low resource usage. The software should be able to run on most home PC’s comfortably, thus encouraging more beginners.

OK, that’s a start. This list has the potential to get very large I’m sure, and I welcome any feedback, ideas, suggestions from anyone & everyone. I realise that this is no easy task, it is definitely something I am not capable of doing by myself, but if enough people come forward who have the required ability & are willing to help me perform such a task – then maybe we can make a little progress.   Let’s see what happens…..

Peace.  

Update:  Thanks to the miners who have kindly offered to help with testing with their asics/fpga hardware, this is greatly appreciated.

I have had contact with a few programmers, but am still looking for someone with more bitcoin experience - if you think you are that person - please get in touch.

I've also been asked a few times the "what's in it for you?" question - I'm not doing this for financial gain, I'm trying to get this going purely for the benefit of p2p mining & Bitcoin decentralization. I see p2p mining as THE way forward for Bitcoin, and everyone should be able to participate no matter what hardware they are using. Any & all payments/earnings will be done solely by donation through the p2p mining program at the discretion of the miner, and these will be transparently shared between everyone who has helped in it's development, as is the p2p way. Miners will donate if it's successful or shows promise, they won't if it isn't. This puts the onus on the devs to make it successful. I will just be happy to see it working, that's my aim.

As always, questions, ideas, feedback & suggestions are greatly appreciated....

Peace.

[1714950782]

1714950782

[1714950782]

1714950782

[1714950782]

1714950782

[notme]

I suggest you put together a list of weaknesses you see in p2pool.  Other than the fact is is written in python, I don't see why we should completely throw it out.  If there are fundamental problems with the design how do you intend to overcome them?

Overall, I'm not opposed to a separate p2pool effort, however it would need significant advantages over the existing p2pool to be viable.  Currently p2pool is struggling to produce 5 blocks a week., so I'm not sure there is as much demand as you suggest.

[PatMan]

Hey notme,

Good to hear from you. I don’t want to bash p2pool, far from it – I’m a fan & have been using it for nearly 18 months up until a couple of months ago, when I left due to the lack of support for the next generation of mining equipment, loss of work, badly implemented stratum support & general lack of development. Me & other miners there asked for many months for these issues to be addressed but to no avail, so it’s time to move on. Had these issues been addressed when their miners requested then I’m sure it would have been in a much healthier state than it is now. The simple fact is that if a pool can’t support all mining hardware, especially the next generation that is slowly shipping out now (?) then it will eventually fail. I hope it survives, as competition is only good for everyone – a look at other pools confirms this, and there are more pools appearing now than there ever was – so why not p2p?
I have always said that a fully functional p2p pool would blow the other pools away, but it has to work with everything – like every other pool out there does. Spinning as many proxies as you can is all well and good, but the non-compatibility issues will still be there, it just seems a pointless exercise – why not just fix the compatibility issues?  A fresh start is what is required, from the ground up.

What say you? 

[kinlo]

One of the "problems" with bitcoin is block propagation.  Once a block is found, every other node needs to receive the block and mine on top of it.  If you don't mine on top of the longest chain, you will loose, and basicly, your block will become stale (sometimes called orphan)

To calculate the chance of this happening - anyone can jump in and correct my math: we have the following formulla: p/i

p would be the time it takes to propagate a block over the network, and i would be the interval time.  If we look at bitcoin, the value for i would be (on average) 600 seconds, as every 10 minutes a block is found.   Let's say it takes 5 seconds to propagate over the network: we would have 5/600, which means 0.8% of the time you are mining, that you are at risk of not being on the longest chain, and that you will create orphans, and if you find a block the same happens in reverse, someone else might build upon your block and causing your block to have a chance to be orphanned.

p2pool uses the same algorithm as bitcoin, only in a smaller version, their "blockchain" has an interval of 10 seconds.  Since the network is smaller, the propagation times are lower/better, but still, if it would take 1 second, you're down to 10% chance of finding a stale.

So unless my math is wrong, it is clear that shorter rounds will lead to higher stales.

So, anything "new" should find a way to use longer rounds inside the p2p network to make sure the stales are reduced.

[kinlo]

p2pool uses very short rounds.  Every time a round is started, all current work is going to be basically useless, and your miner needs to restart all work. 

I understand especially some asics have big issues with that.   longer rounds should reduce the negative influence greatly, and should actually increase the hashrate of your miners, as you can continue mining for longer periods, without having to forcefully restart your miner.

[notme]

One of the "problems" with bitcoin is block propagation.  Once a block is found, every other node needs to receive the block and mine on top of it.  If you don't mine on top of the longest chain, you will loose, and basicly, your block will become stale (sometimes called orphan)

To calculate the chance of this happening - anyone can jump in and correct my math: we have the following formulla: p/i

p would be the time it takes to propagate a block over the network, and i would be the interval time.  If we look at bitcoin, the value for i would be (on average) 600 seconds, as every 10 minutes a block is found.   Let's say it takes 5 seconds to propagate over the network: we would have 5/600, which means 0.8% of the time you are mining, that you are at risk of not being on the longest chain, and that you will create orphans, and if you find a block the same happens in reverse, someone else might build upon your block and causing your block to have a chance to be orphanned.

p2pool uses the same algorithm as bitcoin, only in a smaller version, their "blockchain" has an interval of 10 seconds.  Since the network is smaller, the propagation times are lower/better, but still, if it would take 1 second, you're down to 10% chance of finding a stale.

So unless my math is wrong, it is clear that shorter rounds will lead to higher stales.

So, anything "new" should find a way to use longer rounds inside the p2p network to make sure the stales are reduced.

p2pool's high "stales" actually don't hurt payout one bit.  If a block is found, but it is "stale" for p2pool, it is still a valid bitcoin block.  It will still be broadcast and p2pool miners will earn the reward.  As long as you don't have more stales than the average p2pool miner you will not lose because of the higher level of stale p2pool shares.

There is however an issue with BFL miners because they can only change work once every 5 seconds.  From what I've read the other ASICs work fine with p2pool and I can confirm no issues with ASICMiner hardware.

[PatMan]

ASICS hardware is not compatible with p2pool as it stands without a patch/workaround - and it's far from optimal, leaving the user with a less efficient & very expensive miner. This is why people don't use them on p2pool, and why I want to incorporate ASICS & every other hardware setup into the program, there should be no need to have to hack it or apply patches to make it "sort of" work - it should just work once the program knows/detects what is connected to it. And, it should work properly, adjusting accordingly. This is what the aim should be, as I mentioned in the first post, otherwise we will be going the same way as the other p2p program and will achieve nothing with the same results. The only way to achieve this is to change the programming and think differently, a new approach is needed, and a programmer who is capable of thinking slightly out of the box will be able to do this I'm sure. If it is to be successful then it must be compatible with everything, no compromises.

This can be done, with the right people & attitude.

As for "p2pool's high "stales" actually don't hurt payout one bit" - of course it does: 100% stales = 0 payout, 0% stales = 100% payout - adjust accordingly. It might not matter so much with p2p - but it still matters. If other pools can have 99.8% then so can p2p. Miners don't want to spend 30,000 on the latest rig only to see 25% of their investment go up in stales straight off the bat- 25% of stales is 25% lost revenue, no matter what way you look at it.

[yxxyun]

why not use go lang, I think it is a good web service program language.

[kinlo]

why not use go lang, I think it is a good web service program language.

+1 for go   I'm using it a lot too, including parts of my pool.

In any case, I don't think there is a big issue with python.  As long as it is properly packaged, people don't really care.  C or C++ would just slow down development imho

[kinlo]

p2pool's high "stales" actually don't hurt payout one bit.  If a block is found, but it is "stale" for p2pool, it is still a valid bitcoin block.  It will still be broadcast and p2pool miners will earn the reward.  As long as you don't have more stales than the average p2pool miner you will not lose because of the higher level of stale p2pool shares.

There is however an issue with BFL miners because they can only change work once every 5 seconds.  From what I've read the other ASICs work fine with p2pool and I can confirm no issues with ASICMiner hardware.

I don't agree.  Yes indeed, if the block is "stale" for p2pool it can still be a valid bitcoin block.  Therefore this particular issue doesn't hurt p2pool in regards to what the pool will earn in bitcoin (altough the 2nd post I did should influence)

But regardless, if you are badly connected to p2pool, you will have many stales, so yes, the bitcoin blocks you find will not be punished by that effect, but the p2pool shares will be.  And if you can find less p2pool shares, then you will automatically earn fewer coin....

[PatMan]

Exactly - more shares = more payout, this is mining, and is why all other pools are more successful. If the new p2p program is to compete & survive then it has to at least match what other pools are offering as standard, but preferably better them. We all know that statistics show that p2p has the highest payout of most pools over time, so if we can add to that the lost/stale/doa/orphaned percentage of shares then p2p will undoubtedly flourish. These are high targets of course, but it's what is necessary to be successful. As for the program language, I think ultimately this would be at the discretion of the programmer that him/herself is most competent & comfortable using, I quoted c or c++ simply because it was the most widely used/accepted code, but as with everything I'm open to ideas & options, although to be honest, so far I have had bad experiences with python - but this could be simply because of the way it was written. I'll edit the original post to reflect the changeability.

I like the direction this is going, more input please........

[notme]

p2pool's high "stales" actually don't hurt payout one bit.  If a block is found, but it is "stale" for p2pool, it is still a valid bitcoin block.  It will still be broadcast and p2pool miners will earn the reward.  As long as you don't have more stales than the average p2pool miner you will not lose because of the higher level of stale p2pool shares.

There is however an issue with BFL miners because they can only change work once every 5 seconds.  From what I've read the other ASICs work fine with p2pool and I can confirm no issues with ASICMiner hardware.

I don't agree.  Yes indeed, if the block is "stale" for p2pool it can still be a valid bitcoin block.  Therefore this particular issue doesn't hurt p2pool in regards to what the pool will earn in bitcoin (altough the 2nd post I did should influence)

But regardless, if you are badly connected to p2pool, you will have many stales, so yes, the bitcoin blocks you find will not be punished by that effect, but the p2pool shares will be.  And if you can find less p2pool shares, then you will automatically earn fewer coin....

If you are badly connected to any pool you will have many stales.  I still don't see how p2pool is any worse in this respect.  I'll concede it has problems with SOME ASICs.

Fundamentally, it comes down to a tradeoff between high stales and high variance.  If you slow down the share rate, it is much harder to find a share and variance will go through the roof.  In order to keep everyone happy from big miners to small, we need a tiered system.  This is actually what I am working towards with my proxy project, but you seem to want to throw out everything already built and start from scratch rather than work incrementally towards a system that suits everyone.  Yes, I want the lower tiers to eventually be decentralized, but I need to work out the kinks in just proxying to an unmodified p2pool for now.  Ultimately, we need to allow multiple addresses to take credit for p2pool shares rather than a single address per share.  Once this is in place, we can have second tier pools that submit p2pool shares that pay out based on their easier share chain.  A centralized proxy is just a first step towards this vision and will help us understand a part of the target system that is achievable today.  Then there is the issue of BFL ASICs.  Once we have a tiered system, we will be able to lengthen the share time on p2pool to something acceptable for all hardware and the second tier pools can provide easier shares and lower variance.

I still don't understand why we should throw the baby out with the bathwater when there is a clear path to achieving the variance and stale share characteristics we desire for every size miner.

[notme]

Contradicting myself, one possibility if we did throw out p2pool would be to have the pool hand out per address difficulty requirements and then ensure the share rate from that address matches what they expect.  Then we allow a constant number of shares per address per time, regardless of how much hash power is producing it.  If course, there will be variance in the share rate, and that means that it we will have to use stochastic models to "ensure the share rate matches what we expect".  This might leave the payout system open to some sort of gaming, but I need to think about it some more.

This would result in every participant seeing the same variance no matter their size, but every new participant raises the variance for everyone.

[PatMan]

Contradicting myself, one possibility if we did throw out p2pool would be to have the pool hand out per address difficulty requirements and then ensure the share rate from that address matches what they expect.  Then we allow a constant number of shares per address per time, regardless of how much hash power is producing it.  If course, there will be variance in the share rate, and that means that it we will have to use stochastic models to "ensure the share rate matches what we expect".  This might leave the payout system open to some sort of gaming, but I need to think about it some more.

This would result in every participant seeing the same variance no matter their size, but every new participant raises the variance for everyone.

Yes, that’s a very interesting & fresh approach that could definitely lead somewhere. I also like the idea of keeping the field playable for everyone – it’s very in line with the p2p ethos. This is exactly what I meant by the software being able to adjust, smart software so to speak, the payout system would be easy enough to work out I’m sure. Good thinking.

I don’t think it’s a case of throwing out p2pools software, more like letting it do its own thing, and learning from its mistakes/inabilities. Like I said, competition is healthy. It simply doesn’t have the scalability or flexibility “built in” to its foundations to allow it to adjust to new conditions/hardware. It was great back when gpu’s were all there was, in fact it was the best, but things have changed since then and software has to be able to change with new technology – future proof so to speak. Over the last few months I’ve seen many experienced coders/developers try and fail to get it to play nicely with various hardware – even a couple of well known & respected miner devs had a try at it with no joy. This to me screams of “build a new, better & updated one”. I know at least one of them has been asked to do this before, but did not have the time or inclination to do so, I hope he’s reading this; maybe he could give us some helpful info……(hint hint)
All options have to be looked into, there is always room for improvement & new ideas.

Notme, what is your preferred choice of code - both generally & suggested for an application like this?

[kinlo]

If you are badly connected to any pool you will have many stales.  I still don't see how p2pool is any worse in this respect.  I'll concede it has problems with SOME ASICs.

I'm afraid this is not true.   Regular pools give you 30 seconds to submit a share.  p2pool uses the first share it finds of a certain difficulty, and you have to build on top of that.  So you need to race to be the first in p2pool in rounds that last 10 seconds.  You have 30 seconds on a regular pool and you don't have to race any peers. 

p2pool is greatly affected by your connectivity, regular pools are not

[PatMan]

If you are badly connected to any pool you will have many stales.  I still don't see how p2pool is any worse in this respect.  I'll concede it has problems with SOME ASICs.

I'm afraid this is not true.   Regular pools give you 30 seconds to submit a share.  p2pool uses the first share it finds of a certain difficulty, and you have to build on top of that.  So you need to race to be the first in p2pool in rounds that last 10 seconds.  You have 30 seconds on a regular pool and you don't have to race any peers. 

p2pool is greatly affected by your connectivity, regular pools are not

I think I agree with you there Kinlo, this is one of p2pools current drawbacks for sure. I'd like to see this p2p application fall in line with all other pools in this regard, if it works for them it should work for us.

[notme]

If you are badly connected to any pool you will have many stales.  I still don't see how p2pool is any worse in this respect.  I'll concede it has problems with SOME ASICs.

I'm afraid this is not true.   Regular pools give you 30 seconds to submit a share.  p2pool uses the first share it finds of a certain difficulty, and you have to build on top of that.  So you need to race to be the first in p2pool in rounds that last 10 seconds.  You have 30 seconds on a regular pool and you don't have to race any peers.  

p2pool is greatly affected by your connectivity, regular pools are not

Those who get the shares first will always have an advantage.  I'm not sure it is possible to remove this advantage unless you somehow punish peers who have low latency.  Increasing the share time to 30 seconds would address the other concern.
But, yes, it is harder to maintain a sufficiently connected p2pool instance.  It is really easy to be sufficiently connected to a centralized pool or a particular p2pool instance.

[PatMan]

Hmmm. Maybe some kind of basic self check feature could be incorporated into the software that could test connections to both the user node and a remote node - giving simple connection results & maybe advising the user of potential connection issues? This would also be of great benefit to the less experienced user/beginner as well as having the benefit of less connectivity questions on the support thread.....User friendliness was one of my original suggested requirements......

EDIT: Also, the more users who have stable/optimal connections the better it is for the network.

[kinlo]

If you are badly connected to any pool you will have many stales.  I still don't see how p2pool is any worse in this respect.  I'll concede it has problems with SOME ASICs.

I'm afraid this is not true.   Regular pools give you 30 seconds to submit a share.  p2pool uses the first share it finds of a certain difficulty, and you have to build on top of that.  So you need to race to be the first in p2pool in rounds that last 10 seconds.  You have 30 seconds on a regular pool and you don't have to race any peers.  

p2pool is greatly affected by your connectivity, regular pools are not

Those who get the shares first will always have an advantage.  I'm not sure it is possible to remove this advantage unless you somehow punish peers who have low latency.  Increasing the share time to 30 seconds would address the other concern.
But, yes, it is harder to maintain a sufficiently connected p2pool instance.  It is really easy to be sufficiently connected to a centralized pool or a particular p2pool instance.

The reason I've been making all my posts are simple: My message to you is think outside the box:  Do not make just another bitcoin clone.  p2pool did it, and it works fine but we see it has some drawbacks.  If you're going to create another implementation of the same basic idea, you're going to end up with the same problems.  I am not telling you I know the answer.  This is a difficult question to solve.  But if you want to start an alternative to p2pool, make sure you use a totally different way of thinking so you will have a product that doesn't work the same way so you have your own advantages (and perhaps disadvantages) over p2pool.

[PatMan]

Completely agree Kinlo, as I said earlier - a fresh start from the ground up is required. I appreciate your input Kinlo, it's always good to hear from someone who thinks the same way & realises what is needed - please do keep the ideas & suggestions coming - the more input from the bitcoin community the better chance there is of developing a successful p2p alternative.