[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[jtoomim]

Fleshing it out a bit more, you could define the share height H as being the length of the shortest path from the root to a particular share. The PPLNS window could be sized to include all shares with a height greater than (H - X), where X is some arbitrary number roughly corresponding to the N in the LNS system. Shares would be allowed to name any valid share as a direct parent as long as that share was not already an ancestor through any of the other parents, possibly with some arbitrary maximum number of parents hardcoded to reduce the size of the p3pool share data that needs to be incorporated into the blockchain. Or something like that.

My brother passed me this paper as being relevant, but I haven't read it yet. https://eprint.iacr.org/2013/881.pdf

[1714042131]

1714042131

[1714042131]

1714042131

[1714042131]

1714042131

[1714042131]

1714042131

[1714042131]

1714042131

[kano]

Cross-post from the Spondoolies thread, more relevant here:

centralization incoming

If only everyone used p2pool.......

If everyone did, the share difficulty would be through the roof.  2,222,770,798 if I understand it right..

p2pool is not the answer.

a rewrite perhaps.  but not in its current incarnation.

M

I was thinking about this the other day, actually, and I think I got a good solution to the problem.

p2pool uses a share chain much like the block chain for its reward allocation. This means that if you're mining on a share that isn't the most recent, your work is wasted. The blockchain was designed to solve the timestamping problem for creating a distributed ledger system for transactions, which is why the blockchain has to be serialized. With p2pool, there is no such requirement for timestamping and serializing the shares, so the shares do not necessarily need to be arranged in a chain. The fact that they are was probably mostly just a programming convenience in being able to reuse the bitcoin blockchain data structures. I think a better structure for p3pool shares would be a share tree, where each share references at least one parent share instead of exactly one parent share. That way, work done on shares that would end up marked as stale on the current p2pool (i.e. branching shares) could still be incorporated into the p3pool share tree. Add in a small reward to the miner who found the share for each parent share that's referenced for the first time, and everybody's shares should get incorporated as long as the share is based on the most recent block.

Since miners no longer would need to abandon work immediately when someone on p3pool finds a share, p3pool could target much higher share frequencies, such as 1 per second or faster. This would decrease share variance and allow for small miners to use p2pool effectively, and it would also allow p3pool to grow larger while keeping individual share targets reasonable.

Everyone on p2pool has to have the same shares and agree the same shares are valid.
Thus the share chain exists and is based on the BTC block chain idea.

If you remove the share chain, you have to keep a pool database of shares (yeah something I know quite a bit about )
You suddenly need a full fledged database that's able to store all the shares required and also ... the biggest issue of all ... ensure that database is the same on EVERY p2pool node ... so how do you do that? Share chain ...

[mdude77]

I was thinking about this the other day, actually, and I think I got a good solution to the problem.

p2pool uses a share chain much like the block chain for its reward allocation. This means that if you're mining on a share that isn't the most recent, your work is wasted. The blockchain was designed to solve the timestamping problem for creating a distributed ledger system for transactions, which is why the blockchain has to be serialized. With p2pool, there is no such requirement for timestamping and serializing the shares, so the shares do not necessarily need to be arranged in a chain. The fact that they are was probably mostly just a programming convenience in being able to reuse the bitcoin blockchain data structures. I think a better structure for p3pool shares would be a share tree, where each share references at least one parent share instead of exactly one parent share. That way, work done on shares that would end up marked as stale on the current p2pool (i.e. branching shares) could still be incorporated into the p3pool share tree. Add in a small reward to the miner who found the share for each parent share that's referenced for the first time, and everybody's shares should get incorporated as long as the share is based on the most recent block.

Since miners no longer would need to abandon work immediately when someone on p3pool finds a share, p3pool could target much higher share frequencies, such as 1 per second or faster. This would decrease share variance and allow for small miners to use p2pool effectively, and it would also allow p3pool to grow larger while keeping individual share targets reasonable.

That's similar to my idea, except mine was all current rounds of shares are based on a prior psuedo block, instead of the prior p2pool share.  One huge advantage of the current layout is each node verifies all the prior shares.  That's why each share has to contain the payout info for the p2pool miners, so when the "lucky one" that is a block is found, everyone is paid accordingly.

My idea would mean payouts are based on the last pseudo block, not last shares found.  The work submitted is based upon a "pseudo" block based upon all the shares submitted between the last two BTC blocks.  That "block" would be built in a defined order, ie:

root: last "true" BTC block
share #1: first share (based on timestamp) submitted
share #2: second share (based on timestamp) submitted

and so forth.  If a collision on timestamp is found, them some other logic has to be used to determine the tie breaker.

That way each node can still verify all the prior shares and work is still decentralized.

It also means work is only restarted when a BTC block is found, roughly every 10 mins, instead of every 30 seconds.

There may be a flaw in my theory.

M

[jtoomim]

Everyone on p2pool has to have the same shares and agree the same shares are valid.
Thus the share chain exists and is based on the BTC block chain idea.

If you remove the share chain, you have to keep a pool database of shares (yeah something I know quite a bit about )
You suddenly need a full fledged database that's able to store all the shares required and also ... the biggest issue of all ... ensure that database is the same on EVERY p2pool node ... so how do you do that? Share chain ...

Yes, there needs to be some data structure (which you call a database) to keep track of all of the shares. The point I'm raising is that the structure for linking the shares together does not have to be a chain. In a chain, each link has one parent and one child. This forces the serialization of share mining and submission, which causes performance problems for parallel and distributed systems. I'm proposing that we instead use a different structure, such as a directed acyclic graph (which I previously incorrectly called a tree).

You don't need to ensure that the database is the same on every node. There are some types of differences (a couple of extra leaves on the DAG, for example, or orphans on the share chain) that do not cause problems. You just need to ensure that if someone sends you a share, you can find all its (recent) ancestors and ensure that the submitted share rewards them all appropriately.

Essentially, the problem falls to how you deal with branching. Let's imagine shares as if they were mathematicians. Pythagoras proved a^2 + b^2 = c^2. Euclid referenced Pythagoras when he wrote Elements on the subject of geometry. Skipping a few years, Newton created calculus while making use of Euclid's geometry. At about the same time, Leibniz created calculus while making use of Euclid's geometry. A century later, Euler used calculus to define the number e. Who does he credit for calculus? If we're using a share chain, then even if Euler knows both Newton's and Leibniz's works, he has to choose one of them to ignore simply because they didn't know each other's works. What I'm proposing is that we allow Euler to credit both Leibniz and Newton at the same time, as siblings in science. Then, when Riemann comes along and cites Euler, we know that Riemann inherited influence from both Leibniz and Newton, and we don't have to stiff Leibniz just because Newton was better connected. Heck, we could even have the Indian genius Ramanujan (early 20th century) reference both Riemann (19th century) and Madhava of the Kerala school (14th century), so long neglected by Europeans.

When verifying a share (i.e. a candidate block), you collect all of the ancestors of that share and ensure that the candidate share includes appropriate payouts to the mining address associated with those ancestors. As long as you have verified all of the immediate parents of that share (of which there might be 10), you will also have all of the more distant ancestors. If you are missing any of those parent shares, you can get them from the node that's forwarding the candidate share to you, and then you can recursively verify those, etc. until you get to the height limit for the PPLNS system.

One (optional?) constraint that makes sense for which shares can be incorporated: all parent shares must have been efforts at solving the same bitcoin block. No need to reward work that is actually obsolete. Something else might be needed here. Need to think about it later.

The main benefit is that verifying shares and switching the share you're working on to one that's based on the most recent published shares is no longer time-sensitive. If you aren't working off the tallest share branch in existence, the shares you published will still probably be incorporated into the share DAG. Thus, even if new shares are being published every second, you don't have to rerun GetBlockTemplate() and flush the work on your miner for many tens of seconds, and possibly hundreds of seconds. The only constraint is that you need to publish your share early enough that it's cited before a new block is found.

[IYFTech]

Well, today is a sad day.

For the first time in over 2 years I have had to make the difficult decision of distributing my hash rate away from my p2pool node onto other centralised pools. This was not done through greed or profit hunting, but through necessity, as I have bills coming up that can't be paid unless I have some BTC coming in. I've always been happy just to break even, making a profit was just an added bonus, because more important to me was helping to promote decentralised mining & help secure the Bitcoin network, which I still believe in wholeheartedly. However, the way p2pool has performed since the last diff increase has made even this impossible & shows the desperate & well overdue need for a complete overhaul of the p2pool software.

I find it so frustrating that over the last 2+ years there has been zero attempt by the dev to address the problems that have been repeatedly pointed out by p2pool users, this was highlighted by his temper tantrum over donations before deciding to ignore any & all efforts by p2pool users to communicate with him before he abandoned this forum completely. I've lost count of how many times I & other users have said that the moment development of software stops, it is out of date & will die, and by development I mean discussing & addressing known issues & flaws & improving the user experience, not adding another shitcoin or some other useless feature that nobody will use or cares for.

We can't keep telling ourselves "don't worry, they'll come back", because eventually they won't. Not until things change, & I can't see that happening anytime soon. In the meantime, I'll be keeping my nodes up for as long as I can in the faint hope that some knight in shining armour will gallop into the github, fork it,  rewrite p2pool in C whilst simultaneously fixing all the known about issues over the last 2+ years. Wishful thinking.

Yes, today is a sad day indeed  

[reallive1]

Hi
Would documentation be slightly better (https://en.bitcoin.it/wiki/P2Pool_code_documentation, code documentation is one thing, project documentation is another which seems to be absent altogether), it would be easier to grant your wish.

Use of C++(mostly boost) could also be an option, but plain C could be feasible.

[PatMan]

Hi
Would documentation be slightly better (https://en.bitcoin.it/wiki/P2Pool_code_documentation, code documentation is one thing, project documentation is another which seems to be absent altogether), it would be easier to grant your wish.

Use of C++(mostly boost) could also be an option, but plain C could be feasible.

Maybe if you contacted forrestv via github he can provide you with the info you need, no point in PM'ing him here, he hasn't even logged on for several months. Do you have an idea on how to fix the variance/scaling issue?

Yes, today is a sad day indeed 

Sorry to hear that buddy, but I can't say I blame you - I'll be doing the same soon 

[reallive1]

Do you have an idea on how to fix the variance/scaling issue?

At the moment, I'm more interested in understanding how the whole shebang works, then maybe fixing the issue will be something I'll look into. I'll start by digging into the github code (I'm not too familiar with python, anyway), sending a message to the guy will surely helps.

[PatMan]

Do you have an idea on how to fix the variance/scaling issue?

At the moment, I'm more interested in understanding how the whole shebang works, then maybe fixing the issue will be something I'll look into. I'll start by digging into the github code (I'm not too familiar with python, anyway), sending a message to the guy will surely helps.

I'm sure I speak for every p2pool user when I say "THANKS!" Any kind of input is greatly appreciated, & if you need any info, this thread is/was full of some very knowledgable p2pool users. I'm no coder/programmer, but will help in any way I can of course. If it's any help, you might try looking at this thread also:

https://bitcointalk.org/index.php?topic=213051.0

...where I first suggested an upgraded p2pool system - there are some good suggestions & info there, albeit quite old. 

[reallive1]

Well, I see that it does requires some polishing, throwing exception just because bitcoind is still downloading block...

Code:

2015-02-20 12:50:05.782411 p2pool (version 13.4-67-gbcd9a50)
2015-02-20 12:50:05.782574
2015-02-20 12:50:05.782741 Testing bitcoind RPC connection to 'http://127.0.0.1:8332/' with username 'reallive1'...
2015-02-20 12:50:06.576906 > Error getting work from bitcoind:
2015-02-20 12:50:06.577017 > Traceback (most recent call last):
2015-02-20 12:50:06.577082 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
2015-02-20 12:50:06.577147 >     current.result = callback(current.result, *args, **kw)
2015-02-20 12:50:06.577208 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1155, in gotResult
2015-02-20 12:50:06.577267 >     _inlineCallbacks(r, g, deferred)
2015-02-20 12:50:06.577326 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
2015-02-20 12:50:06.577386 >     result = result.throwExceptionIntoGenerator(g)
2015-02-20 12:50:06.577448 >   File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
2015-02-20 12:50:06.577509 >     return g.throw(self.type, self.value, self.tb)
2015-02-20 12:50:06.577569 > --- <exception caught here> ---
2015-02-20 12:50:06.577626 >   File "/home/<userreact>/src/p2pool/p2pool/util/deferral.py", line 41, in f
2015-02-20 12:50:06.577683 >     result = yield func(*args, **kwargs)
2015-02-20 12:50:06.577739 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
2015-02-20 12:50:06.577797 >     result = result.throwExceptionIntoGenerator(g)
2015-02-20 12:50:06.577854 >   File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
2015-02-20 12:50:06.577912 >     return g.throw(self.type, self.value, self.tb)
2015-02-20 12:50:06.577969 >   File "/home/<userreact>/src/p2pool/p2pool/bitcoin/helper.py", line 36, in getwork
2015-02-20 12:50:06.578027 >     work = yield go()
2015-02-20 12:50:06.578082 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
2015-02-20 12:50:06.578140 >     result = result.throwExceptionIntoGenerator(g)
2015-02-20 12:50:06.578195 >   File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
2015-02-20 12:50:06.578253 >     return g.throw(self.type, self.value, self.tb)
2015-02-20 12:50:06.578310 >   File "/home/<userreact>/src/p2pool/p2pool/util/jsonrpc.py", line 133, in _http_do
2015-02-20 12:50:06.578370 >     raise Error_for_code(resp['error']['code'])(resp['error']['message'], resp['error'].get('data', None))
2015-02-20 12:50:06.578430 > p2pool.util.jsonrpc.NarrowError: -10 Bitcoin is downloading blocks...

[windpath]

Well, I see that it does requires some polishing, throwing exception just because bitcoind is still downloading block...

Code:

2015-02-20 12:50:05.782411 p2pool (version 13.4-67-gbcd9a50)
2015-02-20 12:50:05.782574
2015-02-20 12:50:05.782741 Testing bitcoind RPC connection to 'http://127.0.0.1:8332/' with username 'reallive1'...
2015-02-20 12:50:06.576906 > Error getting work from bitcoind:
2015-02-20 12:50:06.577017 > Traceback (most recent call last):
2015-02-20 12:50:06.577082 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
2015-02-20 12:50:06.577147 >     current.result = callback(current.result, *args, **kw)
2015-02-20 12:50:06.577208 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1155, in gotResult
2015-02-20 12:50:06.577267 >     _inlineCallbacks(r, g, deferred)
2015-02-20 12:50:06.577326 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
2015-02-20 12:50:06.577386 >     result = result.throwExceptionIntoGenerator(g)
2015-02-20 12:50:06.577448 >   File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
2015-02-20 12:50:06.577509 >     return g.throw(self.type, self.value, self.tb)
2015-02-20 12:50:06.577569 > --- <exception caught here> ---
2015-02-20 12:50:06.577626 >   File "/home/<userreact>/src/p2pool/p2pool/util/deferral.py", line 41, in f
2015-02-20 12:50:06.577683 >     result = yield func(*args, **kwargs)
2015-02-20 12:50:06.577739 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
2015-02-20 12:50:06.577797 >     result = result.throwExceptionIntoGenerator(g)
2015-02-20 12:50:06.577854 >   File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
2015-02-20 12:50:06.577912 >     return g.throw(self.type, self.value, self.tb)
2015-02-20 12:50:06.577969 >   File "/home/<userreact>/src/p2pool/p2pool/bitcoin/helper.py", line 36, in getwork
2015-02-20 12:50:06.578027 >     work = yield go()
2015-02-20 12:50:06.578082 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
2015-02-20 12:50:06.578140 >     result = result.throwExceptionIntoGenerator(g)
2015-02-20 12:50:06.578195 >   File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
2015-02-20 12:50:06.578253 >     return g.throw(self.type, self.value, self.tb)
2015-02-20 12:50:06.578310 >   File "/home/<userreact>/src/p2pool/p2pool/util/jsonrpc.py", line 133, in _http_do
2015-02-20 12:50:06.578370 >     raise Error_for_code(resp['error']['code'])(resp['error']['message'], resp['error'].get('data', None))
2015-02-20 12:50:06.578430 > p2pool.util.jsonrpc.NarrowError: -10 Bitcoin is downloading blocks...

Not much to polish there other then maybe notifying the user that you can't mine without the latest block....

I admire your enthusiasm for rewriting p2pool in C, and if its bonafide would be happy to run a bounty and help raise you some cash for its completion.

Can you tell us a little about your crypto/coding background?

I've been having some shower thoughts lately about how Multisig or Smart Contracts could be used to solve the variance problem for smaller miners, by building a trustless escrow system to handle small payouts with a payment threshold...

Look forward to hearing more.

[aurel57]

Well, today is a sad day.


Yes, today is a sad day indeed  

It is indeed a sad day to see someone as yourself that was so passionate about p2pool to have to move your miners. I would be very interested in which pool you have decided to mine on and your reasons why? If you don't want to post it here I understand.   

[reallive1]

Not much to polish there other then maybe notifying the user that you can't mine without the latest block....

I admire your enthusiasm for rewriting p2pool in C, and if its bonafide would be happy to run a bounty and help raise you some cash for its completion.

Can you tell us a little about your crypto/coding background?

I've been having some shower thoughts lately about how Multisig or Smart Contracts could be used to solve the variance problem for smaller miners, by building a trustless escrow system to handle small payouts with a payment threshold...

Look forward to hearing more.

As far as rewriting goes:
Stratum being a "human readable protocol"(json based), it's string manipulation in the end, python should be good enough.

My point in posting the exception log is this: if you manage a simple case like: "bitcoind is not ready" with exception throwing, which are at least in C++ and Java have a lot of overhead and thus performance draw back, I wonder what might be next.

As far as my background, I do embedded software development as a living, mostly C/C++.

[Meuh6879]

It is indeed a sad day to see someone as yourself that was so passionate about p2pool to have to move your miners. I would be very interested in which pool you have decided to mine on and your reasons why? If you don't want to post it here I understand.   

i think it's only the 0.10 bitcoin core installation that it reduces the mining powa on P2pool, actually ... 
for me, it's 2 days without the bitcoin core -from 0.9.1 to 0.10-

[IYFTech]

Well, today is a sad day.


Yes, today is a sad day indeed  

It is indeed a sad day to see someone as yourself that was so passionate about p2pool to have to move your miners. I would be very interested in which pool you have decided to mine on and your reasons why? If you don't want to post it here I understand.   

I've spread it out over a few pools, as I hate centralization & hopefully it will help keep variance down. Any pool that pays on time, isn't plagued by shills & is run by a reputable & trustworthy dev is good by me.

So no, not BAN 

[PatMan]

I've spread it out over a few pools, as I hate centralization & hopefully it will help keep variance down. Any pool that pays on time, isn't plagued by shills & is run by a reputable & trustworthy dev is good by me.

So no, not BAN 

 

Nice one bud.

[aurel57]

Well, today is a sad day.


Yes, today is a sad day indeed  

It is indeed a sad day to see someone as yourself that was so passionate about p2pool to have to move your miners. I would be very interested in which pool you have decided to mine on and your reasons why? If you don't want to post it here I understand.  

I've spread it out over a few pools, as I hate centralization & hopefully it will help keep variance down. Any pool that pays on time, isn't plagued by shills & is run by a reputable & trustworthy dev is good by me.

So no, not BAN  

I already knew that    but I have been spreading mine around as well.  I am on Kano's, Slush and have moved some to BAN to check on payments. I have rented miners for p2pool and will move miners from the PPS pool to p2pool as I want to keep shares.

[nreal]

Stratehm/stratum-proxy works just perfect on p2pool, not sure why but even one s4 hosted at umisoo works just fine without any changes to queue.
Best results ever   worth to try, and it supports nicehash too so when the price is right...

https://github.com/Stratehm/stratum-proxy

[-ck]

As far as rewriting goes:
Stratum being a "human readable protocol"(json based), it's string manipulation in the end, python should be good enough.

The stratum protocol bits are just string manipulation but that part alone is only one tiny component of writing pool software. It was good enough in the days when only one client was expected to connect to a p2pool instance, but if you want p2pool to be useful it needs to attract big miners to push the hashrate which means hundreds if not thousands of clients for a local p2pool instance. It does not remotely scale. There are far more fundamental problems that need addressing in the modern world of mining than the mostly cosmetic startup exception issue.

[IYFTech]

Incidentally, I too would have no problem contributing to a bounty fund for a rewrite if anyone suitable/capable can be found. I fear that if nothing is done, p2pools days are numbered.