[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[Polyatomic]

so I was digging around and found some settings that may help with p2pool.

In cgminer and bfgminer there are options for --scan-time and --expiry also queue

I read that --scan-time 1 --expiry 1 and queue set to 0 is best for p2pool.

I managed to add these to my antminer so will monitor how it goes.

to do so you can edit \etc\init.d\cgminer and add in the PARAMS= section

e.g. PARAMS="$AOPTIONS $POOL1 $POOL2 $POOL3 $_pb --api-listen --scan-time 1 --expiry 1"

queue = 0 is already in the \etc\Config\cgminer file from Bitmain.

norgan, check ./cgminer-api config , what is the value for queue there?.

[1714028961]

1714028961

[1714028961]

1714028961

[1714028961]

1714028961

[1714028961]

1714028961

[norgan]

so I was digging around and found some settings that may help with p2pool.

In cgminer and bfgminer there are options for --scan-time and --expiry also queue

I read that --scan-time 1 --expiry 1 and queue set to 0 is best for p2pool.

I managed to add these to my antminer so will monitor how it goes.

to do so you can edit \etc\init.d\cgminer and add in the PARAMS= section

e.g. PARAMS="$AOPTIONS $POOL1 $POOL2 $POOL3 $_pb --api-listen --scan-time 1 --expiry 1"

queue = 0 is already in the \etc\Config\cgminer file from Bitmain.

norgan, check ./cgminer-api config , what is the value for queue there?.

on the s1 there is no cgminer-api file. (I only use cgminer on the ant, for everything else I used bfg+multiminer)

[Polyatomic]

so I was digging around and found some settings that may help with p2pool.

In cgminer and bfgminer there are options for --scan-time and --expiry also queue

I read that --scan-time 1 --expiry 1 and queue set to 0 is best for p2pool.

I managed to add these to my antminer so will monitor how it goes.

to do so you can edit \etc\init.d\cgminer and add in the PARAMS= section

e.g. PARAMS="$AOPTIONS $POOL1 $POOL2 $POOL3 $_pb --api-listen --scan-time 1 --expiry 1"

queue = 0 is already in the \etc\Config\cgminer file from Bitmain.

norgan, check ./cgminer-api config , what is the value for queue there?.

 on the s1 there is no cgminer-api file. (I only use cgminer on the ant, for everything else I used bfg+multiminer)

Its there man,

Code:

root@antMiner:/usr/bin# ls -larths
     0 drwxrwxr-x    1 root     root           0 Dec  2  2013 ..
     9 -rwxr-xr-x    1 root     root        9.1K Dec 25 13:48 lua
     7 -rwxr-xr-x    1 root     root        7.0K Dec 25 14:20 jshn
    12 -rwxr-xr-x    1 root     root       11.8K Dec 25 14:24 luci-bwc
   355 -rwxr-xr-x    1 root     root      354.8K Feb  6 22:57 cgminer.original
     1 -rwxr-xr-x    1 root     root         681 Feb  6 22:57 cgminer-monitor.original
     6 -rwxr-xr-x    1 root     root        6.0K Feb  6 22:57 cgminer-api

you can do it from your pc aswell if you want to, you can point cgminer-api at your S1. Just do ./cgminer-api summary IP_OF_YOUR_S1 bfgminer-rpc will work also.

[norgan]

so I was digging around and found some settings that may help with p2pool.

In cgminer and bfgminer there are options for --scan-time and --expiry also queue

I read that --scan-time 1 --expiry 1 and queue set to 0 is best for p2pool.

I managed to add these to my antminer so will monitor how it goes.

to do so you can edit \etc\init.d\cgminer and add in the PARAMS= section

e.g. PARAMS="$AOPTIONS $POOL1 $POOL2 $POOL3 $_pb --api-listen --scan-time 1 --expiry 1"

queue = 0 is already in the \etc\Config\cgminer file from Bitmain.

norgan, check ./cgminer-api config , what is the value for queue there?.

Ahh I was looking under \etc I'll have a look when I get back home. What should it be?

 on the s1 there is no cgminer-api file. (I only use cgminer on the ant, for everything else I used bfg+multiminer)

Its there man,

Code:

root@antMiner:/usr/bin# ls -larths
     0 drwxrwxr-x    1 root     root           0 Dec  2  2013 ..
     9 -rwxr-xr-x    1 root     root        9.1K Dec 25 13:48 lua
     7 -rwxr-xr-x    1 root     root        7.0K Dec 25 14:20 jshn
    12 -rwxr-xr-x    1 root     root       11.8K Dec 25 14:24 luci-bwc
   355 -rwxr-xr-x    1 root     root      354.8K Feb  6 22:57 cgminer.original
     1 -rwxr-xr-x    1 root     root         681 Feb  6 22:57 cgminer-monitor.original
     6 -rwxr-xr-x    1 root     root        6.0K Feb  6 22:57 cgminer-api

you can do it from your pc aswell if you want to, you can point cgminer-api at your S1. Just do ./cgminer-api summary IP_OF_YOUR_S1 bfgminer-rpc will work also.

I was looking in \etc what am I looking for in cgminer-API? I'll check it out when I get home.

[jedimstr]

Norgen: sorry I'm a little late to the discussion, but you may have missed this:
https://bitcointalk.org/index.php?topic=329860.0
 
Late last year, the Litecoin Core Devs and many cheerleaders for p2pool gathered donations and ideas for advancing/fixing p2pool and gave them to forrestv directly.  Previous to that forrestv was at least somewhat active here and this was supposed to be incentive to continue development for the good of the network. This was as close to a kickstarter as we could get without an actual kickstarter, including a growing Development plan document.

Forrestv got his initial $2000 and subsequent $1000.   He even made a GPG signed statement that he was working on these improvements. So promises were made based on this grant. There were more donations from others.  

Then he more or less disappeared with no passing of the torch or any word about it.  Development on the github repository virtually stopped except for the occasional pull request of someone else's work (usually months after the request came in). Along with what Patman mentioned, this is why people are jaded.   We essentially tried everything you stated already in a fairly formal way and gave $$$ and some direction on where we wanted p2pool to go.  We got the cold shoulder or more accurately, a hit-and-run.  

Good luck with your version of these endeavors, maybe third time's a charm.  But don't begrudge the oldies here... Their complaints are more than valid and it's extremely unfair to call them ingrates.  It just makes you look like an ass instead of an optimist.  I do agree with these efforts, but to do them we need p2pool core developers with deep knowledge and passion for the work.  So far, no one has stepped up to take the reigns.  If a good fork was made, everyone would definitely use it.  It's not like there are a lot of us left anyways.

[norgan]

Ah I see, thanks for that. It's not the oldies so much I get the shits with, I've seen some very Newby posts complaining. For them there is no excuse but I am understanding more now why there is so much pain with the older members here.

Like I said to Patman, it can come across that way and I'm sure some see me as coming across arrogant or worse.
I'm certainly quite fresh in regards to p2pool but that brings fresh enthusiasm so perhaps I can get something going with that fresh enthusiasm, who knows. Can't blame me for trying right...

[Carlton Banks]

Forrestv got his initial $2000 and subsequent $1000.   He even made a GPG signed statement that he was working on these improvements. So promises were made based on this grant. There were more donations from others.  

Then he more or less disappeared with no passing of the torch or any word about it.  Development on the github repository virtually stopped except for the occasional pull request of someone else's work (usually months after the request came in). Along with what Patman mentioned, this is why people are jaded.   We essentially tried everything you stated already in a fairly formal way and gave $$$ and some direction on where we wanted p2pool to go.  We got the cold shoulder or more accurately, a hit-and-run.  

Truthfully, no-one knows anything. Forrest could have been hit by a bus in February. He could also be working on prototype versions of the design alteration he suggested. And he could easily be  busy in real life to devote enough time to p2pool right now. We'll find out something at some point. The devs who hang out in mailing lists and irc chat maybe have some idea, but they post to this thread pretty infrequent.

[nreal]

so I was digging around and found some settings that may help with p2pool.

In cgminer and bfgminer there are options for --scan-time and --expiry also queue

I read that --scan-time 1 --expiry 1 and queue set to 0 is best for p2pool.

I managed to add these to my antminer so will monitor how it goes.

to do so you can edit \etc\init.d\cgminer and add in the PARAMS= section

e.g. PARAMS="$AOPTIONS $POOL1 $POOL2 $POOL3 $_pb --api-listen --scan-time 1 --expiry 1"

queue = 0 is already in the \etc\Config\cgminer file from Bitmain.

norgan, check ./cgminer-api config , what is the value for queue there?.

Ahh I was looking under \etc I'll have a look when I get back home. What should it be?

 on the s1 there is no cgminer-api file. (I only use cgminer on the ant, for everything else I used bfg+multiminer)

Its there man,

Code:

root@antMiner:/usr/bin# ls -larths
     0 drwxrwxr-x    1 root     root           0 Dec  2  2013 ..
     9 -rwxr-xr-x    1 root     root        9.1K Dec 25 13:48 lua
     7 -rwxr-xr-x    1 root     root        7.0K Dec 25 14:20 jshn
    12 -rwxr-xr-x    1 root     root       11.8K Dec 25 14:24 luci-bwc
   355 -rwxr-xr-x    1 root     root      354.8K Feb  6 22:57 cgminer.original
     1 -rwxr-xr-x    1 root     root         681 Feb  6 22:57 cgminer-monitor.original
     6 -rwxr-xr-x    1 root     root        6.0K Feb  6 22:57 cgminer-api

you can do it from your pc aswell if you want to, you can point cgminer-api at your S1. Just do ./cgminer-api summary IP_OF_YOUR_S1 bfgminer-rpc will work also.

I was looking in \etc what am I looking for in cgminer-API? I'll check it out when I get home.

Bfgminers miner.php works just great with ants1 https://forums.butterflylabs.com/blogs/ivan-frimmel/133-setting-up-bfgminer-rpc-access-miner-php-remote-status-update-windows-7-a.html

Just needs webserver, apache and php, its fast and easy. Add  rigs to
$rigs = array('192.168.11.50','192.168.11.51','192.168.11.52','192.168.11.220');

or

# Set $mcast to true to look for your rigs and ignore $rigs
$mcast = false;

to true

Doa dropped with --scan-time 1 --expiry 1, thats good 

[xyzzy099]

Truthfully, no-one knows anything. Forrest could have been hit by a bus in February.

Looking at his forum profile reveals that he was last active June 19, 2014, 09:33:14 PM - so if he was hit by a bus, it is not stopping him from reading the forums.

[jedimstr]

Truthfully, no-one knows anything. Forrest could have been hit by a bus in February.

Looking at his forum profile reveals that he was last active June 19, 2014, 09:33:14 PM - so if he was hit by a bus, it is not stopping him from reading the forums.

And to silence the critiques, he only has to respond with one line...

Either of these are more than acceptable (as an explanation):

1. I'm too busy with my day job/life to devote to this at the moment.
2. I'm working on it still on a local working copy and haven't submitted to the repository until this "one thing" gets resolved.
4. Sorry, I haven't had a lot of time, but I'm back at it now...
3. I'm abandoning this project and it doesn't "do it" for me anymore... so please fork it and run with it.

Since he's not too busy to login and read the forums (as recently as yesterday)...the least he could do is say something/anything.

Not quite $3000 donation worth, but it would be something...
Instead we get _____________________________.

[Carlton Banks]

Truthfully, no-one knows anything. Forrest could have been hit by a bus in February.

Looking at his forum profile reveals that he was last active June 19, 2014, 09:33:14 PM - so if he was hit by a bus, it is not stopping him from reading the forums.

Must have been only a small bus.

And to silence the critiques, he only has to respond with one line...

Either of these are more than acceptable (as an explanation):

1. I'm too busy with my day job/life to devote to this at the moment.
2. I'm working on it still on a local working copy and haven't submitted to the repository until this "one thing" gets resolved.
4. Sorry, I haven't had a lot of time, but I'm back at it now...
3. I'm abandoning this project and it doesn't "do it" for me anymore... so please fork it and run with it.

Since he's not too busy to login and read the forums (as recently as yesterday)...the least he could do is say something/anything.

Not quite $3000 donation worth, but it would be something...
Instead we get _____________________________.

He could be logging in for all number of reasons that aren't checking this thread. People in this thread are becoming obsessed with this guy, there are plenty of other devs with useful input on p2pool issues. Seems like they've not got much to say right now either, and so maybe that's just how it is, thread reflects reality.

[windpath]

This is a great conversation guys, and one that needs to be had.

I have built and managed large development teams in the past for specialized projects, and I have to say that none of them came close to the complexity and challenges offered by what we need in a p2pool dev team.

For p2pool to get to the next level I think we need an active lead dev or chief scientist to lead; and 3 technical achievements, the first 2 are easy:

1. Hardware compatibility and a responsive available individual for both HW and mining SW folks to work with.

2. A great looking front end that provides what miners want that is unified across nodes. p2pool already has the ability to be the most transparent pool in existence, lets get that data out there for miners to mull over and do it in a way that looks so good out of the box that node operators are not inclined to re-invent the front end with each install. When a miner checks out a node it should be familiar and they should know what they are looking at. I'm happy to build this using open source tools and have a solid start already on my node. Combining Bootstrap, PHP, and an SQL DB we could build an open source, cross-platform, front end that both looks great and is data rich.

Which leads me to our 3rd, and biggest challenge:

3. Vertical scalability that reduces variance for all miners.

This is where the real challenges lie.

For p2pool to support large mining operations, and still be able to attract medium, small, and micro miners we need completely new share difficulty and payout structures.

The first step here has nothing to do with code, some completely new concepts need to be developed and generally accepted by p2pool miners. Once we have those concepts down, and can demonstrate them to be technically possible, we can seek out devs with the chops to pull it off.

This may seem like no big deal, but I assure you it is. Here is my first crack at it:

Payouts
While miners meeting a certain threshold could still be paid directly from the generation TX, smaller miners under the payout threshold when a block is found, lets say BTC0.01 to start, would be able to see their p2pool earnings down to the Satoshi in real time, but would not receive a payout until reaching the threshold.

To accomplish this I propose a decentralized, trust-less escrow system.

A p2pool software controlled secure wallet where all payouts from each block that did not meet the payout threshold would be stored, and then paid out to miners when they reach the threshold.

Miners who reach the minimum payout during a round are paid directly from the generation tx, while payments for all miners under the threshold are sent to and stored in the wallet, and payouts are made from the wallet when the threshold is reached.

The caveat is that this needs to be handled entirely by p2pool in a decentralized and trust less way with no single "admin" overseeing them. How to handle the keys securely without anyone but the p2pool code knowing them is a challenge, maybe the guys from the dark wallet or Armory teams would have some technically viable suggestions.

Difficulty
An automated "variable and weighted" share difficulty would need to be assigned to miners based on hash rate. Every major pool in existence does it, we just need a method of doing it across the p2pool block chain in a way that is not easily manipulated. Perhaps the MIT Bitcoin club would be interested in tackling such a challenge?

End game: Protect the network, encourage miners to participate actively in both Bitcoin and p2pools decentralized nature.

Even without active development p2pool is a strong "brand" in the Bitcoin space, most miners are aware of it, and its challenges. Overcoming its biggest challenges along with a decent marketing push (by us collectively) spreading the word could easily push p2pool to the top and keep Bitcoin strong and trust less for the future.

Just my 0.02 bits.

[nreal]

So we should donate to an expert who makes the frontend that works and has maybe the same info as p2poolinfo once had..

[windpath]

So we should donate to an expert who makes the frontend that works and has maybe the same info as p2poolinfo once had..

I'll be done with recreating p2pool.info in a few weeks, no donations requested.

Link: http://minefast.coincadence.com/p2pool-stats.php

The front end will be ongoing project for a while until I think it's right...

Link Node: http://minefast.coincadence.com/index.php
Link Miner: http://minefast.coincadence.com/miner.php?id=19vXrwKGUhK4cCU8tA4kWZgbChcmh9a6qj

[jonnybravo0311]

Code:

This is a great conversation guys, and one that needs to be had.

I have built and managed large development teams in the past for specialized projects, and I have to say that none of them came close to the complexity and challenges offered by what we need in a p2pool dev team.

For p2pool to get to the next level I think we need an active lead dev or chief scientist to lead; and 3 technical achievements, the first 2 are easy:

1. Hardware compatibility and a responsive available individual for both HW and mining SW folks to work with. 

2. A great looking front end that provides what miners want that is unified across nodes. p2pool already has the ability to be the most transparent pool in existence, lets get that data out there for miners to mull over and do it in a way that looks so good out of the box that node operators are not inclined to re-invent the front end with each install. When a miner checks out a node it should be familiar and they should know what they are looking at. I'm happy to build this using open source tools and have a solid start already on my node. Combining Bootstrap, PHP, and an SQL DB we could build an open source, cross-platform, front end that both looks great and is data rich.

Which leads me to our 3rd, and biggest challenge:

[b]3. Vertical scalability that reduces variance for all miners.[/b] 

This is where the real challenges lie. 

For p2pool to support large mining operations, and still be able to attract medium, small, and micro miners we need completely new share difficulty and payout structures.

The first step here has nothing to do with code, some completely new concepts need to be developed and generally accepted by p2pool miners. Once we have those concepts down, and can demonstrate them to be technically possible, we can seek out devs with the chops to pull it off.

This may seem like no big deal, but I assure you it is. Here is my first crack at it:

[b]Payouts[/b]
While miners meeting a certain threshold could still be paid directly from the generation TX, smaller miners under the payout threshold when a block is found, lets say [btc]0.01 to start, would be able to see their p2pool earnings down to the Satoshi in real time, but would not receive a payout until reaching the threshold.

To accomplish this I propose a decentralized, trust-less escrow system. 

A p2pool software controlled secure wallet where all payouts from each block that did not meet the payout threshold would be stored, and then paid out to miners when they reach the threshold. 

Miners who reach the minimum payout during a round are paid directly from the generation tx, while payments for all miners under the threshold are are sent to and stored in the wallet, and payouts are made from the wallet when the threshold is reached. 

The caveat is that this needs to be handled entirely by p2pool in a decentralized and trust less way with no single "admin" overseeing them. How to handle the keys securely without anyone but the p2pool code knowing them is the challenge, maybe the guys from the dark wallet or Armory teams would have some technically viable suggestions.

[b]Difficulty[/b]
An automated "variable and weighted" share difficulty would need to be assigned to miners based on hash rate. Every major pool in existence does it, we just need a method of doing it across the p2pool block chain in a way that is not easily manipulated. Perhaps the MIT Bitcoin club would be interested in tackling such a challenge?

[b]End game:[/b] Protect the network, encourage miners to participate actively in both Bitcoin and p2pools decentralized nature.

Even without active development p2pool is a strong "brand" in the Bitcoin space, most miners are aware of it, and its challenges. Overcoming its biggest challenges along with a decent marketing push (by us collectively) spreading the word could easily push p2pool to the top and keep Bitcoin strong and trust less for the future.

Just my 0.02 bits.

I think the biggest obstacle with your proposal is:

A p2pool software controlled secure wallet where all payouts from each block that did not meet the payout threshold would be stored, and then paid out to miners when they reach the threshold.

I'm not sure the concept of a secured wallet that needs to maintain thresholds for users and process payouts dependent upon when a user crosses that threshold can be translated into a decentralized model.  Centralized pools like GHash/BTCGuild/Eligius/Slush all have their own payout models and can pull this off because they are indeed centralized.

We need a way to reduce the variance that will inevitably occur as more miners are added to the p2pool network.  As I previously wrote, this is what I believe is holding p2pool back from becoming a more mainstream option.  Think of it like this: if every single BTC miner decided to join the p2pool network that means ~100PH/s and share difficulty would become 1/20th of the current BTC block difficulty.

Code:

Current BTC difficulty: 13462580114.5253
Divide that by 20: 673129005.726265

A share difficulty of 673.1M is not a feasible model.  Assuming I have 1TH/s:

Code:

Difficulty * 2**32 / hashrate / 86400 = number of days to find a share
673129005.726265 * 2**32 / 1000000000000 / 86400 = 33.46142437017714

Over a month's hashing at 1TH/s to expect to find a single share, assuming the difficulty remained constant, which it wouldn't, because BTC difficulty is going to adjust every 2016 blocks.  You're looking at needing 12TH/s or more to expect to get a share onto the chain within the 3 day payout window.  Nobody besides the "big guys" currently has that kind of hashing power.  The closest you're going to get will come in August when Spondoolies-Tech starts shipping their 6TH/s SP30s.  If you happened to get in on RoadStress' group buy around Easter time, 2 of those would have cost you about $9000 USD.

Of course, the example I gave is completely contrived.  Nobody expects and/or believes that p2pool will become the sole BTC mining operation in existence.  Maybe we do nothing at all, and just accept the fact that unless you increase your own hashing power, you will experience more and more variance as your hash rate becomes a smaller and smaller portion of the p2pool network's rate.  As I wrote in my first reply on this topic, the current entry level hardware to p2pool is pretty much an Antminer S1.  Soon, that entry level hardware will be 500GH/s (the Antminer S3).  Perhaps this is just the natural progression of things and changing the underlying way p2pool works to accommodate the smaller hash rates is futile.  Even if we go with a multi-tiered approach, as I suggested, how do we define those tiers?  What becomes the "cutoff" value to move from tier 1 to tier 2 to tier n?  How many of these tiers would be "enough" to effectively reduce the variance?

[windpath]

Great feedback!

I'm not sure the concept of a secured wallet that needs to maintain thresholds for users and process payouts dependent upon when a user crosses that threshold can be translated into a decentralized model.  Centralized pools like GHash/BTCGuild/Eligius/Slush all have their own payout models and can pull this off because they are indeed centralized.

You may be right, I tweeted a few of the Dark Wallet devs, some of which I have spoken with before, maybe one of them can share whether they think it is technically possible or not.

Maybe we do nothing at all, and just accept the fact that unless you increase your own hashing power, you will experience more and more variance as your hash rate becomes a smaller and smaller portion of the p2pool network's rate.

This is pretty much where we stand now. When/if a multiple PH/s miner hits the pool the minimum hash rate to even hope for a regular payout will sky rocket, and p2pool is a very attractive option if your building out a large mining operation.

Even if we go with a multi-tiered approach, as I suggested, how do we define those tiers?  What becomes the "cutoff" value to move from tier 1 to tier 2 to tier n?  How many of these tiers would be "enough" to effectively reduce the variance?

I like the concept, but it will require huge adoption to work. Correct me if I'm wrong, you would effectively be diluting any variance reduction we gain by growing in pool hash power by the number of tiers that were offered?

[windpath]

p2pool Stats Update

p2pool Stats updated to include luck per block data starting with block 305238, we will not be able to calculate luck for blocks before 305238 because we do not have the data required.



Updated reports available now in near-real time: http://minefast.coincadence.com/p2pool-stats.php

I'll be checking in occasionally, but will not be back to working on developing this further (miner and payout information tab) until July 1, family vacation...

[CartmanSPC]

3. Vertical scalability that reduces variance for all miners.

This is where the real challenges lie.

For p2pool to support large mining operations, and still be able to attract medium, small, and micro miners we need completely new share difficulty and payout structures.

The first step here has nothing to do with code, some completely new concepts need to be developed and generally accepted by p2pool miners. Once we have those concepts down, and can demonstrate them to be technically possible, we can seek out devs with the chops to pull it off.

P2Pool in it's current form is just not scalable. As others have pointed out the current sharechain model only becomes worse as more hashrate is added.

Perhaps a model where we have "super nodes" that split out smaller sharechains could be considered. I also think the concept of pseudo shares in-between real shares should be eliminated or modified so that work is not wasted.

[OgNasty]

3. Vertical scalability that reduces variance for all miners.

This is where the real challenges lie.

For p2pool to support large mining operations, and still be able to attract medium, small, and micro miners we need completely new share difficulty and payout structures.

The first step here has nothing to do with code, some completely new concepts need to be developed and generally accepted by p2pool miners. Once we have those concepts down, and can demonstrate them to be technically possible, we can seek out devs with the chops to pull it off.

This may seem like no big deal, but I assure you it is. Here is my first crack at it:

Payouts
While miners meeting a certain threshold could still be paid directly from the generation TX, smaller miners under the payout threshold when a block is found, lets say BTC0.01 to start, would be able to see their p2pool earnings down to the Satoshi in real time, but would not receive a payout until reaching the threshold.

We (read as: nonnakip) are currently developing pretty much this exact thing right now for our node at nastyfans.org:9332.  We have been calling it P2Pool PPS...

[norgan]

api shows queue => 9999