Bitcoin Forum
July 23, 2018, 02:17:17 PM *
News: Latest stable version of Bitcoin Core: 0.16.1  [Torrent]. (New!)
 
   Home   Help Search Donate Login Register  
Pages: « 1 ... 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 [791] 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 »
  Print  
Author Topic: [1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool  (Read 2559429 times)
Acejam
Sr. Member
****
Offline Offline

Activity: 291
Merit: 251


View Profile
August 28, 2017, 09:12:09 PM
 #15801

I've been running P2Pool in a Docker container for over a year now with great success so I've decided to make my Docker container public. Nothing super special here, but this should allow you to easily run P2Pool Core regardless of what platform you're on and not have to worry about binaries. (As long as you have Docker installed) I currently have a tag up for 17.0 on Docker Hub, but can add more later once future releases are made.

https://hub.docker.com/r/acejam/p2pool/
https://github.com/acejam/docker-p2pool

Code:
docker run -p 9332:9332 -p 9333:9333 acejam/p2pool:17.0 ${p2pool_command_options_here}
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
Blue Bear
Jr. Member
*
Offline Offline

Activity: 31
Merit: 0

If a Bear sits on a miner does it break?


View Profile
August 29, 2017, 01:58:19 AM
 #15802

Thanks everyone that helped.

I have my node back online and running with no issues.

I have learned alot in these past few days getting my node upgraded to v17.

BB
kano
Legendary
*
Offline Offline

Activity: 2520
Merit: 1045


Linux since 1997 RedHat 4


View Profile
August 29, 2017, 02:04:45 AM
 #15803

...
There is always a statistically significant chance (albeit a small one) that the next mainnet P2Pool block would be found in less than five minutes after the previous block, due to how Bitcoin mining inherently relies on chance.
...
Actually it's a simple CDF calculation that shows how bad it is to limit block sizes to taking an exorbitant 5 minutes to get to 1MB ...

5 minutes is a 50% block on an expected 10 minute network.
So the CDF is 0.39346934028737 or 1 in 1.6 blocks (60.65%) will be over 5 minutes

or reversing that ... 1 in 2.54 blocks (39.3%) will be under 5 minutes ... yeah that's a big % of blocks, not "albeit a small one" at all.

I'm often surprised at how little maths people understand about Bitcoin when they are considered experts in it ... and/or spending money mining on it ...
Note of course, this last comment is directed at someone else not you ... Smiley

Pool: https://kano.is Here on Bitcointalk: Forum BTC: 1KanoPb8cKYqNrswjaA8cRDk4FAS9eDMLU
FreeNode IRC: irc.freenode.net channel #kano.is Majority developer of the ckpool code
Help keep Bitcoin secure by mining on pools with full block verification on all blocks - and NO empty blocks!
Blue Bear
Jr. Member
*
Offline Offline

Activity: 31
Merit: 0

If a Bear sits on a miner does it break?


View Profile
August 29, 2017, 02:10:50 AM
 #15804


Actually it's a simple CDF calculation that shows how bad it is to limit block sizes to taking an exorbitant 5 minutes to get to 1MB ...

5 minutes is a 50% block on an expected 10 minute network.
So the CDF is 0.39346934028737 or 1 in 1.6 blocks (60.65%) will be over 5 minutes

or reversing that ... 1 in 2.54 blocks (39.3%) will be under 5 minutes ... yeah that's a big % of blocks, not "albeit a small one" at all.

I'm often surprised at how little maths people understand about Bitcoin when they are considered experts in it ... and/or spending money mining on it ...
Note of course, this last comment is directed at someone else not you ... Smiley

can you show me the algebraic formula so I can understand it better .... Cheesy
kano
Legendary
*
Offline Offline

Activity: 2520
Merit: 1045


Linux since 1997 RedHat 4


View Profile
August 29, 2017, 02:12:34 AM
 #15805


Actually it's a simple CDF calculation that shows how bad it is to limit block sizes to taking an exorbitant 5 minutes to get to 1MB ...

5 minutes is a 50% block on an expected 10 minute network.
So the CDF is 0.39346934028737 or 1 in 1.6 blocks (60.65%) will be over 5 minutes

or reversing that ... 1 in 2.54 blocks (39.3%) will be under 5 minutes ... yeah that's a big % of blocks, not "albeit a small one" at all.

I'm often surprised at how little maths people understand about Bitcoin when they are considered experts in it ... and/or spending money mining on it ...
Note of course, this last comment is directed at someone else not you ... Smiley

can you show me the algebraic formula so I can understand it better .... Cheesy
https://en.wikipedia.org/wiki/Cumulative_distribution_function

Pool: https://kano.is Here on Bitcointalk: Forum BTC: 1KanoPb8cKYqNrswjaA8cRDk4FAS9eDMLU
FreeNode IRC: irc.freenode.net channel #kano.is Majority developer of the ckpool code
Help keep Bitcoin secure by mining on pools with full block verification on all blocks - and NO empty blocks!
Cryptonomist
Newbie
*
Offline Offline

Activity: 27
Merit: 0


View Profile
August 29, 2017, 09:16:13 AM
 #15806


Actually it's a simple CDF calculation that shows how bad it is to limit block sizes to taking an exorbitant 5 minutes to get to 1MB ...

5 minutes is a 50% block on an expected 10 minute network.
So the CDF is 0.39346934028737 or 1 in 1.6 blocks (60.65%) will be over 5 minutes

or reversing that ... 1 in 2.54 blocks (39.3%) will be under 5 minutes ... yeah that's a big % of blocks, not "albeit a small one" at all.

I'm often surprised at how little maths people understand about Bitcoin when they are considered experts in it ... and/or spending money mining on it ...
Note of course, this last comment is directed at someone else not you ... Smiley

can you show me the algebraic formula so I can understand it better .... Cheesy

Hello,

The Bitcoin mining process can be modelled as a Poisson process. From this we can calculate the probabilistic interarrival times (the time between two events, here finding of two blocks), which are exponentially distributed with lambda = 1/600. The probability of getting interarrival times (IAT) larger than 5 min or 300 sec is: Pr{IAT > 300} = e^(-300/600) ≈ 0.60653 = 60.65%. So the probability of getting two blocks less than 300 sec from each other is: Pr{IAT < 300} = 1 - Pr{IAT > 300} ≈ 0.39347 = 39.34%. This is the probability of the Bitcoin network finding a block in less than 300 sec from the previous one.

We can off course do the same for P2pool. The probability that the P2pool network finds a block less than 300 sec from the other is (assuming P2Pool has 0.1% of the total hashing power of the Bitcoin network, which is more than it currently has): lambda is now 1/600000. Pr{IAT < 300} = 1 - e^(-300/600000) ≈ 0.0005 = 0.05%.

(This is an exercise I did for my self. It may contain errors related to how P2pool works. If I made errors please make me aware of them so I can learn from it.). We can also find the expected number of KBs that P2pool can add to the block if P2pool finds a block after 5 min, assuming 100 KB per share, and no difficulty change of the dynamic difficulty adjustment mechanism, and after the last found block by the network, P2Pool had to start again from scratch, meaning 0 KBs). As explained above the finding of shares is also a Poisson process. The probability to find no shares in an interval of length 300 sec is: Pr(N(300) = 0) = (((300/30)^0)/0!)*e^(-300/30) ≈ 0.00005. In this case P2pool will add 0 KBs to a block. We can continue this until an ridiculous and irrelevant number of shares, so I will truncate it at 20 shares:
Pr(N(300) = 1) ≈ 0.00045   100KB
Pr(N(300) = 2) ≈ 0.00227   200KB
Pr(N(300) = 3) ≈ 0.00757   300KB
...
Pr(N(300) = 9) ≈ 0.12511   900KB
Pr(N(300) = 10) ≈ 0.12511  1000KB
Pr(N(300) = 11) ≈ 0.11374  1100KB
...
Pr(N(300) = 19) ≈ 0.00373  1900KB
Pr(N(300) = 20) ≈ 0.00187  2000KB
From these results we can calculate the weighted average: 996.55KB. So if a Bitcoin block is found by P2pool 300 sec after the previous than the expected amount of KBs in that block is 996.55KB.

I hope this clarifies a thing or two. Or if you find errors please make me aware of them so I can learn from it.
kano
Legendary
*
Offline Offline

Activity: 2520
Merit: 1045


Linux since 1997 RedHat 4


View Profile
August 29, 2017, 12:59:59 PM
 #15807


Actually it's a simple CDF calculation that shows how bad it is to limit block sizes to taking an exorbitant 5 minutes to get to 1MB ...

5 minutes is a 50% block on an expected 10 minute network.
So the CDF is 0.39346934028737 or 1 in 1.6 blocks (60.65%) will be over 5 minutes

or reversing that ... 1 in 2.54 blocks (39.3%) will be under 5 minutes ... yeah that's a big % of blocks, not "albeit a small one" at all.

I'm often surprised at how little maths people understand about Bitcoin when they are considered experts in it ... and/or spending money mining on it ...
Note of course, this last comment is directed at someone else not you ... Smiley

can you show me the algebraic formula so I can understand it better .... Cheesy

Hello,

The Bitcoin mining process can be modelled as a Poisson process. From this we can calculate the probabilistic interarrival times (the time between two events, here finding of two blocks), which are exponentially distributed with lambda = 1/600. The probability of getting interarrival times (IAT) larger than 5 min or 300 sec is: Pr{IAT > 300} = e^(-300/600) ≈ 0.60653 = 60.65%. So the probability of getting two blocks less than 300 sec from each other is: Pr{IAT < 300} = 1 - Pr{IAT > 300} ≈ 0.39347 = 39.34%. This is the probability of the Bitcoin network finding a block in less than 300 sec from the previous one.

We can off course do the same for P2pool. The probability that the P2pool network finds a block less than 300 sec from the other is (assuming P2Pool has 0.1% of the total hashing power of the Bitcoin network, which is more than it currently has): lambda is now 1/600000. Pr{IAT < 300} = 1 - e^(-300/600000) ≈ 0.0005 = 0.05%.

(This is an exercise I did for my self. It may contain errors related to how P2pool works. If I made errors please make me aware of them so I can learn from it.). We can also find the expected number of KBs that P2pool can add to the block if P2pool finds a block after 5 min, assuming 100 KB per share, and no difficulty change of the dynamic difficulty adjustment mechanism, and after the last found block by the network, P2Pool had to start again from scratch, meaning 0 KBs). As explained above the finding of shares is also a Poisson process. The probability to find no shares in an interval of length 300 sec is: Pr(N(300) = 0) = (((300/30)^0)/0!)*e^(-300/30) ≈ 0.00005. In this case P2pool will add 0 KBs to a block. We can continue this until an ridiculous and irrelevant number of shares, so I will truncate it at 20 shares:
Pr(N(300) = 1) ≈ 0.00045   100KB
Pr(N(300) = 2) ≈ 0.00227   200KB
Pr(N(300) = 3) ≈ 0.00757   300KB
...
Pr(N(300) = 9) ≈ 0.12511   900KB
Pr(N(300) = 10) ≈ 0.12511  1000KB
Pr(N(300) = 11) ≈ 0.11374  1100KB
...
Pr(N(300) = 19) ≈ 0.00373  1900KB
Pr(N(300) = 20) ≈ 0.00187  2000KB
From these results we can calculate the weighted average: 996.55KB. So if a Bitcoin block is found by P2pool 300 sec after the previous than the expected amount of KBs in that block is 996.55KB.

I hope this clarifies a thing or two. Or if you find errors please make me aware of them so I can learn from it.
Firstly, the time between any 2 blocks is what matters, when p2pool is the 2nd block, it doesn't mater who found the previous one.

If p2pool happens to get two in a row at the moment, well, good to know, the chance of that is simply the size of p2pool over the whole network Smiley

The point of the problem is that standard p2pool will only allow to add 100KB per 30 seconds given 30 seconds is the expected time per share-chain share.
That's a simple linear function ... 300s = 30s x 10  = 100KB x 10 = 1MB Smiley
Until 300s it's not expected to be a full block, thus the CDF comes into play to simply say that 39.3% of blocks found on p2pool will be 900KB or less.

Most everyone on the bitcoin network itself is expected to use up the same best transactions each block found and thus all pools including p2pool should normally reset back to zero when a block is found, but then again, as before, p2pool will produce, on average, a 100KB block at the first 30s after any previous block, no matter who found it - p2pool or someone else. That's not the case by design of a 'standard' pool, though a lot of pools create empty or partial blocks in the first work generated after a block, but then switch to a full block very shortly after that.

As for the CDF, it's simply the area under the poisson distribution curve - which you normally integrate a function to get the area under a curve, but in the case of the poisson there's simple approximates to calculate it given certain limitations.

The simplest calculation is (1 - e^(-x)) where x is the ratio - e.g. 50% = 0.5
As long as x isn't too small, this gives a good approximate.
i.e. 1 - e^(-0.5) = 0.393... or 39.3%

... thus a simple (and correct) explanation Wink

Pool: https://kano.is Here on Bitcointalk: Forum BTC: 1KanoPb8cKYqNrswjaA8cRDk4FAS9eDMLU
FreeNode IRC: irc.freenode.net channel #kano.is Majority developer of the ckpool code
Help keep Bitcoin secure by mining on pools with full block verification on all blocks - and NO empty blocks!
jtoomim
Hero Member
*****
Offline Offline

Activity: 818
Merit: 1001


View Profile WWW
August 29, 2017, 03:12:41 PM
 #15808

The point of the problem is that standard p2pool will only allow to add 100KB per 30 seconds given 30 seconds is the expected time per share-chain share.
That's a simple linear function ... 300s = 30s x 10  = 100KB x 10 = 1MB Smiley
Note: 100 kB per 30 second share is a bit of an oversimplification of how the mainnet code works. In reality, it's sometimes more than 100 kB (when transactions can be resurrected from a previous block round), and often less than 100 kB (when different nodes are using different transaction sets, or when the transactions in a template change, or when transactions are large). I have looked closely at the shares mined for two block on mainnet. In the first instance, it took around 20 shares before it got up to 1 MB instead of 10 shares. In the second instance, it hadn't gotten to 1 MB even after 16 shares. In both, there were several shares that were actually smaller than the previous share.

Hosting bitcoin miners for $65 to $80/kW/month on clean, cheap hydro power.
http://Toom.im
kano
Legendary
*
Offline Offline

Activity: 2520
Merit: 1045


Linux since 1997 RedHat 4


View Profile
August 29, 2017, 03:26:41 PM
 #15809

The point of the problem is that standard p2pool will only allow to add 100KB per 30 seconds given 30 seconds is the expected time per share-chain share.
That's a simple linear function ... 300s = 30s x 10  = 100KB x 10 = 1MB Smiley
Note: 100 kB per 30 second share is a bit of an oversimplification of how the mainnet code works. In reality, it's sometimes more than 100 kB (when transactions can be resurrected from a previous block round), and often less than 100 kB (when different nodes are using different transaction sets, or when the transactions in a template change, or when transactions are large). I have looked closely at the shares mined for two block on mainnet. In the first instance, it took around 20 shares before it got up to 1 MB instead of 10 shares. In the second instance, it hadn't gotten to 1 MB even after 16 shares. In both, there were several shares that were actually smaller than the previous share.
Well you can probably add the word 'expected' in every sentence I wrote, but that was the point of it all, the expected averages (or limits), not the exact what happened at any particular event.

Pool: https://kano.is Here on Bitcointalk: Forum BTC: 1KanoPb8cKYqNrswjaA8cRDk4FAS9eDMLU
FreeNode IRC: irc.freenode.net channel #kano.is Majority developer of the ckpool code
Help keep Bitcoin secure by mining on pools with full block verification on all blocks - and NO empty blocks!
jtoomim
Hero Member
*****
Offline Offline

Activity: 818
Merit: 1001


View Profile WWW
August 29, 2017, 04:20:07 PM
 #15810

I just pushed a performance improvement commit to 1mb_segwit. It seems to reduce CPU usage on pypy about 45% and RAM usage about 65% by improving the code for data deserialization. Share loading time went from 31s to 17s on pypy and from 101s to 79s on CPython. There may be additional performance benefits to be had by doing some work on data serialization in addition to deserialization.

Hosting bitcoin miners for $65 to $80/kW/month on clean, cheap hydro power.
http://Toom.im
Comandante77
Newbie
*
Offline Offline

Activity: 12
Merit: 0


View Profile
August 29, 2017, 06:21:58 PM
 #15811

Strange situation. Installed and ran p2pool in solo mining 5 days ago for DASH. 3 or 4 days everything was fine.
Smth about 4-5 blocks per 24h. And now 0 blocks per 24h.

May be some tuning will help?

Miners are Baikal.
ChicagoCryptocurrency
Member
**
Offline Offline

Activity: 84
Merit: 10


View Profile
August 29, 2017, 06:58:18 PM
 #15812

Hey, Guys! so glad to see p2pool still in discussion and alive I had a question in regards to Bitcoin Cash can I mine it with p2pool and is there any thread I can be directed to for the resources
@Kano Good to see you still active Thanks for all the help you provide to the Bitcoin Community! you the Man!

Thanks
jtoomim
Hero Member
*****
Offline Offline

Activity: 818
Merit: 1001


View Profile WWW
August 29, 2017, 07:02:33 PM
 #15813

Bitcoin Cash can I mine it with p2pool
Not yet. I've been working on adding support for it to jtoomimnet, but it's only about 40% done.

Hosting bitcoin miners for $65 to $80/kW/month on clean, cheap hydro power.
http://Toom.im
ChicagoCryptocurrency
Member
**
Offline Offline

Activity: 84
Merit: 10


View Profile
August 29, 2017, 07:59:44 PM
 #15814

Bitcoin Cash can I mine it with p2pool
Not yet. I've been working on adding support for it to jtoomimnet, but it's only about 40% done.
keep me posted would love to check it out
belcher
Sr. Member
****
Offline Offline

Activity: 258
Merit: 307


View Profile
August 29, 2017, 09:45:14 PM
 #15815

Thanks for your reply jtoomim.

I would like to strip all transaction data out of the share data structure in the share chain in order to cut this memory footprint issue and reduce the CPU requirements for processing shares, but until that is done, increasing the share chain length is a bad idea.

How would you do this?

In fact, why do shares need references to transactions at all? They already commit to all the transactions via the merkle root. If a hasher turns out to have created an invalid block then they will lose money.

1HZBd22eQLgbwxjwbCtSjhoPFWxQg8rBd9
JoinMarket - CoinJoin that people will actually use.
PGP fingerprint: 0A8B 038F 5E10 CC27 89BF CFFF EF73 4EA6 77F3 1129
jtoomim
Hero Member
*****
Offline Offline

Activity: 818
Merit: 1001


View Profile WWW
August 29, 2017, 10:18:44 PM
 #15816

It's part of p2pool's fast block propagation technology, which for p2pool is part of the consensus layer. P2pool had one of the first systems for propagating a block without transmitting the full serialized size of the block over the network. It did this by making the share objects encode the list of transactions in a block by encoding them as a (previous share's index, transaction index) tuple. Usually, this encoding takes two or three bytes per transaction. For transactions that are being included in a share for the first time, p2pool encodes them in shares as the full 32-byte transaction hash, and assumes that the recipient of the share will already have the mapping from the 32-byte hash to the full transaction. (This is ensured by code in the p2p layer.)

P2pool in principle does not need to include the full (encoded) transaction list in the share object. But getting rid of that means getting rid of p2pool's fast block propagation tech. Since modern fast block propagation tech has advanced so much outside of p2pool (e.g. FIBRE, Falcon, Xthin, Compact Blocks), I think that p2pool should strip it out and leave that task to the better (and faster) code in bitcoind. Alternately, if fast block propagation in p2pool is desirable, it should probably be done outside of the consensus layer, so that it only gets used for actual blocks (where the resource usage is actually useful) rather than on every single share.

Hosting bitcoin miners for $65 to $80/kW/month on clean, cheap hydro power.
http://Toom.im
belcher
Sr. Member
****
Offline Offline

Activity: 258
Merit: 307


View Profile
August 30, 2017, 04:39:15 PM
 #15817

I came up with an idea to improve p2pool's scalability and greatly reduce payout variance: https://bitcointalk.org/index.php?topic=2135429.0

Please tell me what you think.

1HZBd22eQLgbwxjwbCtSjhoPFWxQg8rBd9
JoinMarket - CoinJoin that people will actually use.
PGP fingerprint: 0A8B 038F 5E10 CC27 89BF CFFF EF73 4EA6 77F3 1129
Meuh6879
Legendary
*
Offline Offline

Activity: 1512
Merit: 1000



View Profile
August 30, 2017, 07:03:20 PM
 #15818

Code:
2017-08-30 19:00:53 Switchover imminent. Upgraded: 82.511% Threshold: 95.000%

jtoomim
Hero Member
*****
Offline Offline

Activity: 818
Merit: 1001


View Profile WWW
August 30, 2017, 08:28:53 PM
 #15819

Meanwhile, on the jtoomimnet branch of p2pool...
Code:
2017-08-30 13:29:59.819445 Generating a share with 1000644 bytes (36617 new) and 1779 transactions (49 new)

Hosting bitcoin miners for $65 to $80/kW/month on clean, cheap hydro power.
http://Toom.im
frodocooper
Moderator
Full Member
*
Offline Offline

Activity: 236
Merit: 277


View Profile
August 31, 2017, 06:07:15 AM
 #15820

Here is a more elegant way of getting PyPy up and running for jtoomimnet P2Pool on Ubuntu 16.04 and later.

1. Add the PyPy PPA. (The PyPy in the PPA is more frequently updated than the PyPy in Ubuntu's package repository.)

Code:
$ sudo add-apt-repository ppa:pypy/ppa
$ sudo apt-get update

2. Install PyPy and other dependencies.

Code:
$ sudo apt-get install pypy pypy-dev gcc build-essential git

3. Install pip.

Code:
$ wget https://bootstrap.pypa.io/get-pip.py
$ sudo pypy get-pip.py
$ sudo rm -rf get-pip.py

4. Update pip, setuptools, and wheel.

Code:
$ sudo pypy -m pip install --upgrade pip setuptools wheel

5. Install Twisted, zope.interface, and incremental.

Code:
$ sudo pypy -m pip install twisted zope.interface incremental

6. Download the 1mb_segwit branch of jtoomimnet P2Pool.

Code:
$ git clone https://github.com/jtoomim/p2pool.git
$ cd p2pool
$ git checkout 1mb_segwit

7. Run P2Pool.

Code:
$ pypy run_p2pool.py

For a list of jtoomimnet P2Pool command-line options:

Code:
$ pypy run_p2pool.py --help



Using pip to install third-party Python packages (e.g., Twisted, zope.interface, and incremental), instead of downloading and installing them directly, enables cleaner installs and uninstalls, and simpler and faster updating of the packages. For example,

(a) to update Twisted, zope.interface, and incremental:

Code:
$ sudo pypy -m pip install --upgrade twisted zope.interface incremental

(b) to remove Twisted, zope.interface, and incremental (if you no longer need them):

Code:
$ sudo pypy -m pip uninstall twisted zope.interface incremental

A list of handy pip commands may be found here.

A more comprehensive guide to using pip may be found here.
Pages: « 1 ... 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 [791] 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 »
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!