[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[smoothrunnings]

39.2% so far, 6.4 hours remaining.

I'm going to go ahead and make a bunch of min-diff shares to get some headroom.

Your saying the code in the window is all of how to get p2pool going?

Does the linux for Windows blow away what's currently installed or does it run in a virtual session like they did for Windows 7 with the XP emulator? My currently machine is joined to my home network domain, am I going to loose that too?

Thanks,

[1714189048]

1714189048

[1714189048]

1714189048

[1714189048]

1714189048

[jtoomim]

No, not that post. That post is msg21143072, but the one with instructions for Pypy on WSL/Windows is msg21025074. Since you're having trouble finding it, I'll just quote the whole thing.

One of the dependencies for p2pool on Windows is pywin32. That's available on CPython, but it is not available on PyPy. The only way to get p2pool running on PyPy on Windows is to use WSL or a VM to avoid the pywin32 dependency. PyPy is about 3x as fast as CPython due to the wonders of JIT compilation, whereas the WSL layer only slows it down by about 1-5%.

Instructions below.

I tried installing pypy/p2pool on Windows myself, and while I got a bit farther than you did by installing twisted from source, I eventually ran into a roadblock because twisted requires pywin32 on Windows and pypy does not support pywin32.

However, I had better luck when I tried to run p2pool with pypy on the Windows Subsystem for Linux (Ubuntu for Windows). If you have Windows 10, I suggest you give that a try:

https://msdn.microsoft.com/en-us/commandline/wsl/install_guide

Once you have Ubuntu for Windows installed, run bash and then copy-paste these lines into your bash command line:

Code:

sudo apt-get update
sudo apt-get install pypy pypy-dev pypy-setuptools gcc build-essential git

wget https://bootstrap.pypa.io/ez_setup.py -O - | sudo pypy
sudo rm setuptools-*.zip

wget https://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.1.3.tar.gz#md5=9ae3d24c0c7415deb249dd1a132f0f79
tar zxf zope.interface-4.1.3.tar.gz
cd zope.interface-4.1.3/
sudo pypy setup.py install
cd ..
sudo rm -r zope.interface-4.1.3*

wget https://pypi.python.org/packages/source/T/Twisted/Twisted-15.4.0.tar.bz2
tar jxf Twisted-15.4.0.tar.bz2
cd Twisted-15.4.0
sudo pypy setup.py install
cd ..
sudo rm -r Twisted-15.4.0*

git clone https://github.com/jtoomim/p2pool.git
cd p2pool
git checkout 1mb_segwit

Those commands should also work if you're on native Ubuntu or Debian Linux

You'll also need to install and run your bitcoind of choice (I recommend btc1, because I think segwit2x is the future of Bitcoin) and edit ~/.bitcoin/bitcoin.conf with your bitcoind's RPC username and password (or pass it to p2pool from the pypy run_p2pool.py command line), then from your p2pool directory, run

Code:

pypy run_p2pool.py

[smoothrunnings]

The how-to doc for Windows is the link I posted.

I typed in the first line in your install and got this, please advise.




I am using Windows 10 x64 Enterprise v1709 (Fall Creators Edition), with the Spectre patch.

[smoothrunnings]

And what do you recommend I use to install bitcoind?

I found this site, but do I need to install Python for Bitcoind?

https://degreesofzero.com/article/installing-bitcoind-on-ubuntu.html

Thanks,
 

[jtoomim]

Strange. It worked for me a few months ago. Maybe you need to do "sudo apt-get update" first?

Bitcoin Core works for bitcoind. You can also use Bitcoin ABC or Bitcoin Unlimited if you want to mine Bitcoin Cash.

[smoothrunnings]

Strange. It worked for me a few months ago. Maybe you need to do "sudo apt-get update" first?

Bitcoin Core works for bitcoind. You can also use Bitcoin ABC or Bitcoin Unlimited if you want to mine Bitcoin Cash.

Can I mine bitcoin case using a SHA256 ASIC? I haven't been mining in quite a long time. lol

The 'sudo apt-get update' did the trick. You might want to update your how-to info to include that so that others who aren't familiar with *nix can get ahead without having to ask the same question I did.

Thanks,

[smoothrunnings]

Ran into another problem. Please advise. Thanks!

[smoothrunnings]

Strange. It worked for me a few months ago. Maybe you need to do "sudo apt-get update" first?

Bitcoin Core works for bitcoind. You can also use Bitcoin ABC or Bitcoin Unlimited if you want to mine Bitcoin Cash.

Another line needs to be added before twisted is installed.

sudo apt-get install pypy-setuptools

[jtoomim]

Bitcoin Cash had a spam event 1/13-1/15 (yesterday). Many blocks were mined at 8 MB by other pools. Unfortunately, I had my BCH p2pool node using the default 2 MB limit, so I didn't see any 8 MB shares being mined by my node. But I got this:

Code:

2018-01-15 10:19:19.521629 Generating a share with 1996157 bytes, 7984628 WU (new: 6549 B, 26196 WU) in 362 tx (10 new), plus estimated gentx of 528 bytes/2112 WU
2018-01-15 10:19:19.521795 Total block size, weight: 1996765 bytes, 7986980 WU

My node (pypy, on a i7 4790k) handled the load just fine, maintaining GBT latencies of around 0.1 sec. (I did not have benchmarking enabled, so I don't have more detailed performance stats. Alas.) However, I notice that several other p2pool nodes appear to have collapsed under the load: peer counts were all over the place during the spam event, which indicates that several peers were getting CPU locked, then timing out on network replies, and eventually reconnecting and starting the cycle over again.

[jtoomim]

Can I mine bitcoin case using a SHA256 ASIC? I haven't been mining in quite a long time. lol

Yes.

[smoothrunnings]

Can P2POOL be setup for BCH?

Where do I get the info for configuring it?

Thanks,

[jtoomim]

Assuming you're using my code (1mb_segwit), run p2pool with '--net bitcoincash' as one of the arguments, and make sure a Bitcoin Cash-compatible version of bitcoind is running (e.g. Bitcoin ABC).

Keep in mind that Bitcoin Cash support is still experimental and p2pool's BCH hashrate is very low. No blocks have been mined yet on p2pool BCH.

I intend to switch a substantial portion of my hashrate (> 1 PH/s) to Bitcoin Cash shortly after the next Bitcoin block is found. That will likely not happen for a couple weeks.

[Cryptonomist]

Hello

Is there any way to make p2pool mode modern so it can use multithreading and multi cores etc?

Rewrite it from scratch in another programming language.

Python is inherently single-threaded. https://wiki.python.org/moin/GlobalInterpreterLock

It would be far easier and more effective to remove most of the transaction-handling code from p2pool, and make share objects contain only the merkle root plus the coinbase transaction (and the merkle path thereto). However, I have not had the time to do that, and nobody else seems to be interested in doing it. Even though it would be much easier, it's not super easy to do.

Note: Bitcoin Core is *also* mostly single-threaded, although not quite to the same extent as p2pool.

Wouldn't it be possible to rewrite the code and compile it with cython? This avoids the GIL. It also easily allows to mix python with C/C++ code, which might be useful for some parts of the code, and it would avoid rewriting everything in another language. However, I don't think twisted can be used by cython (I might be wrong on this), so this would need to be replaced with something different to handle the asynchronous networking (which might be quite a lot of work).

I've been running the 1mb_segwit code for quite some time now. It seems that the optimalisations done by jtoomim and the use of pypy has reduced the single core load quite considerably compared to the main code. So, although some people still have problems, it alleviated in my opinion the single thread bottleneck considerably.

@jtoomim: what parts of the transaction handling can be removed in your opinion?
 

[jtoomim]

Cython and pypy are essentially incompatible with each other. To use Cython, we would have to Cythonize enough code that the Cython+CPython combo was faster than pypy. That will require Cythonizing a ton of code.

It might also be possible to rewrite the pack.py code as a C module using CFFI, which would be compatible with both pypy and CPython, but that would complicate the build and installation process for p2pool.

Cythonization would probably take longer than just removing transactions from share objects and eliminating the transaction-related code from the p2p layer. Multithreading using Cython would be even harder than plain Cythonization, and would not provide as much of a benefit.

Multithreading and Cythonization aren't needed if we reduce the amount of work done by p2pool by 90%.

It also appears that there is a lot of duplicate work that's being performed by p2pool during the creation of new work for miners that can be eliminated by caching intermediate results.

The only transaction handling code we really need is handling the transactions handed to us by getblocktemplate, calculating the merkle root hash and merkle path to the coinbase transaction, and then reassembling the full block when a block is found and handing that block back to bitcoind. It used to be nice to include the transactions in compressed form in the shares we send to peers to allow them to publish the blocks simultaneously on their nodes as well, but now that we have FIBRE, Compact Blocks, and XThin, p2pool is much slower at this task than bitcoind, so that no longer makes sense for p2pool to do.

[jtoomim]

If there are any decent Python coders who are interested in contributing to p2pool, I'm willing to pay for good PRs.

[jtoomim]

I have a node running Litecoin using the 1mb_segwit code plus a few new commits:

http://woff.toom.im:9327

I'm going to dump some more hashrate on it in a sec and see if it can mine blocks.

Edit: ran into bug 241 (https://github.com/p2pool/p2pool/issues/241), testing the fix...

Edit2: ran into another bug. Shares are being rejected by other nodes on the ltc network and connections killed.

Edit3: Looks like the above bug was an incompatibility of this timestamp non-clipping commit with mainnet nodes. I added a change that clips timestamps if share_version < 32, and that seems to have resolved that issue so far. Hashing is resuming with ~50 GH/s.

Edit4: Upped hashrate to 105 GH/s. Also moved ~700 TH/s to Bitcoin Cash via http://woff.toom.im:9348.

[sawa]

I have a node running Litecoin using the 1mb_segwit code plus a few new commits:

http://woff.toom.im:9327

I'm going to dump some more hashrate on it in a sec and see if it can mine blocks.

Edit: ran into bug 241 (https://github.com/p2pool/p2pool/issues/241), testing the fix...

Edit2: ran into another bug. Shares are being rejected by other nodes on the ltc network and connections killed.

Edit3: Looks like the above bug was an incompatibility of this timestamp non-clipping commit with mainnet nodes. I added a change that clips timestamps if share_version < 32, and that seems to have resolved that issue so far. Hashing is resuming with ~50 GH/s.

Edit4: Upped hashrate to 105 GH/s. Also moved ~700 TH/s to Bitcoin Cash via http://woff.toom.im:9348.

Can I still use your version of LTC p2pool?
Which branch to download?

[jtoomim]

Can I still use your version of LTC p2pool? Which branch to download?

Yes, go ahead. It's just the same 1mb_segwit branch, but with a few new commits. The new commits shouldn't adversely effect BTC or BCH, but I haven't tested them there yet.

I mined this LTC block with my code last night: http://explorer.litecoin.net/block/df565f1dbfdd5bfd2dd14b3065eb77b12bd1c9a0a222aec7e3cc9fde356726c2. I'd say it's ready for open beta.

If we get too many shares using my code, it will hard fork. We should try to delay that until my code is in p2pool/p2pool master. The threshold for that should be 95%. It might be a good idea to avoid getting over 50% until the code is in master so that people don't get confused when a git pull doesn't fix the warnings about obsolete software. Edit: I switched my node to vote for v17 for now. That should prevent premature forking on LTC.

[kr1z1s]

Can I still use your version of LTC p2pool? Which branch to download?

Yes, go ahead. It's just the same 1mb_segwit branch, but with a few new commits. The new commits shouldn't adversely effect BTC or BCH, but I haven't tested them there yet.

I mined this LTC block with my code last night: http://explorer.litecoin.net/block/df565f1dbfdd5bfd2dd14b3065eb77b12bd1c9a0a222aec7e3cc9fde356726c2. I'd say it's ready for open beta.

If we get too many shares using my code, it will hard fork. We should try to delay that until my code is in p2pool/p2pool master. The threshold for that should be 95%. It might be a good idea to avoid getting over 50% until the code is in master so that people don't get confused when a git pull doesn't fix the warnings about obsolete software.

Why displays the same thing several times? This is LTC, you source

Code:

2018-01-19 21:32:26.370926 Generating a share with 90674 bytes, 358106 WU (new: 51019 B, 202765 WU) in 169 tx (35 new), plus est gentx of 4343 bytes/17408 WU
2018-01-19 21:32:26.371066 Total block stripped size=93567 B, full size=95097 B,  weight: 375754 WU
2018-01-19 21:32:26.389393 Generating a share with 90674 bytes, 358106 WU (new: 51019 B, 202765 WU) in 169 tx (35 new), plus est gentx of 4343 bytes/17408 WU
2018-01-19 21:32:26.389545 Total block stripped size=93567 B, full size=95097 B,  weight: 375754 WU
2018-01-19 21:32:26.406837 Generating a share with 90674 bytes, 358106 WU (new: 51019 B, 202765 WU) in 169 tx (35 new), plus est gentx of 4343 bytes/17408 WU
2018-01-19 21:32:26.406988 Total block stripped size=93567 B, full size=95097 B,  weight: 375754 WU
2018-01-19 21:32:26.424838 Generating a share with 90674 bytes, 358106 WU (new: 51019 B, 202765 WU) in 169 tx (35 new), plus est gentx of 4343 bytes/17408 WU
2018-01-19 21:32:26.424983 Total block stripped size=93567 B, full size=95097 B,  weight: 375754 WU
2018-01-19 21:32:26.442323 Generating a share with 90674 bytes, 358106 WU (new: 51019 B, 202765 WU) in 169 tx (35 new), plus est gentx of 4343 bytes/17408 WU
2018-01-19 21:32:26.442502 Total block stripped size=93567 B, full size=95097 B,  weight: 375754 WU
2018-01-19 21:32:26.460687 Generating a share with 90674 bytes, 358106 WU (new: 51019 B, 202765 WU) in 169 tx (35 new), plus est gentx of 4343 bytes/17408 WU
2018-01-19 21:32:26.460876 Total block stripped size=93567 B, full size=95097 B,  weight: 375754 WU
2018-01-19 21:32:26.478698 Generating a share with 90674 bytes, 358106 WU (new: 51019 B, 202765 WU) in 169 tx (35 new), plus est gentx of 4343 bytes/17408 WU
2018-01-19 21:32:26.478868 Total block stripped size=93567 B, full size=95097 B,  weight: 375754 WU
2018-01-19 21:32:26.499163 Generating a share with 90674 bytes, 358106 WU (new: 51019 B, 202765 WU) in 169 tx (35 new), plus est gentx of 4343 bytes/17408 WU
2018-01-19 21:32:26.499333 Total block stripped size=93567 B, full size=95097 B,  weight: 375754 WU
2018-01-19 21:32:26.517609 Generating a share with 90674 bytes, 358106 WU (new: 51019 B, 202765 WU) in 169 tx (35 new), plus est gentx of 4343 bytes/17408 WU

[smoothrunnings]

Do I need to anything to the "pypy run_p2pool.py" line to start the server in BTC?

Hate to install the GUI for ubuntu so I can see where I was at for the block on bitcoin-qt as for some reason I couldn't get an update from bitcoind that I might make a shortcut to run p2pool.

Thanks,