notbatman
Legendary
Offline
Activity: 2212
Merit: 1038
|
|
October 06, 2015, 02:49:10 PM |
|
CPU usage results for pypy vs python using branch jtoomim_performance of p2pool after 21.7 hours:
Regular python: 68 minutes 3 seconds, 5.22% average
pypy: 39 minutes 20 seconds, 3.82% average
That's 42% lower CPU usage for pypy.
This is multiplicative with the jtoomim_performance benefit, so using the jtoomim_performance branch with pypy has about 65% lower CPU usage than using the head branch with regular python.
Test conditions: Core i7 4790k processor, 12 peers, 76 to 80 TH/s load (SP30s). Both nodes were run on the same server at the same time, with a local BitcoinXT bitcoind.
That's a big improvement - impressive Tying it out now. Yes it is, I just can't figure why he's pushing XT? Anybody with the technical skills to re-code p2pool with a 42% improvement in speed should have the good sense to stay away from this XT nonsense.
|
|
|
|
Richy_T
Legendary
Offline
Activity: 2562
Merit: 2264
1RichyTrEwPYjZSeAYxeiFBNnKC9UjC5k
|
|
October 06, 2015, 07:29:05 PM |
|
Thanks. I don't think those instructions are directly relevant but they point in the right direction. Looks like I'll need to cook up my own cramfs
|
1RichyTrEwPYjZSeAYxeiFBNnKC9UjC5k
|
|
|
jtoomim
|
|
October 07, 2015, 10:36:47 AM |
|
Yes it is, I just can't figure why he's pushing XT? Anybody with the technical skills to re-code p2pool with a 42% improvement in speed should have the good sense to stay away from this XT nonsense.
This isn't really the right thread or forum for that discussion. I'll give a brief bullet-point answer. If you want to have a conversation about this, we should probably move it to a different forum or a separate thread so as not to derail this one. 1. Because I think that large blocks are technically feasible, and will not make Bitcoin insecure, inaccessible, or excessively centralized. 2. Because I think that Bitcoin can handle much larger blocks (up to about 32 MB) and transaction volumes than we currently have without any code changes (except the block size limit) or exotic hardware. 3. Because I think that there are enough possible optimizations in both code and semi-exotic hardware to bring us safely up to 8 GB blocksizes in 5 years if we try hard, and in 20 years if we are lazy. 3. Because I think that BIP101 is better than BIP100 or any other proposal out there due to its simplicity, predictability, and because I think its growth rate is reasonable and unlikely to require later changes. BIP100 would be a distant second place in my opinion if it were implemented. 4. Because I think that large blocks are the most feasible method of paying miners enough in fees once the block subsidy drops below 12.5 BTC. 5. Because I do not think that consensus among developers is a reasonable method for making technical decisions for large or controversial projects. 6. Because I do not think that consensus among any group is a reasonable method for making political decisions, much less consensus among a subset of developers. 7. Because I like the way Gavin thinks about technical and political problems. 8. Because, although I think his conduct could be more gracious and polite, I think that Mike Hearn has been right about a lot of technical and political issues lately. You can also see some of my comments on the issue here: https://www.reddit.com/user/jtoomim/
|
Hosting bitcoin miners for $65 to $80/kW/month on clean, cheap hydro power. http://Toom.im
|
|
|
yslyung
Legendary
Offline
Activity: 1500
Merit: 1002
Mine Mine Mine
|
|
October 07, 2015, 11:52:35 AM |
|
The main branch only includes the BIP101 change and the payout_address web interface change. The performance modification has not been merged into the main branch yet. If you want to test out the performance improvements, you have to use the jtoomim_performance branch, which forrestv is currently testing out. https://github.com/p2pool/p2pool/tree/jtoomim_performancegit pull git checkout jtoomim_performance Edit/note: if you use pypy instead of regular python to run p2pool, the performance benefit is likely to be greater than 40%, since most of the rest of the slowdowns that I've seen would be amenable to pypy acceleration, but the problem I fixed was not. I may do some benchmarks to see if this is true in a while. just downloaded, running now to test it out ! thx for the upgrade running smoothly. can we have more blocks ?
|
|
|
|
jtoomim
|
|
October 07, 2015, 01:24:01 PM |
|
just downloaded, running now to test it out ! thx for the upgrade running smoothly. can we have more blocks ? So greedy. We just gave you one seven hours ago. (It was from one of our nodes running jtoomim_performance on pypy on BitcoinXT, in case you're curious.)
|
Hosting bitcoin miners for $65 to $80/kW/month on clean, cheap hydro power. http://Toom.im
|
|
|
notbatman
Legendary
Offline
Activity: 2212
Merit: 1038
|
|
October 07, 2015, 02:51:52 PM |
|
Yes it is, I just can't figure why he's pushing XT? Anybody with the technical skills to re-code p2pool with a 42% improvement in speed should have the good sense to stay away from this XT nonsense.
This isn't really the right thread or forum for that discussion. I'll give a brief bullet-point answer. If you want to have a conversation about this, we should probably move it to a different forum or a separate thread so as not to derail this one. 1. Because I think that large blocks are technically feasible, and will not make Bitcoin insecure, inaccessible, or excessively centralized. 2. Because I think that Bitcoin can handle much larger blocks (up to about 32 MB) and transaction volumes than we currently have without any code changes (except the block size limit) or exotic hardware. 3. Because I think that there are enough possible optimizations in both code and semi-exotic hardware to bring us safely up to 8 GB blocksizes in 5 years if we try hard, and in 20 years if we are lazy. 3. Because I think that BIP101 is better than BIP100 or any other proposal out there due to its simplicity, predictability, and because I think its growth rate is reasonable and unlikely to require later changes. BIP100 would be a distant second place in my opinion if it were implemented. 4. Because I think that large blocks are the most feasible method of paying miners enough in fees once the block subsidy drops below 12.5 BTC. 5. Because I do not think that consensus among developers is a reasonable method for making technical decisions for large or controversial projects. 6. Because I do not think that consensus among any group is a reasonable method for making political decisions, much less consensus among a subset of developers. 7. Because I like the way Gavin thinks about technical and political problems. 8. Because, although I think his conduct could be more gracious and polite, I think that Mike Hearn has been right about a lot of technical and political issues lately. You can also see some of my comments on the issue here: https://www.reddit.com/user/jtoomim/Ahh I see you're pushing XT for political reasons; backing MitGavin and GoogleHern.
|
|
|
|
Richy_T
Legendary
Offline
Activity: 2562
Merit: 2264
1RichyTrEwPYjZSeAYxeiFBNnKC9UjC5k
|
|
October 07, 2015, 05:57:03 PM |
|
There was zero reason for him to inject the block size discussion into this thread, particularly the way he chose to do so. I have a side but I see no need to denigrate those who have a different opinion.
|
1RichyTrEwPYjZSeAYxeiFBNnKC9UjC5k
|
|
|
btcscot
Newbie
Offline
Activity: 30
Merit: 0
|
|
October 08, 2015, 08:16:38 AM |
|
well im giving up on trying this vps and i have just booted my server 2x E5440 @ 2.83ghz 12gb of ram and 146gb hdd
going to install ubuntu and duck dns so my ip stays connected to the domain i have a good firewall in the house so im not worried about folk trying to hack in for my £0.01 in btc hahaha
hopefully will be up before i go away on holiday if not atleast i will have the ssh port and the 9332 9333 8333 ports open
|
|
|
|
yslyung
Legendary
Offline
Activity: 1500
Merit: 1002
Mine Mine Mine
|
|
October 08, 2015, 09:35:07 AM |
|
just downloaded, running now to test it out ! thx for the upgrade running smoothly. can we have more blocks ? So greedy. We just gave you one seven hours ago. (It was from one of our nodes running jtoomim_performance on pypy on BitcoinXT, in case you're curious.) no ! me NO greedy, want to harvest for the dry season. dry spell for over 4 days is no good
|
|
|
|
Richy_T
Legendary
Offline
Activity: 2562
Merit: 2264
1RichyTrEwPYjZSeAYxeiFBNnKC9UjC5k
|
|
October 08, 2015, 09:03:25 PM |
|
From looking through the update code on the Antminer, it looks like it's a fairly straightforward process...
1)Obtain latest firmware 2)Unpack firmware 3)Uncompress RamFS image 4)Substitute custom cgminer for original cgminer 5)Recompress RamFS image 6)zip only new image into tgz file 7)Use upgrade page on the web interface to update 8)Cry over bricked antminer (?)
If anyone is interested in this and it works, I could put it for download somewhere.
Any other changes that might be worth slipping in?
|
1RichyTrEwPYjZSeAYxeiFBNnKC9UjC5k
|
|
|
jonnybravo0311
Legendary
Offline
Activity: 1344
Merit: 1024
Mine at Jonny's Pool
|
|
October 09, 2015, 01:14:45 PM |
|
For which Antminer? The S7? Also, step 8 seems a bit off... you sure you want to tell people to brick their Antminers? Interesting random fact: 9 of the last 20 p2pool blocks are BIP101.
|
Jonny's Pool - Mine with us and help us grow! Support a pool that supports Bitcoin, not a hardware manufacturer's pockets! No SPV cheats. No empty blocks.
|
|
|
Richy_T
Legendary
Offline
Activity: 2562
Merit: 2264
1RichyTrEwPYjZSeAYxeiFBNnKC9UjC5k
|
|
October 09, 2015, 02:46:33 PM |
|
For which Antminer? The S7? Also, step 8 seems a bit off... you sure you want to tell people to brick their Antminers? Interesting random fact: 9 of the last 20 p2pool blocks are BIP101. No, this is for the S5. 4.9.0 cgminer is working fine but I want to make it survive a reboot.
|
1RichyTrEwPYjZSeAYxeiFBNnKC9UjC5k
|
|
|
jtoomim
|
|
October 09, 2015, 07:34:50 PM |
|
From looking through the update code on the Antminer, it looks like it's a fairly straightforward process...
1)Obtain latest firmware 2)Unpack firmware 3)Uncompress RamFS image 4)Substitute custom cgminer for original cgminer 5)Recompress RamFS image 6)zip only new image into tgz file 7)Use upgrade page on the web interface to update 8)Cry over bricked antminer (?)
If anyone is interested in this and it works, I could put it for download somewhere.
Any other changes that might be worth slipping in?
I think you can also just use the smit1237 firmware. If I remember correctly, if you want to use a custom cgminer with smit1237's FW, you just stick the new cgminer into /config (which is non-volatile) and that becomes the new default.
|
Hosting bitcoin miners for $65 to $80/kW/month on clean, cheap hydro power. http://Toom.im
|
|
|
Richy_T
Legendary
Offline
Activity: 2562
Merit: 2264
1RichyTrEwPYjZSeAYxeiFBNnKC9UjC5k
|
|
October 09, 2015, 08:17:34 PM |
|
I think you can also just use the smit1237 firmware. If I remember correctly, if you want to use a custom cgminer with smit1237's FW, you just stick the new cgminer into /config (which is non-volatile) and that becomes the new default.
If that's the one in the link, that was for the S4 (though they may be compatible?). I'll look into it a bit more. I may want to add some extra stuff in in any case.
|
1RichyTrEwPYjZSeAYxeiFBNnKC9UjC5k
|
|
|
|
jtoomim
|
|
October 11, 2015, 03:55:47 PM |
|
I have been seeing a lot of performance problems with pypy recently. One of my nodes exceeded 4 GB of RAM usage by pypy/p2pool alone. I recommend using regular python, as it seems to give more reliable performance.
|
Hosting bitcoin miners for $65 to $80/kW/month on clean, cheap hydro power. http://Toom.im
|
|
|
p3yot33at3r
|
|
October 11, 2015, 04:39:22 PM |
|
I have been seeing a lot of performance problems with pypy recently. One of my nodes exceeded 4 GB of RAM usage by pypy/p2pool alone. I recommend using regular python, as it seems to give more reliable performance.
I was wondering myself, but thought it was just me.....
|
|
|
|
idonothave
|
|
October 12, 2015, 06:56:20 AM |
|
I have been seeing a lot of performance problems with pypy recently. One of my nodes exceeded 4 GB of RAM usage by pypy/p2pool alone. I recommend using regular python, as it seems to give more reliable performance.
I was wondering myself, but thought it was just me..... I have also noticed that and temporarily got back to python but in my opinion the problem was in combination pypy / high GBT latency so I have added to my bitcoin.conf lines blockmaxsize=250000 # default: 250000 blockprioritysize=27000 # default: 27000 mintxfee=0.0002 # default: 0.0001 minrelaytxfee=0.0002 # default: 0.0001
then latency went down around 0.6 and I switched back to pypy without any problems again but I think it is not permanent solution and if there is anybody experienced with bitcoin.conf tuning I would welcome to have one recommended template of bitcoin.conf usable here
|
|
|
|
jtoomim
|
|
October 12, 2015, 09:55:55 AM |
|
I suspect the performance issue is due to the p2pool's known_txns_var (the equivalent of the p2pool mempool) getting large over time, and pypy taking a lot more space to store this for some reason. In any case, the problems that I've seen appear to only manifest themselves after running p2pool for a while, like a few days. Restarting p2pool clears the caches and reduces memory usage, improving performance.
When a node creates a share that includes large low-fee transactions, it forces all other p2pool nodes to download and store those transactions. Setting minrelaytxfee to something reasonably high like 0.0005 and blockprioritysize to something low like 0 may help performance for all p2pool nodes, because your node would force fewer low-fee spam txns onto those nodes.
|
Hosting bitcoin miners for $65 to $80/kW/month on clean, cheap hydro power. http://Toom.im
|
|
|
idonothave
|
|
October 12, 2015, 10:27:58 AM |
|
I suspect the performance issue is due to the p2pool's known_txns_var (the equivalent of the p2pool mempool) getting large over time, and pypy taking a lot more space to store this for some reason. In any case, the problems that I've seen appear to only manifest themselves after running p2pool for a while, like a few days. Restarting p2pool clears the caches and reduces memory usage, improving performance.
When a node creates a share that includes large low-fee transactions, it forces all other p2pool nodes to download and store those transactions. Setting minrelaytxfee to something reasonably high like 0.0005 and blockprioritysize to something low like 0 may help performance for all p2pool nodes, because your node would force fewer low-fee spam txns onto those nodes.
someone should hold leadership with this to say, guys, to put this and this is the best way for now... and even why not to vote for major changes and p2pool bitcoin.conf best template form you as a provider of quite big part of p2pool hashrate would be that
|
|
|
|
|