[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[p3yot33at3r]

was exactly 3 months ago

Sorry for creating confusion with my slightly unclear post at forum.namecoin.info. Actually it was 3 months ago when the miners had the last chance to mine a NMC block using one of the old Namecoin clients. However, there was a p2pool-mined NMC block six weeks ago, and another one five weeks ago. Both using the new Namecoin Core client.

2015-08-02 23:21 UTC, NMC block 242312, mined by ghash.io. This was the last time an old NMC client was able to sucessfully mine a non-orphaned block.
2015-10-13 21:17 UTC, NMC block 253122, mined by p2pool miner 1GqdAgC1zmzLFMg3XyYctSECWUjkbT7Nv9 using the new Namecoin Core client
2015-10-19 05:10 UTC, NMC block 253874, mined by p2pool miner 14N4xNSDtNYatfCgHhyRaAhdZtaczUBnAa using the new Namecoin Core client

Ah! - that's more like it. Thanks for those stats & putting my concerns at rest cassini - phew!  

Is this due to lack of hashpower in the pool vs others (ie: everyone else is beating us to it) or simply really really bad luck..

I'd say it's a combination of all of those. Long blocks happen on all pools though, large & small, luck is luck 

[1714078609]

1714078609

[lightfoot]

Is this due to lack of hashpower in the pool vs others (ie: everyone else is beating us to it) or simply really really bad luck..

Simpler than that: Someone just pissed off one of the Elder Gods.

[wariner]

hi,

to promote the use of p2pool, it would be possible to make it also "shared" the merged mining?

and permit those who connect to a node add own address? For example add own address nmc instead of the password..

Thank you for reply and sorry for my english..

[idonothave]

hi,

to promote the use of p2pool, it would be possible to make it also "shared" the merged mining?

and permit those who connect to a node add own address? For example add own address nmc instead of the password..

Thank you for reply and sorry for my english..

Personaly I do not think it is possible. It would be possible only under condition if each node was merge mining and they are not and they will not. I find it great to may merge mine but the principles of decentralised mining of bitcoin goes counter your idea. imho

[M8BWNNRFMNdak68c]

so i am mining with high hashrate for 5 days now.. but obviously all my hashpower is being destroyed by "bad luck"..
we are at 16% luck now..
( luck has nothing to do with other pools by the way, just with the difficulty.. orphaned blocks would indicate too much pool-competition )

16% is really bad.. so i am always frightened that something is wrong and we are not noticing it..
and my biggest fear is that i made it broken..

i am merge mining some forgotten coins nobody else is mining ( obviously - i own 90% of the network hash power there, diff quite low. )
so everytime i find a share, i also find 2-4 blocks simultaneously ( each on a different network )

so please tell me that this is no problem for p2pool and i did not break the network.
( it might be confused that one share = 4 blocks on different networks? could that lead to a broken sharechain in p2pool? )

and no i will not tell in public you which networks i use..
( they bring me a 2% bonus on my income.. or with 0% luck... i only loose 98% instead of 100%..  )

concering the merge "bonus": every node operator should definitely set the fee to zero and use the merged coins as his income.. so everybody wins.. ( a negative fee would also be interesting.. but would be quite complicated, as you would have to integrate transactions ("old coins") in the new block )

[nicklello]

so i am mining with high hashrate for 5 days now.. but obviously all my hashpower is being destroyed by "bad luck"..
we are at 16% luck now..
( luck has nothing to do with other pools by the way, just with the difficulty.. orphaned blocks would indicate too much pool-competition )

16% is really bad.. so i am always frightened that something is wrong and we are not noticing it..
and my biggest fear is that i made it broken..

That's been my paranoia for a few days...

[jonnybravo0311]

so i am mining with high hashrate for 5 days now.. but obviously all my hashpower is being destroyed by "bad luck"..
we are at 16% luck now..
( luck has nothing to do with other pools by the way, just with the difficulty.. orphaned blocks would indicate too much pool-competition )

16% is really bad.. so i am always frightened that something is wrong and we are not noticing it..
and my biggest fear is that i made it broken..

i am merge mining some forgotten coins nobody else is mining ( obviously - i own 90% of the network hash power there, diff quite low. )
so everytime i find a share, i also find 2-4 blocks simultaneously ( each on a different network )

so please tell me that this is no problem for p2pool and i did not break the network.
( it might be confused that one share = 4 blocks on different networks? could that lead to a broken sharechain in p2pool? )

and no i will not tell in public you which networks i use..
( they bring me a 2% bonus on my income.. or with 0% luck... i only loose 98% instead of 100%..  )

concering the merge "bonus": every node operator should definitely set the fee to zero and use the merged coins as his income.. so everybody wins.. ( a negative fee would also be interesting.. but would be quite complicated, as you would have to integrate transactions ("old coins") in the new block )

First, I like your optimism... "I only lose 98% instead of 100%". 

Second, no, you didn't break the network.  I have, in the past, merge mined every coin that could be.  there were plenty of times that I would find blocks of multiple coins with a single share.  Think about it like this: every time you find a block of BTC, you have by default also found a block of every coin you are merge mining.

so i am mining with high hashrate for 5 days now.. but obviously all my hashpower is being destroyed by "bad luck"..
we are at 16% luck now..
( luck has nothing to do with other pools by the way, just with the difficulty.. orphaned blocks would indicate too much pool-competition )

16% is really bad.. so i am always frightened that something is wrong and we are not noticing it..
and my biggest fear is that i made it broken..

That's been my paranoia for a few days...

7 day luck is 0%.  30 day luck is 78.77%.  In any case, our current round is painfully long, but certainly not out of the realm of statistical possibility.  Let's assume we've had 1.5PH/s average every day.  With that hash rate, we'd expect to find a block every 2.06 days.  It's almost 10 days.  If we round things off a bit, we can see this is a 500% block.  Painful... but they happen.

In other words, there is nothing wrong here.

[windpath]

Whew, that was getting ugly..... 

[e46btc]

Whew, that was getting ugly..... 

Still nodody posted here?  But we've got block some hours ago. Congratulations
https://blockchain.info/block/0000000000000000118a4186eb63e9eb750b87bac04d2b905e3b188c343cdf97

[p3yot33at3r]

Whew, that was getting ugly..... 

Yeah - let's hope it's a while before we get one of those again. Good to see that the hash rate & miners kept at it though without bailing when luck sucks 

[windpath]

BTW if someone wants to receive instant WhatsApp notifications about new blocks - please send you phone number in P.M.

https://i.imgur.com/qO3l6dB.png

http://forum.bits.media/uploads/monthly_11_2015/post-25987-0-68142200-1446512918.jpg

For present moment I can add new phone numbers myself only, still no subscription interface to this service (TBD).
To unsubscribe you may also send me another message or just block incoming number in your WhatsApp application.

One feature - you will get 24x7 notifications immediately as block found. Still no scheduled messages delivery.  So this may disturb your sleep

I use IFTTT (If This Then That) to notify of new payouts.

I shared the recipe if anyone wants to use it: https://ifttt.com/recipes/338858-if-bitcoin-received-at-address-then-send-me-a-text-message

Basically it monitors the chain.com API for any incoming tx's to your mining address and sends you a SMS text message when coins are received...

[p3yot33at3r]

I use IFTTT (If This Then That) to notify of new payouts.

I shared the recipe if anyone wants to use it: https://ifttt.com/recipes/338858-if-bitcoin-received-at-address-then-send-me-a-text-message

Basically it monitors the chain.com API for any incoming tx's to your mining address and sends you a SMS text message when coins are received...

This is cool - is there a charge for the sms?

[windpath]

I use IFTTT (If This Then That) to notify of new payouts.

I shared the recipe if anyone wants to use it: https://ifttt.com/recipes/338858-if-bitcoin-received-at-address-then-send-me-a-text-message

Basically it monitors the chain.com API for any incoming tx's to your mining address and sends you a SMS text message when coins are received...

This is cool - is there a charge for the sms?

Not from IFTTT, but perhaps from your carrier. If you pay to receive regular txt messages then I imagine there would be the same fee. I have unlimited texting included with my cell plan so no charge for me.

[p3yot33at3r]

I use IFTTT (If This Then That) to notify of new payouts.

I shared the recipe if anyone wants to use it: https://ifttt.com/recipes/338858-if-bitcoin-received-at-address-then-send-me-a-text-message

Basically it monitors the chain.com API for any incoming tx's to your mining address and sends you a SMS text message when coins are received...

This is cool - is there a charge for the sms?

Not from IFTTT, but perhaps from your carrier. If you pay to receive regular txt messages then I imagine there would be the same fee. I have unlimited texting included with my cell plan so no charge for me.

OK, I think I get 500 free sms p/m so I should be fine cos I hardly ever use it. Do you get the sms when the payment is sent to the address - or when it arrives in the wallet? I ask because my receiving wallet is not on 24/7, so it would be handy if the sms was sent when the payment is sent.....

[windpath]

OK, I think I get 500 free sms p/m so I should be fine cos I hardly ever use it. Do you get the sms when the payment is sent to the address - or when it arrives in the wallet? I ask because my receiving wallet is not on 24/7, so it would be handy if the sms was sent when the payment is sent.....

IFTTT uses the chain.com PI to monitor the blockchain, does not matter how you interact with the address. I think it sends on 0 confirmations, but not 100% on that.

Would love to find 500 blocks in a month and run up your phone bill

[nicklello]

Hoping someone else has experienced this and can advise me...

My setup consists of 16Gb memory, 8 cores...

I'm finding that p2pool is reporting loss of connection to bitcoind (actually bitcoinxtd) at least once a day.

CPU is at least 80% idle all day

Memory utilisation is high --- bitcoin is using ~12 Gb and p2pool is using ~3Gb (I was running under pypy but switched back to python to reduce memory usage).

netstat -an shows *hundreds* of abandoned/idle connections in CLOSE_WAIT from the p2pool process to the bitcoin RPC port.

I've tried everything I can think of, I've turned the TCP timeouts down from 2hr to 5min but have only shifted the problem from TIME_WAIT state to CLOSE_WAIT state connections.

Has anyone else experienced this situation; and what (if anything) can be done to get around it ?

[jtoomim]

Memory utilisation is high --- bitcoin is using ~12 Gb and p2pool is using ~3Gb (I was running under pypy but switched back to python to reduce memory usage).
Has anyone else experienced this situation; and what (if anything) can be done to get around it ?

First: in ~/.bitcoin/bitcoin.conf, add

Code:

minrelaytxfee=0.00005

That will reject any transactions that don't include at least 0.05 mBTC/kb in fees. That should drop your mempool size from about 500 MB down to about 5 MB. This is necessary due to about 1 GB of spam that was sent about a month ago, plus the fact that 1 MB blocks just isn't enough any longer.

Second: before you start up bitcoind, run

Code:

export MALLOC_ARENA_MAX=1

in the same terminal (or bash session) that you use to start bitcoind. This is a workaround for an issue described here:

https://github.com/bitcoin/bitcoin/issues/6876

BitcoinXT will probably be switching over to jemalloc soon. Instead of the MALLOC_ARENA_MAX=1 fix, you can also try using jemalloc if you prefer.

Code:

sudo apt-get install libjemalloc libjemalloc-dev

Then, when you want to run bitcoind,

Code:

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so ./bitcoind -daemon

jemalloc does a better job of deallocating memory that's no longer in use than glibc, at least without the MALLOC_ARENA_MAX=1 fix:

http://toom.im/files/bitcoinxt_jemalloc_vs_glibc_16_threads.png

I haven't yet directly compared jemalloc to glibc with MALLOC_ARENA_MAX=1, but the testing I have done suggests that jemalloc is qualitatively better.

(Btw, folks, I'm working on some enhancements to BitcoinXT's getblocktemplate. I think I'll be able to get a 5x reduction in getblocktemplate latency. Stay tuned. I think jtimon is working on something similar for Core too. Edit: oh, and it will likely improve memory usage too, at least as it currently stands.)

[Polyatomic]

# I grabbed latest from git and built that with tests

Code:

pushd jemalloc-4.0.4
# --enable-profto to build extra tests                                                                                                                                                                                                 
EXTRA_CFLAGS="-march=native" \
./configure --enable-prof
# add extra jobs if u like eg. -j4                                                                                                                                                                                                  
make
# run the tests                                                                                                                                                                                                                     
make check
# fake the install directory                                                                                                                                                                                                        
make install_include install_lib DESTDIR=/tmp/jemalloc-4.0.4

popd
# rm the static archives
find /tmp/jemalloc-4.0.4 -name "*.a" -exec rm -vf {} \;
rm -rf jemalloc-4.0.4

build log with tests here: http://dpaste.com/1PJNZV7 http://dpaste.com/31GA0PG

Ran the bitcoind with

Code:

LD_PRELOAD=/tmp/jemalloc-package-4.0.4/usr/local/lib/libjemalloc.so ./bitcoind -daemon

started the p2pool and testing now. Cheers for sharing that man!.

Ed : Wed Nov  4 21:17:15 ACDT 2015
err, I may of balls this up.

Code:

milton@milton:~/temp/bitcoin/src$ cat /proc/$(pidof bitcoind)/maps | grep jemalloc
7f7097b63000-7f7097bac000 r-xp 00000000 00:19 6153736                    /tmp/jemalloc-package-4.0.4/usr/local/lib/libjemalloc.so.2 (deleted)
7f7097bac000-7f7097dab000 ---p 00049000 00:19 6153736                    /tmp/jemalloc-package-4.0.4/usr/local/lib/libjemalloc.so.2 (deleted)
7f7097dab000-7f7097dae000 rw-p 00048000 00:19 6153736                    /tmp/jemalloc-package-4.0.4/usr/local/lib/libjemalloc.so.2 (deleted)
milton@milton:~/temp/bitcoin/src$ cat /proc/$(pidof bitcoind)/smaps | grep jemalloc
7f7097b63000-7f7097bac000 r-xp 00000000 00:19 6153736                    /tmp/jemalloc-package-4.0.4/usr/local/lib/libjemalloc.so.2 (deleted)
7f7097bac000-7f7097dab000 ---p 00049000 00:19 6153736                    /tmp/jemalloc-package-4.0.4/usr/local/lib/libjemalloc.so.2 (deleted)
7f7097dab000-7f7097dae000 rw-p 00048000 00:19 6153736                    /tmp/jemalloc-package-4.0.4/usr/local/lib/libjemalloc.so.2 (deleted)

Back to the docs now ...

Ed : Thu Nov  5 21:16:33 ACDT 2015

Code:

milton@milton:~/temp/bitcoin/src$ cat /proc/$(pidof bitcoind)/smaps | grep jemalloc
7fa78de8f000-7fa78def3000 r-xp 00000000 00:19 6550911                    /tmp/jemalloc-4.0.4/usr/local/lib/libjemalloc.so.2
7fa78def3000-7fa78e0f2000 ---p 00064000 00:19 6550911                    /tmp/jemalloc-4.0.4/usr/local/lib/libjemalloc.so.2
7fa78e0f2000-7fa78e0f5000 rw-p 00063000 00:19 6550911                    /tmp/jemalloc-4.0.4/usr/local/lib/libjemalloc.so.2
milton@milton:~/temp/bitcoin/src$ cat /proc/$(pidof bitcoind)/maps | grep jemalloc
7fa78de8f000-7fa78def3000 r-xp 00000000 00:19 6550911                    /tmp/jemalloc-4.0.4/usr/local/lib/libjemalloc.so.2
7fa78def3000-7fa78e0f2000 ---p 00064000 00:19 6550911                    /tmp/jemalloc-4.0.4/usr/local/lib/libjemalloc.so.2
7fa78e0f2000-7fa78e0f5000 rw-p 00063000 00:19 6550911                    /tmp/jemalloc-4.0.4/usr/local/lib/libjemalloc.so.2

[nicklello]

Firstly, thankyou very much for the assist....

First: in ~/.bitcoin/bitcoin.conf, add

Code:

minrelaytxfee=0.00005

That will reject any transactions that don't include at least 0.05 mBTC/kb in fees. That should drop your mempool size from about 500 MB down to about 5 MB. This is necessary due to about 1 GB of spam that was sent about a month ago, plus the fact that 1 MB blocks just isn't enough any longer.

I had minrelaytxfee set to 0.00001 ; this probably made me wide open to the spam transactions.

Code:

export MALLOC_ARENA_MAX=1

in the same terminal (or bash session) that you use to start bitcoind. This is a workaround for an issue described here:

BitcoinXT will probably be switching over to jemalloc soon. Instead of the MALLOC_ARENA_MAX=1 fix, you can also try using jemalloc if you prefer.

Code:

sudo apt-get install libjemalloc libjemalloc-dev

Then, when you want to run bitcoind,

Code:

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so ./bitcoind -daemon

I had already employed MALLOC_ARENA_MAX which didn't seem to help much (again, I think the spam hit me) ---  however I've switched to preloading libjemalloc WITHOUT the MALLOC_ARENA_MAX setting.

Early indications are that these 2 changes are drastically reducing my memory usage.

Am looking forward to the getblocktemplate enhancements...

[Meuh6879]

Memory utilisation is high --- bitcoin is using ~12 Gb and p2pool is using ~3Gb (I was running under pypy but switched back to python to reduce memory usage).
Has anyone else experienced this situation; and what (if anything) can be done to get around it ?

First: in ~/.bitcoin/bitcoin.conf, add

Code:

minrelaytxfee=0.00005

oh come on ... we don't have a mempool problem actually.

Code:

maxmempooltx=1200
minrelaytxfee=0.00000001

I use this (XT only for the first parameter).