bitHopper: Python Pool Hopper Proxy

[Endeavour79]

move bitminers union to the unhoppable part of your lists

Need an update anyway..

[bmunion]
name: BitMinersUnion.org
mine_address: pool.bitminersunion.org:8341
api_address: http://64.244.102.89/stats.php
api_method: re
api_key: Shares this round:([ 0-9]+)<br/>
api_strip: ' '
url: http://www.bitminersunion.org/

[c00w]

@whoever wanted an lp_penalty option:
I added it. It is lp_penalty : seconds.

[r2edu]

^ you´re quick man!! hehe..

what is the beneffit of that, or what value do you recommend?

[c00w]

You can put a penalty on deepbit if you've done the analysis for your location and determined it is getting too many blocks. Or any other site. Especially if there LP is super slow or delayed.

[deepceleron]

@whoever wanted an lp_penalty option:
I added it. It is lp_penalty : seconds.

Great! This feature adds X seconds to the LP receive time of the specified pool for figuring out who solved it. That means if you analyze the console log, and find one pool that is consistently faster in sending you it's LP block announce than others (especially if a pool responds to you before the one that actually found the block - causing the proxy to designate the wrong pool), then you can 'equalize' the time LPs are received by adding a delay to 'too fast' pools.

Block 140749 found by pool.bitp.it:

[19:28:09] received lp from: bitp
[19:28:09] New Block: 5ea4b4d35340f0a60c238a721e3b0bd72453873fe6a7cec6000005f700
000000
[19:28:09] Block Owner bitp
[19:28:09] LP Call pool.bitp.it:8334/longpoll
[19:28:10] received lp from: mmf
[19:28:10] LP Call mining.mainframe.nl:8343/LP
[19:28:10] received lp from: deepbit
[19:28:10] LP Call pit.deepbit.net:8332/listenChannel
[19:28:10] received lp from: mineco
[19:28:10] LP Call mineco.in:3000/LP
[19:28:10] RPC request [getwork] submitted to polmine
[19:28:11] received lp from: bclc
[19:28:11] LP Call bitcoins.lc:8080/LP
[19:28:12] received lp from: eclipsemc
[19:28:12] LP Call pacrim.eclipsemc.com:8337/LP
[19:28:12] received lp from: arsbitcoin
[19:28:12] LP Call arsbitcoin.com:8344/LP
[19:28:14] received lp from: eligius
[19:28:14] LP Call su.mining.eligius.st:8337/LP

Although this is not the best example (I actually had one block scroll by where mineco was the first to announce the block, beating the actual finding pool), we can see that pools take different amounts of time to respond. Bitp.it was almost beat in announcing their own block by mainframe. I can improve the results by tweaking like this (if I was only looking at the above results and not evaluating pool delays over many blocks):

[mmf]
lp_penalty: 4

[deepbit]
lp_penalty: 4

[mineco]
lp_penalty: 4

[bclc]
lp_penalty: 3

[eclipsemc]
lp_penalty: 2

[arsbitcoin]
lp_penalty: 2

[eligius]
lp_penalty: 0

[bitp]
lp_penalty: 3? (would need to evaluate blocks they didn't solve)

I will test this out and see if it can completely fix false pool block designation. Of course what would be wishful thinking would be if bithopper could learn for itself the average delays it sees from each pool.

[organofcorti]

@whoever wanted an lp_penalty option:
I added it. It is lp_penalty : seconds.

Great! This feature adds X seconds to the LP receive time of the specified pool for figuring out who solved it. That means if you analyze the console log, and find one pool that is consistently faster in sending you it's LP block announce than others (especially if a pool responds to you before the one that actually found the block - causing the proxy to designate the wrong pool), then you can 'equalize' the time LPs are received by adding a delay to 'too fast' pools.

Block 140749 found by pool.bitp.it:

[19:28:09] received lp from: bitp
[19:28:09] New Block: 5ea4b4d35340f0a60c238a721e3b0bd72453873fe6a7cec6000005f700
000000
[19:28:09] Block Owner bitp
[19:28:09] LP Call pool.bitp.it:8334/longpoll
[19:28:10] received lp from: mmf
[19:28:10] LP Call mining.mainframe.nl:8343/LP
[19:28:10] received lp from: deepbit
[19:28:10] LP Call pit.deepbit.net:8332/listenChannel
[19:28:10] received lp from: mineco
[19:28:10] LP Call mineco.in:3000/LP
[19:28:10] RPC request [getwork] submitted to polmine
[19:28:11] received lp from: bclc
[19:28:11] LP Call bitcoins.lc:8080/LP
[19:28:12] received lp from: eclipsemc
[19:28:12] LP Call pacrim.eclipsemc.com:8337/LP
[19:28:12] received lp from: arsbitcoin
[19:28:12] LP Call arsbitcoin.com:8344/LP
[19:28:14] received lp from: eligius
[19:28:14] LP Call su.mining.eligius.st:8337/LP

Although this is not the best example (I actually had one block scroll by where mineco was the first to announce the block, beating the actual finding pool), we can see that pools take different amounts of time to respond. Bitp.it was almost beat in announcing their own block by mainframe. I can improve the results by tweaking like this (if I was only looking at the above results and not evaluating pool delays over many blocks):

[mmf]
lp_penalty: 4

[deepbit]
lp_penalty: 4

[mineco]
lp_penalty: 4

[bclc]
lp_penalty: 3

[eclipsemc]
lp_penalty: 2

[arsbitcoin]
lp_penalty: 2

[eligius]
lp_penalty: 0

[bitp]
lp_penalty: 3? (would need to evaluate blocks they didn't solve)

I will test this out and see if it can completely fix false pool block designation.

Nice work! When you do get it done, could you paste your raw before and after data somewhere (eg google docs or pastebin)? I'd like to get an idea of mean time differences for the pools to your part of the world. Knowing your part of the world (eg Europe, Americas, Australasia - you don't have to be exact) might be handy since if a number of us do this and people from similar parts of the world have similar differences in LP announces then these penalties coud be built into bitHopper as a default,, and the penalties populated automatically.

Thanks for taking the time. I hope Sukrim is watching - he was keen to do something of the sort.

[deepceleron]

Nice work! When you do get it done, could you paste your raw before and after data somewhere (eg google docs or pastebin)? I'd like to get an idea of mean time differences for the pools to your part of the world. Knowing your part of the world (eg Europe, Americas, Australasia - you don't have to be exact) might be handy since if a number of us do this and people from similar parts of the world have similar differences in LP announces then these penalties coud be built into bitHopper as a default,, and the penalties populated automatically.

Thanks for taking the time. I hope Sukrim is watching - he was keen to do something of the sort.

This LP "who solved the block" procedure exploits an assumption, that a pool that finds a block will be sending notifications to miners earlier than other pools. This may not always be correct.

One has to understand these delays and what long polling is.

When a pool's bitcoin detects a new block (whether they found it themselves or found out about it over the p2p network), the the job of sending notifications goes to pushpool. Pushpool looks at all the open RPC connections that have long polling enabled, and pushes out a notification of the new block to the miners. Long polling keeps a connection open between the miner and the pool, so the miners get a quicker notification of the new block than if they were to just request new work every 15 seconds or so. This reduces obsolete share submissions ("stale" shares), improving pool efficiency.

Pushpool can't instantaneously notify everyone however, due to network bandwidth. Essentially, it goes through a list of all open connections and sends each the LP push consecutively. Small pools can send all their miners the notification quickly, but depending on the pool implementation and the number of open connections, this can take more than five seconds on busy pools. If you are on the start of the list, you get notification fast; at the end of the list, it can take longer. This means that the pool delay can be random per miner, much more significantly than packet transit times. Profiling your pool connection time may give different results from others, and may give different results even if you restart bithopper on a different day and make new RPC connections to the pool.

[c00w]

Its right about 50% of the time. And people are working to try and increase that.

[Sukrim]

Pushpool can't instantaneously notify everyone however, due to network bandwidth. Essentially, it goes through a list of all open connections and sends each the LP push consecutively. Small pools can send all their miners the notification quickly, but depending on the pool implementation and the number of open connections, this can take more than five seconds on busy pools. If you are on the start of the list, you get notification fast; at the end of the list, it can take longer. This means that the pool delay can be random per miner, much more significantly than packet transit times. Profiling your pool connection time may give different results from others, and may give different results even if you restart bithopper on a different day and make new RPC connections to the pool.

Thanks for the perfect explanation!

This is also the reason why ultimatively I want to have a stripped down Bitcoin client that watches block announcements and where they come from instead of timing longpolls in the longer run.

To get more data on Longpolls we would need to connect bitHoppers together (via IRC or something else) - something that might not appeal to some people. Also I'm not sure if it really can improve the prediction rate significantly if longpolls really differ by 5 seconds on the same pool. There are 2 interesting factors to consider: How often do we miss a block by a LP pool and how often do we think that an LP pool has solved a block when it in reality hasn't?
The first rate is not that critical, the second one can seriously harm business though.

[c00w]

The 2nd one gets caught by our monitoring of API's.

[joulesbeef]

move bitminers union to the unhoppable part of your lists

Need an update anyway..

[bmunion]
name: BitMinersUnion.org
mine_address: pool.bitminersunion.org:8341
api_address: http://64.244.102.89/stats.php
api_method: re
api_key: Shares this round:([ 0-9]+)<br/>
api_strip: ' '
url: http://www.bitminersunion.org/

what about

with built in hard coding protections to discourage pool hopping (not for people who change pools but for those who intentionally hop pools as a way to cheat the other miners of the pool

i saw it was you chating with him.. did you find out more or is the hard coding is that we need regex?

[anty]

Need an update anyway..

[bmunion]
name: BitMinersUnion.org
mine_address: pool.bitminersunion.org:8341
api_address: http://64.244.102.89/stats.php
api_method: re
api_key: Shares this round:([ 0-9]+)<br/>
api_strip: ' '
url: http://www.bitminersunion.org/

I made the RegEx prettier:

Code:

[bmunion]
name: BitMinersUnion.org
mine_address: pool.bitminersunion.org:8341
api_address: http://64.244.102.89/stats.php
api_method: re
api_key: Shares this round:([0-9]+)<br/>
url: http://www.bitminersunion.org/

[deepceleron]

@whoever wanted an lp_penalty option:
I added it. It is lp_penalty : seconds.

The latest version seems to have a big bug, it basically seems seems to immediately change the block owner every time it gets a new block LP instead of evaluating which came first:

edit: fixed, removed log dump

[lueo]

Anyway, I'm in!

Deepbit Team "bitHoppers": https://deepbit.net/teams/4e424d710691723831000002/join

Feel free to hop in!

[MaGNeT]

This made me think:

Good news everyone! 43% is dead!

We no longer have to hop at 0.43*difficulty!

I haven't been around much lately because I spent the last week running and rewriting 'byteHopper', a multiple pool hopping simulator. When when I got results I didn't want, I rewrote it from scratch. Three times.

Although I made it run faster each time the results are the same, and the cumulative distribution functions follow real world bitcoin results. So I'm finally happy with the results it gives.

I'm only posting a summary here and I'll get around to making blog posts on hoppersden to give more detail, but the gist of it is: If there are 3 or more hoppable pools, you can stay on pools as long as you want and still get the same long term reward as if you hopped on 0.43, but without needing backup and without pissing pool operators as much. Keep in mind that this is not a claim about one particular round, but over, say, 100000 rounds (which I used for the graphs below).

Of course in any given round, after total shares=diff, your shares are worth less than one. But over time the shorter rounds make up for it. Even with only 3 Prop and backup, your efficiency will always be about 1.83, unless you hop off *too soon*. With ten other pools you get about 250% over PPS - sound familiar?

tl;dr BOLD CLAIM: As long as you always hop to the pool with the lowest shares - regardless of hashspeed - you don't need to hop to backup at 0.43.

The ran up some quick graphs which show byteHopper results up to hopping at 3*diff for 0 other pools (ie one proportional pools and one PPS), 1 other pools, 3 other pools and 5 other pools, all plus pps when needed. '0 other pools' mean the same as Raulo's example, and gives the same results as his equation.

 This is basically not 'hopping' much at all. PPS only really has positive effect if you hop only one or two other pools.

                 

I tried several schedulers now and I thought about the following (new "--HopLowestSharecount" scheduler):

- mine the 3 pools with lowest sharecount at any moment (where 3 can be changed to a custom value with "--hop_ x_pools=3" parameter)
- forget about the 0.43 threshold
- It won't need back-up pools, it will always hop between the 3 pools with lowest sharecount and with 18 pools, that should work without problems

It's a bit like the OldDefaultScheduler and SliceScheduler, without the threshold.

[simonk83]

Er...interesting....

https://www.rfcpool.com/account

rfcpool has been shut down because the owner is a fucking idiot - that
should sum it up in brief.


If you were a miner, your money is all accounted for, and you will be paid as
soon as the most recent block matures. If you did not have a wallet address
in your account then email chris@rfcpool.com and after slinging your abuse
at me, tell me the username/email on account/wallet combination and you will
get paid.

So what happened? Well in short, running a pool and dealing with
people's money is a metric fucktonne harder than I ever imagined it would
be, and a series of fuckups relating to handling all said money has left me
about 150BTC out of pocket, so I'm pulling the plug before the situation
gets any worse.

Good luck to all other pool ops.

[MaGNeT]

Er...interesting....

https://www.rfcpool.com/account

rfcpool has been shut down because the owner is a fucking idiot - that
should sum it up in brief.


If you were a miner, your money is all accounted for, and you will be paid as
soon as the most recent block matures. If you did not have a wallet address
in your account then email chris@rfcpool.com and after slinging your abuse
at me, tell me the username/email on account/wallet combination and you will
get paid.

So what happened? Well in short, running a pool and dealing with
people's money is a metric fucktonne harder than I ever imagined it would
be, and a series of fuckups relating to handling all said money has left me
about 150BTC out of pocket, so I'm pulling the plug before the situation
gets any worse.

Good luck to all other pool ops.

Hmmm...

role=disable

 

[pbj sammich]

Just saw that RFC pool notice this morning as well.

You can't even get to your account page, and I was hopping them last nite

I know I had BTC there but how much is a guess

oh well, not a huge loss

[deepceleron]

This made me think:

Good news everyone! 43% is dead!

We no longer have to hop at 0.43*difficulty!
...

Of course in any given round, after total shares=diff, your shares are worth less than one. But over time the shorter rounds make up for it. Even with only 3 Prop and backup, your efficiency will always be about 1.83, unless you hop off *too soon*. With ten other pools you get about 250% over PPS - sound familiar?

tl;dr BOLD CLAIM: As long as you always hop to the pool with the lowest shares - regardless of hashspeed - you don't need to hop to backup at 0.43.

I tried several schedulers now and I thought about the following (new "--HopLowestSharecount" scheduler):

- mine the 3 pools with lowest sharecount at any moment (where 3 can be changed to a custom value with "--hop_ x_pools=3" parameter)
- forget about the 0.43 threshold
- It won't need back-up pools, it will always hop between the 3 pools with lowest sharecount and with 18 pools, that should work without problems

It's a bit like the OldDefaultScheduler and SliceScheduler, without the threshold.

That is bold to make bold claims, and be incorrect at them. A share submitted after 43.3% of difficulty has elapsed in a proportional pool has a lower expected return than if that share was submitted to a 0 fee PPS or even-paying pool, or if a switch was made to solo mining. By mining from the start until 100% of difficulty shares, you reduce your expected return from 28% to 22%. Considering the lag in statistics and delays switching into and out of a proportional pool, you should be triggering an earlier exit than that to maximize your expected earnings. By mining more pools, you have a higher chance of one of the pools being in the sweet spot, and can also choose the highest reward pool at a particular time, but that doesn't change the ultimate temporal valuation of a submitted share.

Indeed, you also shouldn't be 'slicing', shares should be submitted to the highest instantaneous reward pool only. I would have to scroll back 100 pages to see who came up with this stinker... The fallacy of reducing variation is why people are paying 7% for PPS (and 20% for credit cards), they can't see or comprehend the long-term.

[MaGNeT]

This made me think:

Good news everyone! 43% is dead!

We no longer have to hop at 0.43*difficulty!
...

Of course in any given round, after total shares=diff, your shares are worth less than one. But over time the shorter rounds make up for it. Even with only 3 Prop and backup, your efficiency will always be about 1.83, unless you hop off *too soon*. With ten other pools you get about 250% over PPS - sound familiar?

tl;dr BOLD CLAIM: As long as you always hop to the pool with the lowest shares - regardless of hashspeed - you don't need to hop to backup at 0.43.

I tried several schedulers now and I thought about the following (new "--HopLowestSharecount" scheduler):

- mine the 3 pools with lowest sharecount at any moment (where 3 can be changed to a custom value with "--hop_ x_pools=3" parameter)
- forget about the 0.43 threshold
- It won't need back-up pools, it will always hop between the 3 pools with lowest sharecount and with 18 pools, that should work without problems

It's a bit like the OldDefaultScheduler and SliceScheduler, without the threshold.

That is bold to make bold claims, and be incorrect at them. A share submitted after 43.3% of difficulty has elapsed in a proportional pool has a lower expected return than if that share was submitted to a 0 fee PPS or even-paying pool, or if a switch was made to solo mining. By mining from the start until 100% of difficulty shares, you reduce your expected return from 28% to 22%. Considering the lag in statistics and delays switching into and out of a proportional pool, you should be triggering an earlier exit than that to maximize your expected earnings. By mining more pools, you have a higher chance of one of the pools being in the sweet spot, and can also choose the highest reward pool at a particular time, but that doesn't change the ultimate temporal valuation of a submitted share.

Indeed, you also shouldn't be 'slicing', shares should be submitted to the highest instantaneous reward pool only. I would have to scroll back 100 pages to see who came up with this stinker... The fallacy of reducing variation is why people are paying 7% for PPS (and 20% for credit cards), they can't see or comprehend the long-term.

I like the discussion

Do we have graphs for both to compare?