Long Poll and Pool Hopping

[dyngnosis]

I became interested in bitcoin mining, mining pools and the bitcoin network last week so please excuse the fresh smell.

After reading the initial Satoshi paper and seeing how the implementation has evolved and differed from the original (and brilliant) design I started to consider the 'dangers' of large bitcoin mining pools.  It seems that if a determined attacker was able to compromise just two machines they could compromise the entire bitcoin network[2].. scary stuff for anyone with an investment in bitcoin no?  Defeats the purpose of a non centralized trust model too doesn't it? Anyway...

I decided to fire up wireshark on a couple of my mining rigs and set up filters to grab interesting bitcoin related traffic.  Using pylibpcap and massaging the data shows interesting statistical information (long poll data), server response time changes(as workers all find out that they need need data), and changes in the data value in the RPC request seem to correlate nicely and filtering out the noise gets easier with a larger number of miners.

Question:
Are there any open source pool hoppers out there that are actually using longpoll to assist in block completion estimation or are they all just scraping APIs and web pages for block completion times?

[1] http://www.bitcoin.org/bitcoin.pdf
[2] *...by gaining the majority of the bitcoin C/GPU vote The pool administrators are active and easily contacted.  An attacker with an adobe/java/office 0day could potentially social engineer their way into compromising pool operator personal machines and hop the fence from there.  Operators could be coerced. Operators could be malicious.

[hawks5999]

Some appear to be attempting to use LP though I'm not sure if it's open source. Primarily that is relying on pident.

Others are definitely open source, but from root to fork generally rely on json stats or regex scraping. Most the work is focused now on different algorithms, but there is always a search for getting more reliable information more quickly.

[gat3way]

From a miner perspective that was one of the things I targeted for hashkill, but unfortunately it would likely never be released because I have other things to do and bitcoin plugin development is currently frozen. I planned to have two multi-pool modes, failover and load-balance. Failover is easy, but it was exactly long polling that makes load-balance mode difficult to get done properly. My task was also even harder because I spawn a number of workthreads per GPU and utilize all the GPUs within one process. I experimented a bit and found out it is actually hard to get that done properly. Actually I still can't find the best strategy for doing that.

[fpgaminer]

Are there any open source pool hoppers out there that are actually using longpoll to assist in block completion estimation or are they all just scraping APIs and web pages for block completion times?

Not that I know of. There are two issues with trying to identify block finders outside of explicit pool APIs: 1) Block finder estimation has to be accurate. Mining against a proportional pool with >43% shares makes your efficiency <100% (i.e. you're better off mining PPS). So if you guessed wrong, you're now "losing" money. 2) Block finder estimation has to be fast. The most valuable time to mine is at the very beginning of the round. The biggest pools are the primary targets for block finder estimation, and so the problem is much worse. For example, Deepbit's fast rounds (most valuable rounds) are under 10 minutes long. If block finder estimation takes, say, a minute to identify a block's owner then you've lost out on a very large chunk of the round.

The latter problem I assume isn't that bad. Better to have jumped and gotten at least some nice, filthy profit than to not have jumped at all. The former problem is probably the worst. I'm sure someone more involved in the math of pool hopping than I could calculate the required accuracy to make block finder estimation profitable.

The nice thing about having a massive pool like deepbit around is that even a naive block finder estimation (if it isn't any of the known pools, it's probably deepbit) will be correct about 60 to 70% of the time

[iopq]

Are there any open source pool hoppers out there that are actually using longpoll to assist in block completion estimation or are they all just scraping APIs and web pages for block completion times?

Not that I know of. There are two issues with trying to identify block finders outside of explicit pool APIs: 1) Block finder estimation has to be accurate. Mining against a proportional pool with >43% shares makes your efficiency <100% (i.e. you're better off mining PPS). So if you guessed wrong, you're now "losing" money. 2) Block finder estimation has to be fast. The most valuable time to mine is at the very beginning of the round. The biggest pools are the primary targets for block finder estimation, and so the problem is much worse. For example, Deepbit's fast rounds (most valuable rounds) are under 10 minutes long. If block finder estimation takes, say, a minute to identify a block's owner then you've lost out on a very large chunk of the round.

The latter problem I assume isn't that bad. Better to have jumped and gotten at least some nice, filthy profit than to not have jumped at all. The former problem is probably the worst. I'm sure someone more involved in the math of pool hopping than I could calculate the required accuracy to make block finder estimation profitable.

The nice thing about having a massive pool like deepbit around is that even a naive block finder estimation (if it isn't any of the known pools, it's probably deepbit) will be correct about 60 to 70% of the time

why not verify it against the APIs?

step 1. find all the pools that have correct up to date APIs, including PPS, SMPPS, PPLNS, etc.
step 2. if none of them claim to have found a block, your accuracy goes way up

[fpgaminer]

why not verify it against the APIs?

step 1. find all the pools that have correct up to date APIs, including PPS, SMPPS, PPLNS, etc.
step 2. if none of them claim to have found a block, your accuracy goes way up

Yup, that gives fairly high accuracy. If you look at pool size estimation at http://pident.artefact2.com/ you can add up all the pools that don't delay or hide their stats, and figure out how accurate a Process of Elimination algorithm would be.

I think if you add up all the stat-friendly pools (ones that don't delay stats) you've covered about 40% of the landscape. I am terrible at statistics, so feel free to correct the following:

1) 40% of the time: A block will be claimed by a stat-friendly pool. You are thus 100% accurate in this case.
2) 60% of the time: A block isn't immediately claimed. It has a 38 in 60 percent chance of being from Deepbit; that's 63%. If you always guessed Deepbit in this case, you'd thus be right 63% of the time.

(40% * 100%) + (60% * 63%) = 77.8%

By either finding the claiming pool, or guessing Deepbit, you are 77.8% accurate.

Not sure if 77.8% is good enough to profit from hopping to Deepbit. It also means you can never hop to BTCguild or any other stat-unfriendly prop pool.

[iopq]

why not verify it against the APIs?

step 1. find all the pools that have correct up to date APIs, including PPS, SMPPS, PPLNS, etc.
step 2. if none of them claim to have found a block, your accuracy goes way up

Yup, that gives fairly high accuracy. If you look at pool size estimation at http://pident.artefact2.com/ you can add up all the pools that don't delay or hide their stats, and figure out how accurate a Process of Elimination algorithm would be.

I think if you add up all the stat-friendly pools (ones that don't delay stats) you've covered about 40% of the landscape. I am terrible at statistics, so feel free to correct the following:

1) 40% of the time: A block will be claimed by a stat-friendly pool. You are thus 100% accurate in this case.
2) 60% of the time: A block isn't immediately claimed. It has a 38 in 60 percent chance of being from Deepbit; that's 63%. If you always guessed Deepbit in this case, you'd thus be right 63% of the time.

(40% * 100%) + (60% * 63%) = 77.8%

By either finding the claiming pool, or guessing Deepbit, you are 77.8% accurate.

Not sure if 77.8% is good enough to profit from hopping to Deepbit. It also means you can never hop to BTCguild or any other stat-unfriendly prop pool.

now combine that with timing long-polling, and you might be able to predict that it's deepbit almost 100% of the time
and they can't delay that, that would actually screw legitimate miners

[Sukrim]

They can try to detect hoppers and delay only for them - which causes more load on their end, still can screw legitimate miners and would still make it quite efficient (fpgaminer didn't include long polling timing or other timing measures in his approach).

It might even be possible to time long polls of other non-delayed pools for a block from a LP-delayed pool, as the route a block announcement takes through the network is most likely quite static for pools.

I guess 1-2 weeks until deepbit and BTCguild will be hoppable again big time, and there is only 1 simple thing to do for them: Switch to a proper payout system. Everything else either hurts them + their miners too or doesn't help against hopping at all.

[hawks5999]

Mining today is a race between pool operators striving to build bigger and better hopping-proof programs, and the Universe trying to produce bigger and better hopppers. So far, the Universe is winning.

Apologies to Rick Cook.

[Sukrim]

Well, a lot of pool operators already switched to hopping proof algorithms. Once all switch, hopping is dead. As long as even one keeps proportional, it will be hoppable + be hopped.

[ed64]

Hoping will probably always be profitable given that every pool has a run of bad luck now and then. I think the margins will shrink though and make it less viable and more of a hobby...

[Sukrim]

Once you realize a pool has bad luck, it's already too late...

But yes, payouts using any other scheme than pure PPS will show a bit of variance over time.

[lucita777]

Variance alone does not make a pool hoppable. Only unfair payout system does.

Otherwise, good luck with hopping a roulette in the casino - it also has large variance