Does a high pool difficulty lower anyone's profits?

[Liquidfire]

Ok. I had a bit of an epiphany - the argument has been over what diff setting is correct this whole time, but now I understand we really should have been arguing over reward system.

Everything I am saying is accurate for a proportional reward system. If we aren't using a proportional reward system, woah, that probably should have been mentioned weeks ago.

What a lot of the arguments are saying would be true if there was one long block with no block changes.

That's the epiphany I had - we should be arguing to go to PPLNS.


The effect I am describing is present when each block has shares associated with it and when the block is found the reward is shared among those blocks. Everything resets for the new block. Proportional reward.

If we've been using something else, than no wonder we are all disagreeing, we are basically speaking different languages.

But PPLNS would eliminate the bias almost completely. The block change would then have no effect on the payout.

I will concede that in a PPLNS system, yes, all the diff does is introduce variance. I could modify my script in 5 minutes to simulate PPLNS and I bet the numbers would look good.



Lets change this conversation. H20, can you verify what reward system you are using now? It's not explicitly mentioned in your FAQ or announcement statement. Now I realize, unless everyone is on the same page on this we are going to spin our wheels forever.

Second, instead of arguing about diff - which would alleviate the problem I am describing but not eliminate it, can we talk about PPLNS or something similar?

[mueslo]

Ok. I had a bit of an epiphany - the argument has been over what diff setting is correct this whole time, but now I understand we really should have been arguing over reward system.

Everything I am saying is accurate for a proportional reward system. If we aren't using a proportional reward system, woah, that probably should have been mentioned weeks ago.

What a lot of the arguments are saying would be true if there was one long block with no block changes.

That's the epiphany I had - we should be arguing to go to PPLNS.


The effect I am describing is present when each block has shares associated with it and when the block is found the reward is shared among those blocks. Everything resets for the new block. Proportional reward.

If we've been using something else, than no wonder we are all disagreeing, we are basically speaking different languages.

But PPLNS would eliminate the bias almost completely. The block change would then have no effect on the payout.

I will concede that in a PPLNS system, yes, all the diff does is introduce variance. I could modify my script in 5 minutes to simulate PPLNS and I bet the numbers would look good.



Lets change this conversation. H20, can you verify what reward system you are using now? It's not explicitly mentioned in your FAQ or announcement statement. Now I realize, unless everyone is on the same page on this we are going to spin our wheels forever.

Second, instead of arguing about diff - which would alleviate the problem I am describing but not eliminate it, can we talk about PPLNS or something similar?

Nowhere in my previous post am I assuming PPLNS. I am actually assuming straight PPS. Please read it and try to understand it. Or at least look at the python script.

[roy7]

Ah yes, if you are using a proportional payment system which is based solely on the current block, then blocks with no shares you'd get zero and blocks with a share you'd get paid. One could argue that over time it wouldn't matter, if your hash rate is .5 shares per block then the blocks with a share you get paid "double" your average contribution and blocks with no shares you get paid 0. So it averages out to .5 shares/block worth of pay. But only getting paid on blocks where you personally submitted shares kinda violates the spirit of pooled mining, in my opinion.

'Block at a time' proportional payout for a very fast coin where slow miners might not submit shares seems silly. Proportional is a bad payment model anyway, and is hoppable. Best to use DGM or PPLNS if implemented the way Meni Rosefeld says to. Not all pools do PPLNS correctly. Follow this post:

https://bitcointalk.org/index.php?topic=39832

To do it right.

DGM is what I use on my pool and I really like it. (I do not use a negative f value, so there's no risk to the pool at all. PPS is also bad since it virtually guarantees pool bankruptcy if ran long enough.)

[Liquidfire]

You cannot precalculate when an event may happen like this, that would violate you having a constant chance of finding a hash, and would make it dependent on the past.

Thank you for the detailed, thorough response. Give me a chance to digest everything you said. You are clearly more advanced in mathematics and statistics than I (i mean that genuinely not sarcastically) and I would love to learn from you. However, in the mean time, let me ask you this.

If you can't precalculate (you say precalculate, I say predict) the time is takes to solve a block, how is bitcoin (or whatever coin) doing it? Bitcoin itself has to predict the time it will take the network to solve a block, to know how to adjust the difficulty. It MUST maintain an average block find time of 10 minutes, or quite literally the entire thing will come crashing to a screeching halt.

If bitcoin couldn't accurately predict the time it takes to solve a block given X hashpower, we wouldn't be having this conversation because bitcoin would have died, and none of the alt coins that we love to mine at middlecoin would have ever existed.

[Liquidfire]

Nowhere in my previous post am I assuming PPLNS. I am actually assuming straight PPS. Please read it and try to understand it. Or at least look at the python script.

You were assuming straight PPS, I was assuming per block proportional. Do you see? We will never convince each other because we are both right, in the respective systems we think we are operating under.

[roy7]

If you can't precalculate (you say precalculate, I say predict) the time is takes to solve a block, how is bitcoin (or whatever coin) doing it? Bitcoin itself has to predict the time it will take the network to solve a block, to know how to adjust the difficulty. It MUST maintain an average block find time of 10 minutes, or quite literally the entire thing will come crashing to a screeching halt.

Bitcoin adjusts difficulty level based on historical data only, no predictions. It compares the time it took to get the last X blocks to what it "should" have taken, and then adjusts the difficulty accordingly to try and be closer to the target speed next time. Which would be very accurate if hash rate was constant.

[mueslo]

Nowhere in my previous post am I assuming PPLNS. I am actually assuming straight PPS. Please read it and try to understand it. Or at least look at the python script.

You were assuming straight PPS, I was assuming per block proportional. Do you see? We will never convince each other because we are both right, in the respective systems we think we are operating under.

Sorry, but you are wrong. You were not simulating per block proportional, you were assuming a wrong distribution for the time between shares.

Thank you for the detailed, thorough response. Give me a chance to digest everything you said. You are clearly more advanced in mathematics and statistics than I (i mean that genuinely not sarcastically) and I would love to learn from you. However, in the mean time, let me ask you this.

If you can't precalculate (you say precalculate, I say predict) the time is takes to solve a block, how is bitcoin (or whatever coin) doing it? Bitcoin itself has to predict the time it will take the network to solve a block, to know how to adjust the difficulty. It MUST maintain an average block find time of 10 minutes, or quite literally the entire thing will come crashing to a screeching halt.

If bitcoin couldn't accurately predict the time it takes to solve a block given X hashpower, we wouldn't be having this conversation because bitcoin would have died, and none of the alt coins that we love to mine at middlecoin would have ever existed.

I elaborated a bit at the end, because that's what I thought was most likely to be read:

Now on to your simulation: You are using the wrong distribution. If you don't know why, read my post again from the top, read the wiki page or watch the video I linked above.

The probability distribution of the times between finding shares (again, this is a poisson process) is simply e^(-t/T), where T is the average time between shares. Which has exactly the property that you can just restart it at all points without changing anything.

edit to clarify: the chance P of the time you need to solve a block is P(t) = e^(-t/T). For generating t, you can for example use numpy.random.exponential(scale=T)

[Liquidfire]

Nowhere in my previous post am I assuming PPLNS. I am actually assuming straight PPS. Please read it and try to understand it. Or at least look at the python script.

You were assuming straight PPS, I was assuming per block proportional. Do you see? We will never convince each other because we are both right, in the respective systems we think we are operating under.

Sorry, but you are wrong. You were not simulating per block proportional, you were assuming a wrong distribution for the time between shares.

Its not Miner A vs. the Pool and Miner B vs. the pool

Its Miner A vs. Miner B.

The distribution doesn't matter very much, because miner A and miner B both receive it the same system. I arbitrarily picked the averages out of the thin air. I can do that, because on the pool at any given time we can find miners all over the scale. That's why I can say miner A has this distribution, because no matter what, I can find a miner with that average.

To put it another way, its pointless to argue over weather miner A with average X would really distribute that way. Even if you are right, there is another miner with average Y who WOULD, and I can pick any of them since the competition is between the two miners, not miner vs. pool.

[mueslo]

Its not Miner A vs. the Pool and Miner B vs. the pool

Its Miner A vs. Miner B.

The distribution doesn't matter very much, because miner A and miner B both receive it the same system. I arbitrarily picked the averages out of the thin air. I can do that, because on the pool at any given time we can find miners all over the scale. That's why I can say miner A has this distribution, because no matter what, I can find a miner with that average.

To put it another way, its pointless to argue over weather miner A with average X would really distribute that way. Even if you are right, there is another miner with average Y who WOULD, and I can pick any of them since the competition is between the two miners, not miner vs. pool.

If the distribution is not that of a Poisson process, all your results will be wrong. Otherwise you are not simulating a Poisson process (which hashing is). If you like, I can change the script I wrote to a per block reward system, I'm sure it would come out the same. If not, I concede.

Which reward scheme is middlecoin using anyway? I have no idea. But I don't think it's per block reward.

[Liquidfire]

If you like, I can change the script I wrote to a per block reward system, I'm sure it would come out the same. If not, I concede.

I am curious what will happen if you implement the following conditions:

At the end of the block, worker shares get reset to 0.
Just before that happens, the shares are used to calculate payout, and the payout occurs per block.

[mueslo]

I am curious what will happen if you implement the following conditions:

At the end of the block, worker shares get reset to 0.
Just before that happens, the shares are used to calculate payout, and the payout occurs per block.

At the end of every block? That wouldn't make much sense though (there is a big chance of loss for the pool operator if the network suddenly becomes fast).

What makes more sense: when a block is found by the pool itself (and not the whole network), all previous shares are reset to zero and the reward of that block is distributed among shares.  I'm guessing you want the latter?

(I don't think we're using that either, because it's vulnerable to pool hopping if you know when the pool found a block.)

But anyway, if you want to actually simulate this, you can no longer calculate the time when a block is found, because then it could happen that the pool finds a block without any share actually having been submitted. You now have to actually calculate the odds of one of the miners in the pool having found the block.

[Liquidfire]

What makes more sense: when a block is found by the pool itself (and not the whole network), all previous shares are reset to zero and the reward of that block is distributed among shares.  I'm guessing you want the latter?

The fact is I really don't know how it works right now. But, when I am talking about block change I mean block change for the network. I am starting to realize we are all working under a lot of personal assumptions about this because we don't know for sure what the reward system is. My assumption, and the one I've been arguing from (I hope not for nothing, but at least it would be settled), is that we get paid per block per shares that we have for that block.

But anyway, if you want to actually simulate this, you can no longer calculate the time when a block is found, because then it could happen that the pool finds a block without any share actually having been submitted

This could happen, yes (very very unlikely but possible). But, you can simulate it.  That could happen in my simulation and I believe it would handle it correctly.

If my reward is set to 100, and neither worker got a share in time , it would simply pay 0/100 (0) of the reward to each.


We can continue to argue weather my system is correct in a proportional-per-block scenario, but we still aren't sure if that is the case anymore, so I'm just holding out to hear the scoop on that.

Part of me hopes I was dead wrong and H20 is using some other reward system so we can finally put it to bed.

At the end of every block? That wouldn't make much sense though (there is a big chance of loss for the pool operator if the network suddenly becomes fast).

It shouldn't effect the pool as a whole. The pool as a whole would still solve just as many blocks, it should only affect the shares within the pool.
 

[mueslo]

[...]

Here's the code for per pool-found-block/proportional reward. There's no change (as is to be expected, all proportional reward does is increase variance (as long as there's no pool hopping)):

Code:

import numpy.random as rnd
import numpy as np

class worker():
    sharetime = None #time the next share is found by the worker
    def __init__(self,avgsharetime):
        self.avgsharetime = avgsharetime
        self.hashrate = 60/avgsharetime
        self.shares = 0
        self.roundshares = 0
        self.income = 0.
        self.generateShareTime(0.0)
    
    def generateShareTime(self, currenttime):
        self.sharetime = currenttime + rnd.exponential(scale=self.avgsharetime)
    
    def foundShare(self, currenttime):
        self.shares+=1
        self.roundshares+=1
        self.generateShareTime(currenttime)


class pool():
    t = 0. #time elapsed
    blocktime = None #time the next block is found (by the network)
    pblock = 0.02 #probability that a share solves the block
    totalroundshares = 0 #number of total shares solved for this block round
    workers = [] #list of workers in the pool
    def __init__(self,avgnetworkblocktime=10.,workers=[],):
        self.avgnetworkblocktime = avgnetworkblocktime
        self.generateNetworkBlockTime(0.0)
        self.workers=workers
        
    def start(self,duration=100.):
        while self.t<duration:
            nextworker = self.workers[np.argmin([w.sharetime for w in self.workers])]
            nextsharetime = nextworker.sharetime            
            
            if self.blocktime<nextsharetime: #network block found
                self.newNetworkBlock()
            
            else:                            #worker found a share
                self.t = nextworker.sharetime
                self.totalroundshares+=1
                nextworker.foundShare(self.t)

                 
                if rnd.uniform(0,1)<self.pblock: #share solved the block
                    self.newPoolBlock()


        
    def generateNetworkBlockTime(self,currenttime):
        self.blocktime = currenttime + rnd.exponential(scale=self.avgnetworkblocktime)
        
    def distributeRewards(self):
        print "Shares: [",
        for w in self.workers:
            print w.roundshares,
            w.income += float(w.roundshares)/float(self.totalroundshares)
            w.roundshares = 0
        self.totalroundshares = 0
        print "]\n"
        
    def newPoolBlock(self):
        print "\nnew pool block t=",self.t,"-> Payout!"
        self.distributeRewards()
        
        self.blocktime = self.t
        self.newNetworkBlock(echo=False)
        
    def newNetworkBlock(self,echo=True):
        self.t=self.blocktime
        if echo: print "new network block t=",self.t
        for w in self.workers: #if you disable these, nothing changes in the outcome
            w.generateShareTime(self.t)
        
        self.generateNetworkBlockTime(self.t)
        


slowworker = worker(avgsharetime=30)
mediumworker = worker(avgsharetime=10)
fastworker = worker(avgsharetime=3)


pool1 = pool(workers=[fastworker, mediumworker, slowworker],avgnetworkblocktime=10.)
pool1.start(duration=100000.)

print fastworker.shares
print slowworker.shares
    
print "Slow worker has: " + str((float(slowworker.hashrate) / float(fastworker.hashrate)) * 100) + ' percent of the hash power of fast worker'
print "Slow worker has: " + str((float(slowworker.income) / float(fastworker.income)) * 100) + ' percent of the profit of fast worker'
print "Slow worker has: " + str((float(slowworker.shares) / float(fastworker.shares)) * 100) + ' percent of the total shares of fast worker'

And some example outputs:
Slow worker has: 10.0 percent of the hash power of fast worker
Slow worker has: 10.0643834344 percent of the profit of fast worker
Slow worker has: 10.2436090226 percent of the total shares of fast worker

Slow worker has: 10.0 percent of the hash power of fast worker
Slow worker has: 9.89355458187 percent of the profit of fast worker
Slow worker has: 9.99728580476 percent of the total shares of fast worker

Slow worker has: 10.0 percent of the hash power of fast worker
Slow worker has: 10.2071470481 percent of the profit of fast worker
Slow worker has: 10.1000090098 percent of the total shares of fast worker

Slow worker has: 10.0 percent of the hash power of fast worker
Slow worker has: 9.78155423167 percent of the profit of fast worker
Slow worker has: 10.1675711134 percent of the total shares of fast worker

Slow worker has: 10.0 percent of the hash power of fast worker
Slow worker has: 9.84810455803 percent of the profit of fast worker
Slow worker has: 9.8902409926 percent of the total shares of fast worker

example found blocks

[juhakall]

It's proportional

https://bitcointalk.org/index.php?topic=259649.msg2900385#msg2900385

Glad to see this discussion is actually progressing. The lack of personal remarks in this thread is almost inexplicable.

[roy7]

Well done mueslo. You have more patience than I do.

[cverity]

There is one provision with that statement, it takes a relatively long time to average out the larger variance the small miners suffer.

This.

The drawback is that it also increases the day-to-day variance.  In the long-run, it doesn't matter, though.

This.

Just with a higher variance.

This.

Variance is what I've been mentioning in the original thread for days.  High diff will not affect pool's profit, but it will affect small miner's reasonable variance.

h2o has previously stated that we are using a proportional system.  I agree that changing the payout algorithm to something like PPLNS would also help smooth out the variance.

Right now we have the worst possible setup for small miners - high diff with a payout calculation that does nothing to smooth out variance.

Over a long enough period of time (although I have no idea if that's weeks, months, etc) it shouldn't matter. But isn't the whole point of a pool to minimize variance?

[Liquidfire]

I am sure this will make some of you happy...

Mueslo - I finally just took your numpy.random.exponential and replaced my poisson function with it, in my own simulation. When I did, numbers started coming out looking like they should (actual return close to expected). This backed your claim that my distribution was wrong. As I started generating plots of each distribution, things started to make sense.

As hard as I've been dug in on this, most people would just bug out, but it wouldn't be right for me to not tell you that you were right on the distribution. H2O tried to explain this to me too, and a couple others.

My mental roadblock (or rather what cleared it) was the fact that even in a lopsided distribution you can still have an average, but it will be very skewed toward the thick end of the plot. I needed to actually see it plotted before it clicked.

But, my simulation was still correct (yours was too), less the distribution, so I guess I learned I'm better at programming than math :p

Anyway, thanks for actually trying to explain things to me and not being an ass like a couple others. I learned quite a bit researching all of this.


For the record:

<sub>import random
import numpy as np
import numpy.random as rnd

class worker():
    def __init__(self,hashrate):
        self.hashrate = hashrate
        self.sharesolvetime = 60 / hashrate
        self.shares = 0

class pool():
    def __init__(self,blockfindtime):
        self.blockfindtime = blockfindtime

pool1 = pool(30)
worker1 = worker(1)
worker2 = worker(12)
samplesize = 1000000

for n in range(0,samplesize):
    clock = rnd.exponential(scale=pool1.blockfindtime)
    clock1 = clock
    while clock1 > 0:
        sharesolve = rnd.exponential(scale=worker1.sharesolvetime)
        if sharesolve > clock1:
            break
        else:
            worker1.shares = worker1.shares + 1
            clock1 = clock1 - sharesolve
    clock2 = clock
    while clock2 > 0:
        sharesolve = rnd.exponential(scale=worker2.sharesolvetime)
        if sharesolve > clock2:
            break
        else:
            worker2.shares = worker2.shares + 1
            clock2 = clock2 - sharesolve
    
print "Worker 1 has: " + str((float(worker1.hashrate) / float(worker2.hashrate + worker1.hashrate)) * 100) + ' percent of the hash power'
print "But worker 1 has: " + str((float(worker1.shares) / float(worker2.shares + worker1.shares)) * 100) + ' percent of the profit'
print "Over sample size of " + str(samplesize)
print "When worker1's average share-find-speed was: " + str((float(pool1.blockfindtime) / float(worker1.sharesolvetime))) + 'X the block-find-speed'</sub>
    




Very Slow

Worker 1 has: 7.69230769231 percent of the hash power
But worker 1 has: 7.69363865886 percent of the profit
Over sample size of 1000000
When worker1's average share-find-speed was: 8.33333333333X the block-find-speed

Slow

Worker 1 has: 7.69230769231 percent of the hash power
But worker 1 has: 7.6898534448 percent of the profit
Over sample size of 1000000
When worker1's average share-find-speed was: 4.0X the block-find-speed

Medium

Worker 1 has: 7.69230769231 percent of the hash power
But worker 1 has: 7.68459728742 percent of the profit
Over sample size of 1000000
When worker1's average share-find-speed was: 2.0X the block-find-speed

Fast

Worker 1 has: 7.69230769231 percent of the hash power
But worker 1 has: 7.68286758249 percent of the profit
Over sample size of 1000000
When worker1's average share-find-speed was: 1.0X the block-find-speed

Very Fast

Worker 1 has: 7.69230769231 percent of the hash power
But worker 1 has: 7.67015222587 percent of the profit
Over sample size of 1000000
When worker1's average share-find-speed was: 0.5X the block-find-speed

[mueslo]

My mental roadblock (or rather what cleared it) was the fact that even in a lopsided distribution you can still have an average, but it will be very skewed toward the thick end of the plot. I needed to actually see it plotted before it clicked.

This is why I stuck through, I thought I might help someone understand it better Thank you very much for the follow up. I actually learned something too (at first I thought it is always wrong to calculate the times in advance, which, of course, given the correct distribution in times between finding a share, it isn't).

Although there is still one phenomenon I cannot explain. I have a HD 7950 and a HD 5830. The HD 7950 consistently gets 2-5% rejects, while the HD 5830 gets 6-12% rejects, even if I set clock and intensity on the HD 7950 such that the hashrates are equal, so it isn't dependent on hash rates.

[FlungSpun]

that must be something to do with getting the share off the card and onto the pool no?
though that is quite a difference. assuming no HW error type issues, same machine,  switch, router etc

[mueslo]

Yeah, they're both in the same machine, the only difference apart from the actual cards is which PCIe slot they're in, obviously. I do occasionally get HW errors on the HD 5830 (about 1 every 2-3 days), but I don't see the connection.