Bitcoin Forum

Alternate cryptocurrencies => Altcoin Discussion => Topic started by: h2odysee on August 22, 2013, 08:01:42 PM



Title: Does a high pool difficulty lower anyone's profits?
Post by: h2odysee on August 22, 2013, 08:01:42 PM
There has been a lot of discussion about this in the middlecoin.com thread: https://bitcointalk.org/index.php?topic=259649.new#new

Let's move it here, to keep the topics separate.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Liquidfire on August 22, 2013, 08:11:37 PM
I now think the correct distribution for my simulation would be the Poisson distribution. Can any math geeks confirm or deny this for me?


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: jdebunt on August 22, 2013, 08:12:04 PM
only thing that lowers profitability is idiots dumping their coins to the lower buy order.... :)


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Eli0t on August 22, 2013, 08:12:50 PM
it creates very low profits for low hashers who cant submit a share before the round ends on fast coins

your pool sells coins at the lowest price possible, this also reduces profits


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Hydroponica on August 22, 2013, 08:16:01 PM
Holy fuck, why don't you stop discussing it, and lower share difficulty for a day, and see what happens.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: GSnak on August 22, 2013, 08:19:55 PM
only thing that lowers profitability is idiots dumping their coins to the lower buy order.... :)

I thought that too, until I noticed all my orders getting left behind. Now I pretty much dump on the top bid.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: FlungSpun on August 22, 2013, 08:33:47 PM
it creates very low profits for low hashers who cant submit a share before the round ends on fast coins

A - Low hashers cannot expect to submit a share for every block on a fast coin, what makes anyone think that they should be submitting for every block?

B It makes no difference because they will still submit on blocks in relation to their hashrate and earn accordingly.


and what happens when a block is solved in 1 share and it happens to come from a low hasher?
should all the big hasher cry foul ?



Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Eli0t on August 22, 2013, 08:43:37 PM
its possible to prove you worked on a solution for every block if the share diff is reasonable but when its high there can be long periods where you cant prove your working on a solution. when a pool doesnt credit you for the work you put in your wasting your time



Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: FlungSpun on August 22, 2013, 08:51:15 PM
I really don't want to get drawn into this argument because I understand the mechanics but realise I won't be able to explain it clearly enough.

From a confidence perspective it would be nice for smaller hashers to be able to prove work for every block (lower diff)
But
The ratio between the high and low rates of share submission will not change (ergo payouts) because the big hashers will just inundate the pool with PoW shares.
There is one provision with that statement, it takes a relatively long time to average out the larger variance the small miners suffer.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Damnsammit on August 22, 2013, 08:52:02 PM
Regardless of your hashrate, a higher difficulty reduces latency between the miners and the pool server.

The drawback is that it also increases the day-to-day variance.  In the long-run, it doesn't matter, though.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: turtle83 on August 22, 2013, 08:52:25 PM
its possible to prove you worked on a solution for every block if the share diff is reasonable but when its high there can be long periods where you cant prove your working on a solution. when a pool doesnt credit you for the work you put in your wasting your time



In the long run it averages out completely. the hashers would have payout proportional to their hashrate. in the short run, with higher diffs low hashers have an increased variance...

The only reason for higher diff shares is efficiency, because lesser load on the networks, CPU, etc.

Quote
Does a high pool difficulty lower anyone's profits?

No. It only increases variance in the short term... which could go both ways.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: FlungSpun on August 22, 2013, 08:54:40 PM
I'm pretty sure the last three statement have nailed this discussion.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: mueslo on August 22, 2013, 09:04:43 PM
I now think the correct distribution for my simulation would be the Poisson distribution. Can any math geeks confirm or deny this for me?

If you had actually read the rebuttals, you'd know we already wrote that.

https://bitcointalk.org/index.php?topic=259649.msg2979579#msg2979579


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Agrippa on August 22, 2013, 09:05:58 PM
Well it's nice to see some consensus here after all the insanity of the middlecoin thread.

Like I was trying to say before, complaining that your miner would have gotten to get a share had the block lasted just a few more seconds is not only wrong because it is a complete misunderstanding of probability, but because by that logic you might as well complain about all the wasted hashes that would have yielded a share on the very next block.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: FlungSpun on August 22, 2013, 09:19:47 PM
http://upload.wikimedia.org/wikipedia/commons/thumb/1/12/Dice_Distribution_%28bar%29.svg/250px-Dice_Distribution_%28bar%29.svg.png
http://upload.wikimedia.org/wikipedia/commons/thumb/8/8c/Standard_deviation_diagram.svg/250px-Standard_deviation_diagram.svg.png


Quick crib from Wikipedia cos that's always right.

imagine that with the dominoes H2 of middlecoin fame decides he wants to minimise his incoming traffic to keep his pool slick and reactive
but he need to do this fairly.

He tells all miners that he is not interested in dominoes that add up to 7 (they are the most common to be found)

UNLESS A 7 HASH SOLVES THE BLOCK they wont get sent.

Everyone has to abide by the same rules and have exactly the same chance of finding the outlier dominoes and be paid for the proof they are doing work.
by removing the central column of probability there is less network traffic generated but the odds remain the same for all.

http://en.wikipedia.org/wiki/Probability_distribution (http://en.wikipedia.org/wiki/Probability_distribution)


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: mueslo on August 22, 2013, 09:55:40 PM

He tells all miners that he is not interested in dominoes that add up to 7 (they are the most common to be found)

UNLESS A 7 HASH SOLVES THE BLOCK they wont get sent.


That's pretty flawed analogy by any standard. Those distributions have nothing whatsoever to do with what actually happens, they aren't scalable.

This is the probability distribution of zeros: P(n) = 2^(-n), where 32≤n≤256 the number of zeros.

https://i.imgur.com/rveuphy.gif


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: turtle83 on August 22, 2013, 10:06:17 PM

He tells all miners that he is not interested in dominoes that add up to 7 (they are the most common to be found)

UNLESS A 7 HASH SOLVES THE BLOCK they wont get sent.


That's pretty flawed analogy by any standard. Those distributions have nothing whatsoever to do with what actually happens, they aren't scalable.

This is the probability distribution of zeros: P(n) = 2^(-n), where 32≤n≤256 the number of zeros.

https://i.imgur.com/rveuphy.gif

A 20 GH/s miner is twice as likely to solve a diff 1 share than a 10 GH/s miner.
A 20 GH/s miner is twice as likely to solve a diff 10 share than a 10 GH/s miner.
A 20 GH/s miner is twice as likely to solve a diff 100 share than a 10 GH/s miner.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: FlungSpun on August 22, 2013, 10:13:29 PM
mueslo
yes very guilty of oversimplification but the analogy is not so flawed as to miss the point for some of the misconceptions I am trying to clear up.
I hope my description simplifies the issues without preaching anything baseless. I'm not explaining how crypto mining works just the fairness aspect of pool difficulty.

I didn't think presenting the information in the form you have above was going to help the discussion at this point.

If you want to take the discussion further in to the a detailed description and minutia of mathematics please be my guest.
I was getting sick of the huge misunderstanding some people were labouring under.






Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Liquidfire on August 22, 2013, 10:41:05 PM
I am in the process of updating my simulation script to include a more accurate distribution of block/share times.

I believe this to be a positive Poisson distribution at this time. I don't believe it will make much difference, but it needs to be addressed before some people will accept my work as proof.


For you new comers, the "con" argument to high diff is essentially this. Fast miners are more efficient at reporting their work, that is, they have less work go "unreported". Slower miners are less efficient at reporting their work, because the interruption of new blocks makes a larger percentage of their work go unrecognized. They will have been working longer without finding a share at block change.

Result: no change to overall pool profit. The effect is of the distribution if rewards.

The fastest miners are basically stealing some of the slowest miners hashrate, in terms of reward.

Now, I am no conpiracy theorist. However, it has crossed my mind that some of the people in this forum are some of the fastest miners. They might know about this effect, know it benefits them, and are therefore trying to keep the status quo. I am certainly not accusing anyone of that and I think it's unlikely that this is the case, just something to tuck away in the back of your minds.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: h2odysee on August 22, 2013, 10:48:51 PM
In your simulation, I suggest you don't use any of those formulas, and just model mining as though it really is at the base level. Randomly generate a number between 1 and 100000, and if it's below 1000, it's a share. If it's below 10, it's a block.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Liquidfire on August 22, 2013, 10:57:39 PM
In your simulation, I suggest you don't use any of those formulas, and just model mining as though it really is at the base level. Randomly generate a number between 1 and 100000, and if it's below 1000, it's a share. If it's below 10, it's a block.

I thought about doing this. The problem is then I need to process the workers in a parallell fashion. Meaning run multiple threads/processes. This is because now both miners are racing each other. This complicates the code significantly, but also now you have introduced variables that you didn't have before, race conditions within the operating system.

I agree that would be a more accurate method, but to do it really properly you almost be better off just setting up a real pool and have real miners mine shares at different rates and take stats.

Minus my admittedly inefficient block/share time random generation (working on that), as long as you trust the probability it should be accurate, to the best of my knowledge. The block solve for each block is used by both workers on that block, so they can accurately be evaluated serially.

Edit: also, 2 workers is a really small sample in terms of solving a block, something that is in reality really consistent


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: h2odysee on August 22, 2013, 11:06:15 PM
So here's a sim I just made: http://pastebin.com/nud90qDK

After doing it, I realized that there's really not much to simulate. It's pretty trivial.

No matter how fast the block rate, I get the same result:

Shares per step: 0.01029


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: turtle83 on August 22, 2013, 11:06:28 PM
Fast miners are more efficient at reporting their work, that is, they have less work go "unreported". Slower miners are less efficient at reporting their work, because the interruption of new blocks makes a larger percentage of their work go unrecognized. They will have been working longer without finding a share at block change.

Result: no change to overall pool profit. The effect is of the distribution if rewards.

Look forward to your simulations...

but.. i think the argument is invalid.

Say the duration to the interruption is 20 seconds.

A 100 GH/s miner will have exactly 10x the odds of finding a share than 10 GH/s miner. irrespective of the share difficulty. Over time its the same distribution of rewards.

The argument sounds like saying : Pools using diff 1 shares is unfair, because I have done many hashes and didnt find a single diff 1 hash to submit. The work was done and went unrecognized.

I am pretty sure when you do the simulation, you will find that over time, at any difficulty, the 100GH/s miner would have submitted 10x of what 10 GH/s miner would have.

Going by h2odysee suggestion, try this. 2 processes generating random numbers between 1 and 1000000. Process A generates 1000 randoms a second, B does 10000 / sec. Log the output in one place if the number is 100 and another place if its below 10. run the processes many times for 10 second intervals.... im pretty sure both A and B will have similar ratio for the 2 difficulties... over time... i.e. B's ratio will be more consistent than A's but over time itll be the same.



Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Liquidfire on August 23, 2013, 12:10:46 AM
Fast miners are more efficient at reporting their work, that is, they have less work go "unreported". Slower miners are less efficient at reporting their work, because the interruption of new blocks makes a larger percentage of their work go unrecognized. They will have been working longer without finding a share at block change.

Result: no change to overall pool profit. The effect is of the distribution if rewards.

Look forward to your simulations...

but.. i think the argument is invalid.

Say the duration to the interruption is 20 seconds.

A 100 GH/s miner will have exactly 10x the odds of finding a share than 10 GH/s miner. irrespective of the share difficulty. Over time its the same distribution of rewards.

The argument sounds like saying : Pools using diff 1 shares is unfair, because I have done many hashes and didnt find a single diff 1 hash to submit. The work was done and went unrecognized.

I am pretty sure when you do the simulation, you will find that over time, at any difficulty, the 100GH/s miner would have submitted 10x of what 10 GH/s miner would have.

Going by h2odysee suggestion, try this. 2 processes generating random numbers between 1 and 1000000. Process A generates 1000 randoms a second, B does 10000 / sec. Log the output in one place if the number is 100 and another place if its below 10. run the processes many times for 10 second intervals.... im pretty sure both A and B will have similar ratio for the 2 difficulties... over time... i.e. B's ratio will be more consistent than A's but over time itll be the same.




When a block changes, every miner will have been working towards their next share.

But one share means different things for different miners. Losing one share for a slow miner is a bigger hit than for a very fast miner.

In the time the slow miner was working, getting nothing (yet), the fast miner already got work in, and got it accepted.

Higher ratio of recognized work / work actually done.



Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: mueslo on August 23, 2013, 12:50:37 AM
So here's a sim I just made: http://pastebin.com/nud90qDK

After doing it, I realized that there's really not much to simulate. It's pretty trivial.

No matter how fast the block rate, I get the same result:

Shares per step: 0.01029

I also made one in the old thread, but again, Liquidfire didn't even answer. https://bitcointalk.org/index.php?topic=259649.msg2988147#msg2988147

mueslo
yes very guilty of oversimplification but the analogy is not so flawed as to miss the point for some of the misconceptions I am trying to clear up.
I hope my description simplifies the issues without preaching anything baseless. I'm not explaining how crypto mining works just the fairness aspect of pool difficulty.

I didn't think presenting the information in the form you have above was going to help the discussion at this point.

If you want to take the discussion further in to the a detailed description and minutia of mathematics please be my guest.
I was getting sick of the huge misunderstanding some people were labouring under.

An anology with a die would is much better, it's simpler and more extensible.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: roy7 on August 23, 2013, 01:07:08 AM
In the time the slow miner was working, getting nothing (yet), the fast miner already got work in, and got it accepted.

How is that a problem? The slow miner hasn't found a share yet. Whenever he does find one, he'll report it to the pool and get credited.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Arros on August 23, 2013, 01:38:15 AM
In your simulation, I suggest you don't use any of those formulas, and just model mining as though it really is at the base level. Randomly generate a number between 1 and 100000, and if it's below 1000, it's a share. If it's below 10, it's a block.

I thought about doing this. The problem is then I need to process the workers in a parallell fashion. Meaning run multiple threads/processes. This is because now both miners are racing each other. ...

I think you can still simulate their race in sequence by seeing how long each one takes to get something, and compare their "times" to see who won.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Liquidfire on August 23, 2013, 02:01:55 AM
In the time the slow miner was working, getting nothing (yet), the fast miner already got work in, and got it accepted.

How is that a problem? The slow miner hasn't found a share yet. Whenever he does find one, he'll report it to the pool and get credited.

Because the block changed. We he finds one, its a whole new block by then. All his work on that last block will never be given credit.

Meanwhile, fast miner got 9 on the last block (or 8, or 10, whatever you like). Even if slow miner finds a block this time, fast miner probably got 8-12 shares again.

Over time, the block changing hurts the slow miner more.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: mueslo on August 23, 2013, 02:18:14 AM
Because the block changed. We he finds one, its a whole new block by then. All his work on that last block will never be given credit.

That's just not the case. He has the exact same chances per hashrate of finding a share before the block changes as a high hashrate miner. Just with a higher variance.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: the joint on August 23, 2013, 02:24:03 AM
I think of a higher difficulty in a pool as solo-mining with a lower difficulty.  If you want to try to be lucky for a day as a GPU or Block Erupter miner, set your stratum difficulty to 128 or 256 and watch the variance kick in.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Liquidfire on August 23, 2013, 03:20:36 AM
Because the block changed. We he finds one, its a whole new block by then. All his work on that last block will never be given credit.

That's just not the case. He has the exact same chances per hashrate of finding a share before the block changes as a high hashrate miner. Just with a higher variance.

If the block changes every 30 seconds on average, and you find a share every 30 seconds on average, and someone else finds a share every 5 seconds, how do you have the same chances?

You'll completely whiff on any given block just as often as you get one share.. meanwhile the other guy gets 6 in. If he has a little bad luck, he gets 5. You have a little bad luck? you get 0. But, it all evens out right? A little bad luck this time, a little good luck next time. Of course every time you have a little good luck, you don't get 2, you'll still get 1. He'll get 7

You each have a good block and a bad block. You have 1 share. He has 12.

1/12 is not the same ratio as the ratio as 1/6.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: CryptoBullion on August 23, 2013, 03:40:57 AM
visit pools in my sig ... i have tuned the pplns to cover for exactly what you are talking about  ;D i noticed this happening a long time ago and couldnt stand seeing the little miners get poor pplns score. my diff is set to 63 which will save everyone bandwidth but the pplns scoring i use will even what you were asking about.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Liquidfire on August 23, 2013, 03:50:46 AM
Alright. In the main forum, I posted my simulation script showing the skew towards fast miners. If you want the background, read that.

The main criticism was a fair one - I was "cheating" and using 0.5x and 1.5x of the average as the range for my random values for the share time, and block times. It was a shortcut that, admittedly skewed the results, but in such a negligible way that I felt it would still expose the effect. It was essentially cutting off the tiny sliver on the far right end the pierces out into infinity (plus some of the end on both sides). Its a minuscule amount of the distribution, but it was certainly a fair criticism.

So I did some research. The appropriate distribution is the poisson distribution. This is the same distribution that is used by bitcoin itself to calculate the expected difficulty for the next difficulty change. It is used in the prediction of the network, to maintain the statistical probability of one block every 10 minutes. Bitcoin would collapse without this prediction.

So it turns out theres a wonderful module in python called numpy. It has a built in poisson distribution random generator, making my life easy and making the change a breeze.

The main point about the poisson distribution, that should address the concerns: The Poisson distribution is not symmetrical; it is skewed toward the infinity end.

I also found and fixed a bug in my stats calculation. It didn't change the results enough to invalidate my previous conclusion, but these results should be accurate.

So with that, I reran my tests from before.


Slowest Coin

Worker 1 has: 7.69230769231 percent of the hash power
But worker 1 has: 7.29973071787 percent of the profit
Over sample size of 100000
When worker1's average share-find-speed was: 8.33333333333X the block-find-speed

Slow Coin

Worker 1 has: 7.69230769231 percent of the hash power
But worker 1 has: 6.86225439815 percent of the profit
Over sample size of 100000
When worker1's average share-find-speed was: 4.0X the block-find-speed

Medium Coin

Worker 1 has: 7.69230769231 percent of the hash power
But worker 1 has: 6.00872764122 percent of the profit
Over sample size of 100000
When worker1's average share-find-speed was: 2.0X the block-find-speed

Fast Coin

Worker 1 has: 7.69230769231 percent of the hash power
But worker 1 has: 4.23694576719 percent of the profit
Over sample size of 100000
When worker1's average share-find-speed was: 1.0X the block-find-speed

Very Fast Coin

Worker 1 has: 7.69230769231 percent of the hash power
But worker 1 has: 0.0129950864524 percent of the profit
Over sample size of 100000
When worker1's average share-find-speed was: 0.5X the block-find-speed

As you can see, the results are very similar.



And of course, obligatory code so you know I am not full of shit and can try for yourself.

import random
import numpy as np

class worker():
    def __init__(self,hashrate):
        self.hashrate = hashrate
        self.sharesolvetime = 60 / hashrate
        self.shares = 0

class pool():
    def __init__(self,blockfindtime):
        self.blockfindtime = blockfindtime

pool1 = pool(500)
worker1 = worker(1)
worker2 = worker(12)
samplesize = 100000

for n in range(0,samplesize):
    clock = np.random.poisson(pool1.blockfindtime)
    clock1 = clock
    while clock1 > 0:
        sharesolve = np.random.poisson(worker1.sharesolvetime)
        if sharesolve > clock1:
            break
        else:
            worker1.shares = worker1.shares + 1
            clock1 = clock1 - sharesolve
    clock2 = clock
    while clock2 > 0:
        sharesolve = np.random.poisson(worker2.sharesolvetime)
        if sharesolve > clock2:
            break
        else:
            worker2.shares = worker2.shares + 1
            clock2 = clock2 - sharesolve
    
print "Worker 1 has: " + str((float(worker1.hashrate) / float(worker2.hashrate + worker1.hashrate)) * 100) + ' percent of the hash power'
print "But worker 1 has: " + str((float(worker1.shares) / float(worker2.shares + worker1.shares)) * 100) + ' percent of the profit'
print "Over sample size of " + str(samplesize)
print "When worker1's average share-find-speed was: " + str((float(pool1.blockfindtime) / float(worker1.sharesolvetime))) + 'X the block-find-speed'
    



If you want to run it yourself, you need numpy. http://www.numpy.org/


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: h2odysee on August 23, 2013, 05:15:26 AM
Looking at a graph of a poisson distribution, it starts low, peaks, then falls off. That's not the right distribution to use here. We need to start high, and fall off. The peak should be the very first sample.

If you are using a difficulty such that you have a 50% chance of getting a share each hash, the distribution will look like so:
1st hash: 50%
2nd hash: 25%
3rd hash: 12.5%
4th hash: 6.25%
and so on, dividing by two each time.

So whatever kind of distribution you call that.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: CoinBuzz on August 23, 2013, 08:38:11 AM
In other pools, VARDIFF select 64 difficulty for my hashrate,

How VARDIFF decide on that? (maybe this could help us to find the answer)


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: minerapia on August 23, 2013, 01:05:20 PM
It merely calculates how many share you post in second and adjust the diff so its in desirable range (shares per sec)


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: roy7 on August 23, 2013, 01:19:07 PM
Because the block changed. We he finds one, its a whole new block by then. All his work on that last block will never be given credit.

There is no "partial work" on blocks. Each random hash is either a valid share or not. When you get notified there is a new block, you keep trying hashes to see if they meet the new block instead of the old block. The only time a new block has any effect is if you report a share moment too late and it is 'stale'. But that's true for anyone.

The probability of finding a share is independent of your miner's speed or how many blocks you've looked at previously. A new block notification has no effect on this (unless the difficulty for the new block has changed).

Meanwhile, fast miner got 9 on the last block (or 8, or 10, whatever you like). Even if slow miner finds a block this time, fast miner probably got 8-12 shares again.

If the fast miner is 9x the speed of the slow miner, they should be reporting shares 9x as often as the slow miner. If it takes about 5 shares to find a block, then for every 2 blocks found on average that is 10 total shares, 1 share from the slow miner and 9 shares from the fast miner. So every other block the slow miner didn't report a share during that block. But that doesn't matter. The slow miner is reporting 1/10 of the total shares and will earn 1/10 of the total payout.

Over time, the block changing hurts the slow miner more.

Untrue.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Liquidfire on August 23, 2013, 01:26:28 PM
Looking at a graph of a poisson distribution, it starts low, peaks, then falls off. That's not the right distribution to use here. We need to start high, and fall off. The peak should be the very first sample.

If you are using a difficulty such that you have a 50% chance of getting a share each hash, the distribution will look like so:
1st hash: 50%
2nd hash: 25%
3rd hash: 12.5%
4th hash: 6.25%
and so on, dividing by two each time.

So whatever kind of distribution you call that.

Your example would solve something like, "what is the distribution of many hashes it should take to solve a share given 50% success rate".

What I am using poisson for, is to solve "given a known mean of 60 seconds, what is the distribution of time (in seconds) it will take to solve a share".

So the low, peak, low shape (skewed towards right for infinity) is correct for my use.



I can tell you'd prefer a full simulation instead of a half simulation. Mine is a half simulation since I am just using probability to say how long the block/share time took.

Let me give a full simulation some thought. I initially thought it would be hard to do, but the more I think about it, I think it might be possible to make each worker a thread. I thought there would be a race condition, but I forgot about the global interpreter lock in python. Basically python can't achieve true multi-threading because of the GIL which locks the interpreter to one thread at a time, but in this case, it would  actually be favorable.

Something like calculating how often a worker attempts a hash per second based on hash rate, putting a sleep(x) command in appropriate to that hash rate, and letting them all go to town until someone solves the block.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: roy7 on August 23, 2013, 01:28:33 PM
You'll completely whiff on any given block just as often as you get one share.. meanwhile the other guy gets 6 in. If he has a little bad luck, he gets 5. You have a little bad luck? you get 0. But, it all evens out right? A little bad luck this time, a little good luck next time. Of course every time you have a little good luck, you don't get 2, you'll still get 1. He'll get 7

Miners aren't paid based on some sort of weighted system where it's the # of shares per block you submit. It's the total number of shares you submit vs the total number of shares all miners have submitted.

Say the slow miner's average hash rate is about 1 share every 2 blocks and the fast miner is 9 shares every 2 blocks. Look at the slow miner over a period of 4 blocks. Maybe he submits 0 shares in blocks 1-3, and 2 shares in block 4. Or 2 shares in block 1 and no shares in block 2-4. Or one share in block 1 and one share in block 4, with no shares in blocks 2-3. The end result is after four blocks he's submitted 2 shares.

The fast miner, on the other hand, would average about 18 shares over these 4 blocks. In total after 4 blocks, on average, we'd see 20 total shares. 2 from the slow miner, 18 from the fast miner. And 2/20 of the coins being generated by the pool would be getting paid to the slow miner. Which specific blocks the slow miner submitted the shares on doesn't matter.

If it's more clear, raise the slow miner's average speed to 1 share per block. Whether he gets 1 share each in blocks 1-4, or 4 shares in block 1 and 0 in blocks 2-4, he still submitted 4 shares and is paid 4/total_shares_in_pool. How the shares were spread between the blocks doesn't change this payout.

(I'm assuming something like proportional or PPLNS payout of course. DGM doesn't use a proportional ratio although steady miners over time end up with the exactly fair proportional income based on their relative speeds.)


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: mueslo on August 23, 2013, 02:03:26 PM

If the block changes every 30 seconds on average, and you find a share every 30 seconds on average, and someone else finds a share every 5 seconds, how do you have the same chances?

You cannot precalculate when an event may happen like this, that would violate you having a constant chance of finding a hash, and would make it dependent on the past. If I find a share now, I have the exact same chance of finding it one share later, since you find shares (approx) instantly. Or let me put it this way: assuming you have been unlucky, and got e.g. 0 shares in the time were on average supposed to get two: this does not make it any more likely that you will now find shares in the next timespan. Since you are constantly looking for shares and with the same probability at each point in time, the Poisson process continues even over block changes, so you can't just reset the time.

I explained this in the old thread. If you have an event that has a constant chance of happening in time (e.g. finding a hash, nuclear decay in atoms, ...), the amount of times the event occurs on average in a given timespan (here: between two blocks) is given by the Poisson distribution.
https://i.imgur.com/QCkSaWu.gif
Here is the probability (y axis) that you find N shares (x-axis) between the two blocks if your hashrate is equal to the block find time.

Another analogy. You are gambling with slot machines that are free to use. Every time the block changes you have to change slot machines (but you can do so instantly, we are not simulating latency). By your argument, someone who can pull the lever five times as fast, would get more than five times what you got. Why should that be the case?

I suggest you read up a bit on the Poisson processes (https://www.khanacademy.org/math/probability/random-variables-topic/poisson_process/v/poisson-process-1), I don't think you understand it quite correctly at the moment. Additionally, I'm confident I know what I'm talking about, I study physics. I'm also by no means a fast miner, I have a measly 1 MH/s.



You'll completely whiff on any given block just as often as you get one share.. meanwhile the other guy gets 6 in. If he has a little bad luck, he gets 5. You have a little bad luck? you get 0. But, it all evens out right? A little bad luck this time, a little good luck next time. Of course every time you have a little good luck, you don't get 2, you'll still get 1. He'll get 7

You each have a good block and a bad block. You have 1 share. He has 12.

1/12 is not the same ratio as the ratio as 1/6.

He might get 7, but that's only 17% more than he should have gotten. When you get 2, that's a 100% more than what you should have gotten. Here's the picture accompanying the above, for a miner that has 6x the hashrate of the share-rate-equal-to-block-rate miner:

https://i.imgur.com/tnwjYDO.gif


Now on to your simulation: You are using the wrong distribution. If you don't know why, read my post again from the top, read the wiki page or watch the video I linked above.

The pobability distribution of the times between finding shares (again, this is a poisson process) is simply e^(-t/T), where T is the average time between shares. Which has exactly the property that you can just restart it at all points without changing anything.


Here is the correct version of your code:
Code:
import numpy.random as rnd

class worker():
    sharetime = None #time the next share is found
    def __init__(self,avgsharetime):
        self.avgsharetime = avgsharetime
        self.hashrate = 60/avgsharetime
        self.shares = 0
        self.generatesharetime(0.0)
        
    def generatesharetime(self, currenttime):
        self.sharetime = currenttime + rnd.exponential(scale=self.avgsharetime)

class pool():
    blocktime = None #time the next block is found
    def __init__(self,avgblocktime):
        self.avgblocktime = avgblocktime
        self.generateblocktime(0.0)
    def generateblocktime(self,currenttime):
        self.blocktime = currenttime + rnd.exponential(scale=self.avgblocktime)

pool1 = pool(2)
worker1 = worker(12)
worker2 = worker(1)
duration = 1000.

t=0.
while t<duration:
    if pool1.blocktime<worker1.sharetime and pool1.blocktime<worker2.sharetime:
        t=pool1.blocktime
        print "new block t=",t
        worker1.generatesharetime(t) #if you disable these, nothing changes in the outcome
        worker2.generatesharetime(t) #
        pool1.generateblocktime(t)
        
    elif worker1.sharetime<worker2.sharetime:
        t=worker1.sharetime
        print "worker 1 found a share t=",t
        worker1.shares+=1
        worker1.generatesharetime(t)

    elif worker2.sharetime<worker1.sharetime:
        t=worker2.sharetime  
        print "worker 2 found a share t=",t
        worker2.shares+=1
        worker2.generatesharetime(t)
    else:
        print "this is hugely improbable"


print worker1.shares
print worker2.shares
    
print "Worker 1 has: " + str((float(worker1.hashrate) / float(worker2.hashrate + worker1.hashrate)) * 100) + ' percent of the hash power'
print "But worker 1 has: " + str((float(worker1.shares) / float(worker2.shares + worker1.shares)) * 100) + ' percent of the profit'
#print "Over sample size of " + str(samplesize)
print "When worker1's average share-find-speed was: " + str((float(pool1.avgblocktime) / float(worker1.avgsharetime))) + "x the block time"
 

Example Output:

blocks and shares over time (http://pastebin.com/aDTjSZqa)

Worker 1 has: 7.69230769231 percent of the hash power
But worker 1 has: 8.26645264848 percent of the profit
When worker1's average share-find-speed was: 0.166666666667x the block time


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Liquidfire on August 23, 2013, 02:20:16 PM
Ok. I had a bit of an epiphany - the argument has been over what diff setting is correct this whole time, but now I understand we really should have been arguing over reward system.

Everything I am saying is accurate for a proportional reward system. If we aren't using a proportional reward system, woah, that probably should have been mentioned weeks ago.

What a lot of the arguments are saying would be true if there was one long block with no block changes.

That's the epiphany I had - we should be arguing to go to PPLNS.


The effect I am describing is present when each block has shares associated with it and when the block is found the reward is shared among those blocks. Everything resets for the new block. Proportional reward.

If we've been using something else, than no wonder we are all disagreeing, we are basically speaking different languages.

But PPLNS would eliminate the bias almost completely. The block change would then have no effect on the payout.

I will concede that in a PPLNS system, yes, all the diff does is introduce variance. I could modify my script in 5 minutes to simulate PPLNS and I bet the numbers would look good.



Lets change this conversation. H20, can you verify what reward system you are using now? It's not explicitly mentioned in your FAQ or announcement statement. Now I realize, unless everyone is on the same page on this we are going to spin our wheels forever.

Second, instead of arguing about diff - which would alleviate the problem I am describing but not eliminate it, can we talk about PPLNS or something similar?





Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: mueslo on August 23, 2013, 02:33:10 PM
Ok. I had a bit of an epiphany - the argument has been over what diff setting is correct this whole time, but now I understand we really should have been arguing over reward system.

Everything I am saying is accurate for a proportional reward system. If we aren't using a proportional reward system, woah, that probably should have been mentioned weeks ago.

What a lot of the arguments are saying would be true if there was one long block with no block changes.

That's the epiphany I had - we should be arguing to go to PPLNS.


The effect I am describing is present when each block has shares associated with it and when the block is found the reward is shared among those blocks. Everything resets for the new block. Proportional reward.

If we've been using something else, than no wonder we are all disagreeing, we are basically speaking different languages.

But PPLNS would eliminate the bias almost completely. The block change would then have no effect on the payout.

I will concede that in a PPLNS system, yes, all the diff does is introduce variance. I could modify my script in 5 minutes to simulate PPLNS and I bet the numbers would look good.



Lets change this conversation. H20, can you verify what reward system you are using now? It's not explicitly mentioned in your FAQ or announcement statement. Now I realize, unless everyone is on the same page on this we are going to spin our wheels forever.

Second, instead of arguing about diff - which would alleviate the problem I am describing but not eliminate it, can we talk about PPLNS or something similar?

Nowhere in my previous post am I assuming PPLNS. I am actually assuming straight PPS. Please read it and try to understand it. Or at least look at the python script.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: roy7 on August 23, 2013, 02:34:14 PM
Ah yes, if you are using a proportional payment system which is based solely on the current block, then blocks with no shares you'd get zero and blocks with a share you'd get paid. One could argue that over time it wouldn't matter, if your hash rate is .5 shares per block then the blocks with a share you get paid "double" your average contribution and blocks with no shares you get paid 0. So it averages out to .5 shares/block worth of pay. But only getting paid on blocks where you personally submitted shares kinda violates the spirit of pooled mining, in my opinion.

'Block at a time' proportional payout for a very fast coin where slow miners might not submit shares seems silly. Proportional is a bad payment model anyway, and is hoppable. Best to use DGM or PPLNS if implemented the way Meni Rosefeld says to. Not all pools do PPLNS correctly. Follow this post:

https://bitcointalk.org/index.php?topic=39832

To do it right. :)

DGM is what I use on my pool and I really like it. (I do not use a negative f value, so there's no risk to the pool at all. PPS is also bad since it virtually guarantees pool bankruptcy if ran long enough.)


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Liquidfire on August 23, 2013, 02:37:39 PM

You cannot precalculate when an event may happen like this, that would violate you having a constant chance of finding a hash, and would make it dependent on the past.

Thank you for the detailed, thorough response. Give me a chance to digest everything you said. You are clearly more advanced in mathematics and statistics than I (i mean that genuinely not sarcastically) and I would love to learn from you. However, in the mean time, let me ask you this.

If you can't precalculate (you say precalculate, I say predict) the time is takes to solve a block, how is bitcoin (or whatever coin) doing it? Bitcoin itself has to predict the time it will take the network to solve a block, to know how to adjust the difficulty. It MUST maintain an average block find time of 10 minutes, or quite literally the entire thing will come crashing to a screeching halt.

If bitcoin couldn't accurately predict the time it takes to solve a block given X hashpower, we wouldn't be having this conversation because bitcoin would have died, and none of the alt coins that we love to mine at middlecoin would have ever existed.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Liquidfire on August 23, 2013, 02:40:11 PM
Nowhere in my previous post am I assuming PPLNS. I am actually assuming straight PPS. Please read it and try to understand it. Or at least look at the python script.

You were assuming straight PPS, I was assuming per block proportional. Do you see? We will never convince each other because we are both right, in the respective systems we think we are operating under.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: roy7 on August 23, 2013, 02:42:43 PM
If you can't precalculate (you say precalculate, I say predict) the time is takes to solve a block, how is bitcoin (or whatever coin) doing it? Bitcoin itself has to predict the time it will take the network to solve a block, to know how to adjust the difficulty. It MUST maintain an average block find time of 10 minutes, or quite literally the entire thing will come crashing to a screeching halt.

Bitcoin adjusts difficulty level based on historical data only, no predictions. It compares the time it took to get the last X blocks to what it "should" have taken, and then adjusts the difficulty accordingly to try and be closer to the target speed next time. Which would be very accurate if hash rate was constant.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: mueslo on August 23, 2013, 02:45:24 PM
Nowhere in my previous post am I assuming PPLNS. I am actually assuming straight PPS. Please read it and try to understand it. Or at least look at the python script.

You were assuming straight PPS, I was assuming per block proportional. Do you see? We will never convince each other because we are both right, in the respective systems we think we are operating under.

Sorry, but you are wrong. You were not simulating per block proportional, you were assuming a wrong distribution for the time between shares.


Thank you for the detailed, thorough response. Give me a chance to digest everything you said. You are clearly more advanced in mathematics and statistics than I (i mean that genuinely not sarcastically) and I would love to learn from you. However, in the mean time, let me ask you this.

If you can't precalculate (you say precalculate, I say predict) the time is takes to solve a block, how is bitcoin (or whatever coin) doing it? Bitcoin itself has to predict the time it will take the network to solve a block, to know how to adjust the difficulty. It MUST maintain an average block find time of 10 minutes, or quite literally the entire thing will come crashing to a screeching halt.

If bitcoin couldn't accurately predict the time it takes to solve a block given X hashpower, we wouldn't be having this conversation because bitcoin would have died, and none of the alt coins that we love to mine at middlecoin would have ever existed.

I elaborated a bit at the end, because that's what I thought was most likely to be read:

Now on to your simulation: You are using the wrong distribution. If you don't know why, read my post again from the top, read the wiki page or watch the video I linked above.

The probability distribution of the times between finding shares (again, this is a poisson process) is simply e^(-t/T), where T is the average time between shares. Which has exactly the property that you can just restart it at all points without changing anything.

edit to clarify: the chance P of the time you need to solve a block is P(t) = e^(-t/T). For generating t, you can for example use numpy.random.exponential(scale=T)


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Liquidfire on August 23, 2013, 02:56:06 PM
Nowhere in my previous post am I assuming PPLNS. I am actually assuming straight PPS. Please read it and try to understand it. Or at least look at the python script.

You were assuming straight PPS, I was assuming per block proportional. Do you see? We will never convince each other because we are both right, in the respective systems we think we are operating under.

Sorry, but you are wrong. You were not simulating per block proportional, you were assuming a wrong distribution for the time between shares.

Its not Miner A vs. the Pool and Miner B vs. the pool

Its Miner A vs. Miner B.

The distribution doesn't matter very much, because miner A and miner B both receive it the same system. I arbitrarily picked the averages out of the thin air. I can do that, because on the pool at any given time we can find miners all over the scale. That's why I can say miner A has this distribution, because no matter what, I can find a miner with that average.

To put it another way, its pointless to argue over weather miner A with average X would really distribute that way. Even if you are right, there is another miner with average Y who WOULD, and I can pick any of them since the competition is between the two miners, not miner vs. pool.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: mueslo on August 23, 2013, 03:00:37 PM
Its not Miner A vs. the Pool and Miner B vs. the pool

Its Miner A vs. Miner B.

The distribution doesn't matter very much, because miner A and miner B both receive it the same system. I arbitrarily picked the averages out of the thin air. I can do that, because on the pool at any given time we can find miners all over the scale. That's why I can say miner A has this distribution, because no matter what, I can find a miner with that average.

To put it another way, its pointless to argue over weather miner A with average X would really distribute that way. Even if you are right, there is another miner with average Y who WOULD, and I can pick any of them since the competition is between the two miners, not miner vs. pool.

If the distribution is not that of a Poisson process, all your results will be wrong. Otherwise you are not simulating a Poisson process (which hashing is). If you like, I can change the script I wrote to a per block reward system, I'm sure it would come out the same. If not, I concede.

Which reward scheme is middlecoin using anyway? I have no idea. But I don't think it's per block reward.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Liquidfire on August 23, 2013, 03:11:40 PM
If you like, I can change the script I wrote to a per block reward system, I'm sure it would come out the same. If not, I concede.

I am curious what will happen if you implement the following conditions:

At the end of the block, worker shares get reset to 0.
Just before that happens, the shares are used to calculate payout, and the payout occurs per block.










Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: mueslo on August 23, 2013, 03:20:36 PM
I am curious what will happen if you implement the following conditions:

At the end of the block, worker shares get reset to 0.
Just before that happens, the shares are used to calculate payout, and the payout occurs per block.

At the end of every block? That wouldn't make much sense though (there is a big chance of loss for the pool operator if the network suddenly becomes fast).

What makes more sense: when a block is found by the pool itself (and not the whole network), all previous shares are reset to zero and the reward of that block is distributed among shares.  I'm guessing you want the latter?

(I don't think we're using that either, because it's vulnerable to pool hopping if you know when the pool found a block.)

But anyway, if you want to actually simulate this, you can no longer calculate the time when a block is found, because then it could happen that the pool finds a block without any share actually having been submitted. You now have to actually calculate the odds of one of the miners in the pool having found the block.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Liquidfire on August 23, 2013, 04:50:06 PM
What makes more sense: when a block is found by the pool itself (and not the whole network), all previous shares are reset to zero and the reward of that block is distributed among shares.  I'm guessing you want the latter?

The fact is I really don't know how it works right now. But, when I am talking about block change I mean block change for the network. I am starting to realize we are all working under a lot of personal assumptions about this because we don't know for sure what the reward system is. My assumption, and the one I've been arguing from (I hope not for nothing, but at least it would be settled), is that we get paid per block per shares that we have for that block.

Quote
But anyway, if you want to actually simulate this, you can no longer calculate the time when a block is found, because then it could happen that the pool finds a block without any share actually having been submitted


This could happen, yes (very very unlikely but possible). But, you can simulate it.  That could happen in my simulation and I believe it would handle it correctly.

If my reward is set to 100, and neither worker got a share in time , it would simply pay 0/100 (0) of the reward to each.


We can continue to argue weather my system is correct in a proportional-per-block scenario, but we still aren't sure if that is the case anymore, so I'm just holding out to hear the scoop on that.

Part of me hopes I was dead wrong and H20 is using some other reward system so we can finally put it to bed.


At the end of every block? That wouldn't make much sense though (there is a big chance of loss for the pool operator if the network suddenly becomes fast).


It shouldn't effect the pool as a whole. The pool as a whole would still solve just as many blocks, it should only affect the shares within the pool.
 


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: mueslo on August 23, 2013, 04:57:47 PM
[...]

Here's the code for per pool-found-block/proportional reward. There's no change (as is to be expected (https://en.bitcoin.it/wiki/Mining_pool_reward_FAQ#What_reward_systems_are_there.3F), all proportional reward does is increase variance (as long as there's no pool hopping)):

Code:
import numpy.random as rnd
import numpy as np

class worker():
    sharetime = None #time the next share is found by the worker
    def __init__(self,avgsharetime):
        self.avgsharetime = avgsharetime
        self.hashrate = 60/avgsharetime
        self.shares = 0
        self.roundshares = 0
        self.income = 0.
        self.generateShareTime(0.0)
    
    def generateShareTime(self, currenttime):
        self.sharetime = currenttime + rnd.exponential(scale=self.avgsharetime)
    
    def foundShare(self, currenttime):
        self.shares+=1
        self.roundshares+=1
        self.generateShareTime(currenttime)


class pool():
    t = 0. #time elapsed
    blocktime = None #time the next block is found (by the network)
    pblock = 0.02 #probability that a share solves the block
    totalroundshares = 0 #number of total shares solved for this block round
    workers = [] #list of workers in the pool
    def __init__(self,avgnetworkblocktime=10.,workers=[],):
        self.avgnetworkblocktime = avgnetworkblocktime
        self.generateNetworkBlockTime(0.0)
        self.workers=workers
        
    def start(self,duration=100.):
        while self.t<duration:
            nextworker = self.workers[np.argmin([w.sharetime for w in self.workers])]
            nextsharetime = nextworker.sharetime            
            
            if self.blocktime<nextsharetime: #network block found
                self.newNetworkBlock()
            
            else:                            #worker found a share
                self.t = nextworker.sharetime
                self.totalroundshares+=1
                nextworker.foundShare(self.t)

                
                if rnd.uniform(0,1)<self.pblock: #share solved the block
                    self.newPoolBlock()


        
    def generateNetworkBlockTime(self,currenttime):
        self.blocktime = currenttime + rnd.exponential(scale=self.avgnetworkblocktime)
        
    def distributeRewards(self):
        print "Shares: [",
        for w in self.workers:
            print w.roundshares,
            w.income += float(w.roundshares)/float(self.totalroundshares)
            w.roundshares = 0
        self.totalroundshares = 0
        print "]\n"
        
    def newPoolBlock(self):
        print "\nnew pool block t=",self.t,"-> Payout!"
        self.distributeRewards()
        
        self.blocktime = self.t
        self.newNetworkBlock(echo=False)
        
    def newNetworkBlock(self,echo=True):
        self.t=self.blocktime
        if echo: print "new network block t=",self.t
        for w in self.workers: #if you disable these, nothing changes in the outcome
            w.generateShareTime(self.t)
        
        self.generateNetworkBlockTime(self.t)
        


slowworker = worker(avgsharetime=30)
mediumworker = worker(avgsharetime=10)
fastworker = worker(avgsharetime=3)


pool1 = pool(workers=[fastworker, mediumworker, slowworker],avgnetworkblocktime=10.)
pool1.start(duration=100000.)

print fastworker.shares
print slowworker.shares
    
print "Slow worker has: " + str((float(slowworker.hashrate) / float(fastworker.hashrate)) * 100) + ' percent of the hash power of fast worker'
print "Slow worker has: " + str((float(slowworker.income) / float(fastworker.income)) * 100) + ' percent of the profit of fast worker'
print "Slow worker has: " + str((float(slowworker.shares) / float(fastworker.shares)) * 100) + ' percent of the total shares of fast worker'

And some example outputs:
Slow worker has: 10.0 percent of the hash power of fast worker
Slow worker has: 10.0643834344 percent of the profit of fast worker
Slow worker has: 10.2436090226 percent of the total shares of fast worker

Slow worker has: 10.0 percent of the hash power of fast worker
Slow worker has: 9.89355458187 percent of the profit of fast worker
Slow worker has: 9.99728580476 percent of the total shares of fast worker

Slow worker has: 10.0 percent of the hash power of fast worker
Slow worker has: 10.2071470481 percent of the profit of fast worker
Slow worker has: 10.1000090098 percent of the total shares of fast worker

Slow worker has: 10.0 percent of the hash power of fast worker
Slow worker has: 9.78155423167 percent of the profit of fast worker
Slow worker has: 10.1675711134 percent of the total shares of fast worker

Slow worker has: 10.0 percent of the hash power of fast worker
Slow worker has: 9.84810455803 percent of the profit of fast worker
Slow worker has: 9.8902409926 percent of the total shares of fast worker

example found blocks (http://pastebin.com/CKNZNPQR)


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: juhakall on August 23, 2013, 05:03:39 PM
It's proportional

https://bitcointalk.org/index.php?topic=259649.msg2900385#msg2900385

Glad to see this discussion is actually progressing. The lack of personal remarks in this thread is almost inexplicable.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: roy7 on August 23, 2013, 06:10:16 PM
Well done mueslo. You have more patience than I do. :)


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: cverity on August 23, 2013, 07:24:37 PM
Quote
There is one provision with that statement, it takes a relatively long time to average out the larger variance the small miners suffer.
This.

Quote
The drawback is that it also increases the day-to-day variance.  In the long-run, it doesn't matter, though.
This.

Quote
Just with a higher variance.
This.

Variance is what I've been mentioning in the original thread for days.  High diff will not affect pool's profit, but it will affect small miner's reasonable variance.

h2o has previously stated that we are using a proportional system.  I agree that changing the payout algorithm to something like PPLNS would also help smooth out the variance.

Right now we have the worst possible setup for small miners - high diff with a payout calculation that does nothing to smooth out variance.

Over a long enough period of time (although I have no idea if that's weeks, months, etc) it shouldn't matter. But isn't the whole point of a pool to minimize variance?


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: Liquidfire on August 23, 2013, 07:49:46 PM
I am sure this will make some of you happy...

Mueslo - I finally just took your numpy.random.exponential and replaced my poisson function with it, in my own simulation. When I did, numbers started coming out looking like they should (actual return close to expected). This backed your claim that my distribution was wrong. As I started generating plots of each distribution, things started to make sense.

As hard as I've been dug in on this, most people would just bug out, but it wouldn't be right for me to not tell you that you were right on the distribution. H2O tried to explain this to me too, and a couple others.

My mental roadblock (or rather what cleared it) was the fact that even in a lopsided distribution you can still have an average, but it will be very skewed toward the thick end of the plot. I needed to actually see it plotted before it clicked.

But, my simulation was still correct (yours was too), less the distribution, so I guess I learned I'm better at programming than math :p

Anyway, thanks for actually trying to explain things to me and not being an ass like a couple others. I learned quite a bit researching all of this.


For the record:

import random
import numpy as np
import numpy.random as rnd

class worker():
    def __init__(self,hashrate):
        self.hashrate = hashrate
        self.sharesolvetime = 60 / hashrate
        self.shares = 0

class pool():
    def __init__(self,blockfindtime):
        self.blockfindtime = blockfindtime

pool1 = pool(30)
worker1 = worker(1)
worker2 = worker(12)
samplesize = 1000000

for n in range(0,samplesize):
    clock = rnd.exponential(scale=pool1.blockfindtime)
    clock1 = clock
    while clock1 > 0:
        sharesolve = rnd.exponential(scale=worker1.sharesolvetime)
        if sharesolve > clock1:
            break
        else:
            worker1.shares = worker1.shares + 1
            clock1 = clock1 - sharesolve
    clock2 = clock
    while clock2 > 0:
        sharesolve = rnd.exponential(scale=worker2.sharesolvetime)
        if sharesolve > clock2:
            break
        else:
            worker2.shares = worker2.shares + 1
            clock2 = clock2 - sharesolve
    
print "Worker 1 has: " + str((float(worker1.hashrate) / float(worker2.hashrate + worker1.hashrate)) * 100) + ' percent of the hash power'
print "But worker 1 has: " + str((float(worker1.shares) / float(worker2.shares + worker1.shares)) * 100) + ' percent of the profit'
print "Over sample size of " + str(samplesize)
print "When worker1's average share-find-speed was: " + str((float(pool1.blockfindtime) / float(worker1.sharesolvetime))) + 'X the block-find-speed'
    




Very Slow

Worker 1 has: 7.69230769231 percent of the hash power
But worker 1 has: 7.69363865886 percent of the profit
Over sample size of 1000000
When worker1's average share-find-speed was: 8.33333333333X the block-find-speed

Slow

Worker 1 has: 7.69230769231 percent of the hash power
But worker 1 has: 7.6898534448 percent of the profit
Over sample size of 1000000
When worker1's average share-find-speed was: 4.0X the block-find-speed

Medium

Worker 1 has: 7.69230769231 percent of the hash power
But worker 1 has: 7.68459728742 percent of the profit
Over sample size of 1000000
When worker1's average share-find-speed was: 2.0X the block-find-speed

Fast

Worker 1 has: 7.69230769231 percent of the hash power
But worker 1 has: 7.68286758249 percent of the profit
Over sample size of 1000000
When worker1's average share-find-speed was: 1.0X the block-find-speed

Very Fast

Worker 1 has: 7.69230769231 percent of the hash power
But worker 1 has: 7.67015222587 percent of the profit
Over sample size of 1000000
When worker1's average share-find-speed was: 0.5X the block-find-speed


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: mueslo on August 23, 2013, 08:24:56 PM
My mental roadblock (or rather what cleared it) was the fact that even in a lopsided distribution you can still have an average, but it will be very skewed toward the thick end of the plot. I needed to actually see it plotted before it clicked.

This is why I stuck through, I thought I might help someone understand it better :) Thank you very much for the follow up. I actually learned something too (at first I thought it is always wrong to calculate the times in advance, which, of course, given the correct distribution in times between finding a share, it isn't).

Although there is still one phenomenon I cannot explain. I have a HD 7950 and a HD 5830. The HD 7950 consistently gets 2-5% rejects, while the HD 5830 gets 6-12% rejects, even if I set clock and intensity on the HD 7950 such that the hashrates are equal, so it isn't dependent on hash rates.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: FlungSpun on August 23, 2013, 08:33:36 PM
that must be something to do with getting the share off the card and onto the pool no?
though that is quite a difference. assuming no HW error type issues, same machine,  switch, router etc


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: mueslo on August 23, 2013, 08:40:53 PM
Yeah, they're both in the same machine, the only difference apart from the actual cards is which PCIe slot they're in, obviously. I do occasionally get HW errors on the HD 5830 (about 1 every 2-3 days), but I don't see the connection.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: cverity on August 23, 2013, 08:59:33 PM
Yeah, they're both in the same machine, the only difference apart from the actual cards is which PCIe slot they're in, obviously. I do occasionally get HW errors on the HD 5830 (about 1 every 2-3 days), but I don't see the connection.

Strange, it's hard to imagine that any minor difference between the cards design would affect its ability to get the share from the card to the pool that significantly.

Out of curiosity, is the 5830 also serving your monitor?


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: mueslo on August 23, 2013, 09:37:33 PM
Strange, it's hard to imagine that any minor difference between the cards design would affect its ability to get the share from the card to the pool that significantly.

Out of curiosity, is the 5830 also serving your monitor?

Nope, that's the 7950's job.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: h2odysee on August 23, 2013, 09:43:08 PM
Although there is still one phenomenon I cannot explain. I have a HD 7950 and a HD 5830. The HD 7950 consistently gets 2-5% rejects, while the HD 5830 gets 6-12% rejects, even if I set clock and intensity on the HD 7950 such that the hashrates are equal, so it isn't dependent on hash rates.

I'm guessing that older cards have more latency. So to predict your rejection rate, you'd need to add up the ping to middlecoin.com, plus whatever latency is associated with your GPU.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: h2odysee on August 23, 2013, 09:49:34 PM
Thanks for doing that, liquidfire.

I should really learn numpy, and scipy. They look fun to use. I could make use of them, to make my selling bot better.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: mueslo on August 23, 2013, 09:52:47 PM
Thanks for doing that, liquidfire.

I should really learn numpy, and scipy. They look fun to use. I could make use of them, to make my selling bot better.

It really is useful.  This is what I learned Python/Numpy/Scipy/Matplotlib with.: http://www.math.ethz.ch/education/bachelor/lectures/fs2012/other/nm_pc/NumPhys_handout1.pdf


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: cverity on August 23, 2013, 10:42:44 PM
Although there is still one phenomenon I cannot explain. I have a HD 7950 and a HD 5830. The HD 7950 consistently gets 2-5% rejects, while the HD 5830 gets 6-12% rejects, even if I set clock and intensity on the HD 7950 such that the hashrates are equal, so it isn't dependent on hash rates.

I'm guessing that older cards have more latency. So to predict your rejection rate, you'd need to add up the ping to middlecoin.com, plus whatever latency is associated with your GPU.

That does seem to be the most reasonable conclusion. It's hard to imagine that the latency between the GPU and CGMiner is measurable compared to internet latency, but I'm no hardware expert.

[Edit] If that's true, then you'd think everyone running a 5830 would get similar results. Anyone else with an older card experiencing the same thing compared to a newer card?


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: FlungSpun on August 23, 2013, 11:15:16 PM
could be due to a mixture of cards forcing inefficient driver performance?
perhaps there is a software bottleneck
I seem to remember some chatter about cards of different generations not playing well together but didn't pay much heed as I'm all 7950s & 70s
and 1 6950 on its own with the bios flashed

If some mixtures don't work maybe some are a bit iffy ....


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: CoinBuzz on August 24, 2013, 07:38:58 AM
why cant we test it in a real pool?

we can set high difficulty (512) for about a week, then switch to lower difficulty and compare the results with each other.

That would be a real examination about this topic.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: chup on August 24, 2013, 05:21:51 PM
Not possible running a trial. Middlecoin creator said that 512 difficulty is hardcoded. Is higher difficulty favourising faster miners over slower? Answer is on the middlecoin.com main page. Just calculate rejected over accepted MH/s as a percentage and it will come that percentage is raising going down the table...  :-\


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: mueslo on August 24, 2013, 10:45:22 PM
Just calculate rejected over accepted MH/s as a percentage and it will come that percentage is raising going down the table...  :-\

Here, I made a little something. This should finally clear up misconceptions.

http://mueslo.de/host/middlecoin/miners.png

Regenerates every 5 minutes.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: FlungSpun on August 25, 2013, 12:11:12 AM
So there is a little variance wobble for the small hash rates, seems to be around the same average as the rest of the miners ... well what do you know eh

if you animate those pngs you can see the variance dance.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: FlungSpun on August 25, 2013, 12:28:10 AM
Answer is on the middlecoin.com main page. Just calculate rejected over accepted MH/s as a percentage and it will come that percentage is raising going down the table...  :-\


No it doesn't , no it isn't.


if you plot a moving average against that data there are some small hash rate miners around 200th on the list getting better rejection averages than in the top 50. its dynamic and mueslos
graph shows what going on nicely.



Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: chup on August 25, 2013, 07:18:09 PM
On the png I can see miners (+s) forming nice curves that are exponentialy rising with lower hashrate. At least now 21:05 25-th of August.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: chup on August 25, 2013, 07:37:12 PM
if you plot a moving average against that data there are some small hash rate miners around 200th on the list getting better rejection averages than in the top 50. its dynamic and mueslos
graph shows what going on nicely.


The lowest hash rate miners has some total irrelevant data shown that are distorted by quantisation. Please, to see how 512 and 16 difficulty is different for small miners, start a small cpu/cuda miner with 10KH/s, mine for a day at middlecoin with 512 difficulty, and the second day at pool2.us.multipool.us with 16 difficulty. First day you will see 99% lines with "stratum detected new block", 0,5-0,75% lines with "yay" and 0,25-0,5% with "booo". Second day will be different story.
Please note that my post is not in a way of flaming or making someone's hard work look worse, but in a way of making things better, if possible, because I like the middlecoin idea.


Title: Re: Does a high pool difficulty lower anyone's profits?
Post by: mueslo on August 25, 2013, 08:13:54 PM
On the png I can see miners (+s) forming nice curves that are exponentialy rising with lower hashrate. At least now 21:05 25-th of August.


Yep, but it jumps around a lot there due to there being very few miners there. On http://mueslo.de/host/middlecoin/ you can view the last hour.