Neighbourhood Pool Watch

[Meni Rosenfeld]

@kano I suggest you look up the word "Large". Since you clearly have no interest in carrying out a reasonable discussion and would rather play dumb and ignore what your interlocutor says, I'll refrain from further responding to you.

For reference, here is my previous correspondence with kano about this issue.


I'll clarify one thing. In operator-risky methods like PPS, geometric method, DGM, the operator can either gain or lose on a round. These movements are in the operator's personal funds. Just like a riskless pool (like PPLNS) can collect fees for the service, so can a risky pool collect high or negative fees on a per-round basis. The variability in the operator's income is a risk he is willing to take in order to make the pool more attractive.

For his own personal accounting, the operator would do well to set aside a portion of his funds as the pool's "reserve". This makes sure that technically the funds have a place to come from, and allows a better handle on profits or loss. He could share information on his reserve if he wants. But this is still his own funds, which he can take away or add to as he pleases. The current reserve has no effect on the attractiveness, in terms of expected reward, variance and maturity time, of mining in the pool.

If the reserves run out, it is the operator's choice whether to allocate more of his funds to the pool, or to shut the pool down. Other than having to look for a new pool (which is not to be taken lightly), this has no ill effects on the miners - the debt to them, expressed as unrealized score, will be cashed out immediately based on the expected reward for it. Since the total possible debt is bounded (the bound can be controlled with parameters), the operator can simply set aside the necessary amount as an "emergency fund". There is no risk of growing an unbounded debt and then not being able to repay.

DGM settles good and bad luck on time and vents it as miner reward variance (or in the worst case, only likely with aggressive parameters, a clean bankruptcy). This allows it to continue in a sustainable steady state. SMPPS defers having to deal with luck indefinitely, until one day the debt grows so large it can no longer continue.

[Inaba]

If the pool stays in a large negative buffer long enough, it is a likely that all miners will leave, and the pool will fail.

Which we ALREADY know to be FALSE.

Wait... what?  How on earth do you come to this conclusion?  Arsbitcoin is a prime example of why it's true.  The buffer is negative, it's never going to be repaid, the miners who are owed money are screwed.  The few "hangers on" that are still around are the same people who buy lottery tickets, except in this case, they are hoping to win the lottery for the money they already earned, not extra income.

Meni loves to say that "there is no buffer" so I take that to mean that when I look at a DGM pool that is paying more than 50BTC per block in a bad luck streak, that pool is in fact in serious trouble and about to fail?
Since there is no buffer, how can they pay more than 50BTC per block?

Meni's comments aside, speaking from experience, I can tell you there is no meaningful "buffer."  As Meni has already explained, it comes out of the operators pocket - in my case, it comes out of my pocket. I refill my pocket on short blocks and pay it out on long blocks.  The total "buffer" that ever gets into the negative is no more than a few BTC at the worst of times.  If you consider 3 or 4 BTC to be a lot/large, well then I stand corrected, but to me, 1200 BTC is tending towards LARGE and 5 BTC is tending towards SMALL, especially when speaking of a pool buffer.

Those DGM pools paying substantially more than 50 BTC per block are doing so out of the operators pockets, not out of any "buffer."  Each round, you are paid what you earned, no buffering required.  I could close up EMC's shop today and cash everyone out for full payment and be right about even - specifically because I don't carry an arbitrary buffer.

[organofcorti]

Anyway, as long as I'm here:

You talk about the maturity time, but it should be emphasized that the value I'm analyzing is the time of average payback, not the time of being paid in full. For example, if you get 50% of the debt back in 1 round and the other 50% in 3 rounds, your maturity time is 2 rounds.

I did gloss over that. I'll fix it tomoorow. Thanks for pointing it out.

The second chart, giving the PDF of the pool debt after n rounds, looks completely wrong, I have no idea how you got that. The pool debt, assuming constant difficulty, is (X/D - n)*B, where X is the number of shares it took to get n blocks, which is distributed negative binomial. So it's really a shifted negative binomial, which doesn't have the cusp we see.

I'll have to get back to you on that. I wrote that part a while ago, and I'll have to find my notes. It was strange, I puzzled over it for ages, but I did confirm it with a simulation. I'll post it to pastebin when I have a chance to tidy the code up (it's in R) so you can try it for yourself.

Thanks for your interest, Meni

[organofcorti]

The whole discussion about SMPPS is based on:

The assumption that this is true:

If the pool stays in a large negative buffer long enough, it is a likely that all miners will leave, and the pool will fail.

Which we ALREADY know to be FALSE.

You're arguing about the the conclusion which I posted here without reading the materiel upon which it was based. Otherwise you'd try to prove your point instead of presenting it as a magical fait accompli.

Please read the actual post, Kano. I think you mean well but if you aren't reading the post then I can't argue with you about it. You just come off seeming like a clever chatbot that repeats itself.

[kano]

The whole discussion about SMPPS is based on:

The assumption that this is true:

If the pool stays in a large negative buffer long enough, it is a likely that all miners will leave, and the pool will fail.

Which we ALREADY know to be FALSE.

You're arguing about the the conclusion which I posted here without reading the materiel upon which it was based. Otherwise you'd try to prove your point instead of presenting it as a magical fait accompli.

Please read the actual post, Kano. I think you mean well but if you aren't reading the post then I can't argue with you about it. You just come off seeming like a clever chatbot that repeats itself.

I've read the actual post twice already, ok 3 times now

The rest is based on the concept that over infinite time, during that infinite time, the value of the buffer can be any size - negative or positive - but taking that point to a specific case.

Which of course is true.
(though you didn't seem to bother to point out that it can be both positive and negative in the same way )

That is of course true of DGM also.
The DGM argument is that the pool will shutdown before that happens and the pool does not need to disclose the buffer.
I see that as a conflicting statement in itself.

Again you also make the assumption at the end

A strategic miner will only mine at an SMPPS pool when it has a positive buffer.

Which is in the least questionable as to it's relevance ...

[kano]

If the pool stays in a large negative buffer long enough, it is a likely that all miners will leave, and the pool will fail.

Which we ALREADY know to be FALSE.

Wait... what?  How on earth do you come to this conclusion?  Arsbitcoin is a prime example of why it's true.  The buffer is negative, it's never going to be repaid, the miners who are owed money are screwed.  The few "hangers on" that are still around are the same people who buy lottery tickets, except in this case, they are hoping to win the lottery for the money they already earned, not extra income.

Doesn't seem that way.
People keep mining there, the issue seems to be that BT is not maintaining the pool, not that the buffer is negative.
More people were mining on Ars with an approx -1000 buffer than were mining on EMC at some points.
The pool stopped so that's when mining stopped, not when the buffer got to -1000

Meni loves to say that "there is no buffer" so I take that to mean that when I look at a DGM pool that is paying more than 50BTC per block in a bad luck streak, that pool is in fact in serious trouble and about to fail?
Since there is no buffer, how can they pay more than 50BTC per block?

Meni's comments aside, speaking from experience, I can tell you there is no meaningful "buffer."  As Meni has already explained, it comes out of the operators pocket - in my case, it comes out of my pocket. I refill my pocket on short blocks and pay it out on long blocks.  The total "buffer" that ever gets into the negative is no more than a few BTC at the worst of times.  If you consider 3 or 4 BTC to be a lot/large, well then I stand corrected, but to me, 1200 BTC is tending towards LARGE and 5 BTC is tending towards SMALL, especially when speaking of a pool buffer.

Those DGM pools paying substantially more than 50 BTC per block are doing so out of the operators pockets, not out of any "buffer."  Each round, you are paid what you earned, no buffering required.  I could close up EMC's shop today and cash everyone out for full payment and be right about even - specifically because I don't carry an arbitrary buffer.

So what are the grounds when you would close EMC?
A few weeks back DeepBit had 26 blocks where the average shares per block (for 26 blocks) was ~200%
What debt for you would that represent with your EMC numbers?



Aside: there is also the argument about delaying payout due to a negative buffer, but on DGM full payout is always delayed, on SMPPS payout is only delayed when the buffer is negative.

[Inaba]

Full payout is never delayed, unless you count the "run-up" period (on EMC, it's 7 blocks), which is paid out on the run-down period (again, another approximately 7 blocks)... otherwise, you are paid in full each block.

As far as what grounds would I close EMC if the buffer were too negative?  Is that the question?  I would say -100 BTC, which would mean we would need a run of 200%+ blocks for over 200 blocks to even get vaguely close to -100BTC.  I just pull -100BTc out of my ass, BTW, just to prove a point.  I probably wouldn't even close it then, because I can cover that out of my pocket.  Although, if we did end up with a 200 block unlucky streak I would seriously be considering closing up shop, but not because of a negative buffer - I would pay whatever was negative out of my pocket to close up shop immediately on the 200th unlucky block.  Heck, if anyone was still even mining after the first 100 unlucky blocks I would be absolutely shocked.

My point is, it's virtually impossible with the proper parameters to have a negative buffer that has meaning... when we are talking about single digit BTC values, it's really meaningless in a pool context.

As far as people that keep mining on Ars is evidence that they don't understand the system.  Who understands how SMPPS works and is still mining there?  I'd like someone to speak up as to why you'd do such a thing?  There is nothing I can think of that would compel me to mine on a pool that has a negative buffer, where the best I can expect is to eventually, maybe, get what I am owed.  It makes absolutely no sense.

[cablepair]

top TRUST WORTHY Pools
(not in any kind of order)

Slush
BTCGuild
Eclipse
Eligius
Ozcoin
P2pPool

if you voted that you do not trust any of these pools you are just plain clueless.

[Graet]

top TRUST WORTHY Pools
(not in any kind of order)

Slush
BTCGuild
Eclipse
Eligius

if you voted that you do not trust any of these pools you are just plain clueless.

I'm lost
p2Pool is the clear winner, followed by EMC and Ozcoin. Well done to all in the top three pools - you've gained significantly more trust in the community than others. from https://bitcointalk.org/index.php?topic=66026.msg806445#msg806445
or are you referring to the current "untrusted" poll?

[cablepair]

top TRUST WORTHY Pools
(not in any kind of order)

Slush
BTCGuild
Eclipse
Eligius
Ozcoin

if you voted that you do not trust any of these pools you are just plain clueless.

I'm lost
p2Pool is the clear winner, followed by EMC and Ozcoin. Well done to all in the top three pools - you've gained significantly more trust in the community than others. from https://bitcointalk.org/index.php?topic=66026.msg806445#msg806445
or are you referring to the current "untrusted" poll?

let me edit

this is MY OPINION - not result of any poll
(sure no one cares but here you go anyway)

and yes P2POOL goes without saying - but its not really a pool is it

ozcoin is trustworthy also graet

I just dont understand why some of these people voted - does not make sense to me so I had to share my opinion

I mean seriously 15% think Slush is not trustworthy? Come on...

this poll is FUBAR
 

[kano]

Full payout is never delayed, unless you count the "run-up" period (on EMC, it's 7 blocks), which is paid out on the run-down period (again, another approximately 7 blocks)... otherwise, you are paid in full each block.

Yes of course that is what I am referring to
Whatever that number of lead in blocks is, you will never be paid in full for those blocks until either after you stop mining on the pool, or if the pool closes down and pays out the debt in full.

As far as what grounds would I close EMC if the buffer were too negative?  Is that the question?  I would say -100 BTC, which would mean we would need a run of 200%+ blocks for over 200 blocks to even get vaguely close to -100BTC.  I just pull -100BTc out of my ass, BTW, just to prove a point.  I probably wouldn't even close it then, because I can cover that out of my pocket.  Although, if we did end up with a 200 block unlucky streak I would seriously be considering closing up shop, but not because of a negative buffer - I would pay whatever was negative out of my pocket to close up shop immediately on the 200th unlucky block.  Heck, if anyone was still even mining after the first 100 unlucky blocks I would be absolutely shocked.

My point is, it's virtually impossible with the proper parameters to have a negative buffer that has meaning... when we are talking about single digit BTC values, it's really meaningless in a pool context.

As far as people that keep mining on Ars is evidence that they don't understand the system.  Who understands how SMPPS works and is still mining there?  I'd like someone to speak up as to why you'd do such a thing?  There is nothing I can think of that would compel me to mine on a pool that has a negative buffer, where the best I can expect is to eventually, maybe, get what I am owed.  It makes absolutely no sense.

I'm just curious about how DMG works in reality rather than reading Meni's comments that certainly have an element of self bias in them

I mine on Ozcoin that is DGM, I'm not saying "throw it out, it's no good", however, but making comparisons that are dubious to promote DGM is not acceptable either.

Edit: though I could point out something else that everyone seems to be ignoring:
If someone mines on Ars and has an unpaid debt, they may consider it in their own interest to keep mining ...

Edit2: and if they had been paying 1% into the buffer to cover orphans, what would their balance be now

[eleuthria]

I just dont understand why some of these people voted - does not make sense to me so I had to share my opinion

I mean seriously 15% think Slush is not trustworthy? Come on...

this poll is FUBAR

To be fair, any poll in the Pools forum is pretty much just a test of how rabid the users of pools are.  The polls on this thread have been popularity contests (or how loud the haters were), rather than their trustworthiness (or lack thereof).  Even polls specifically for a pool on their own thread tend to give pretty awkward results that don't give an accurate representation of the overall pool users.

Examples:

• Slush has 15 votes.  Not the largest number of votes, but very surprising.  Slush is the ORIGINAL mining pool.  The only bad news I've ever seen on Slush's pool was the Linode hack, which didn't impact users, and was not Slush's fault.
• Deepbit has more "untrustworthy" votes than Slush & BTC Guild combined.  Yet their track record is spotless, and they are the second oldest (still running) pool.  Only bad thing Deepbit has going are the people convinced that Deepbit is a threat, which is a vocal minority.
• P2Pool - This shouldn't have a single vote for being "untrustworthy".  It's a completely open source project that has no pool operator.  It shouldn't even be on the poll, since the entire point of p2pool is no trust is needed.

Those aren't the only ones I disagree with, but they're the 3 most obvious.  The two oldest pools who I have never seen do their users wrong, should not be getting remotely as many votes as they have.

[rjk]

I just dont understand why some of these people voted - does not make sense to me so I had to share my opinion

I mean seriously 15% think Slush is not trustworthy? Come on...

this poll is FUBAR

To be fair, any poll in the Pools forum is pretty much just a test of how rabid the users of pools are.  The polls on this thread have been popularity contests (or how loud the haters were), rather than their trustworthiness (or lack thereof).  Even polls specifically for a pool on their own thread tend to give pretty awkward results that don't give an accurate representation of the overall pool users.

+1, pretty much. It's really difficult to get an accurate representative sample of users here.

[organofcorti]

The second chart, giving the PDF of the pool debt after n rounds, looks completely wrong, I have no idea how you got that. The pool debt, assuming constant difficulty, is (X/D - n)*B, where X is the number of shares it took to get n blocks, which is distributed negative binomial. So it's really a shifted negative binomial, which doesn't have the cusp we see.

Meni, I couldn't find my notes so I've reproduced what I could from memory. I might have left out a step or two, but the results hold, and you can test them out if you like.

Let x1, x2, x3, …, xn be independent geometrically distributed random variables with the same probability, p.

Let Sj be the sum of x1 + x2 + x3 + ….+xj so that:

    S1 = x1
    S2 = S1 + x2
    S3 = S2 + x3
       …
    Sj = Sj-1 + xj
       …..
    Sn = Sn-1 + xn


Since x1, x2, x3, … ,xj …,xn are geometrically distributed, Sj is a random variable from a shifted negative binomial distribution with target number of successful trials = j, with density:

Pr(X=Sj) =  Γ(x+j)/(Γ(j) * Γ(x+1)) * p^j * (1-p)^x, so it follows:

Pr(X∈(S1, S2, S3, …, Sj, … ,Sn))
    = sum from 1 to n [Γ(x+j)/(Γ(j) * Γ(x+1)) * p^j * (1-p)^x]/n (1)

So the probability density of a cumulative sum of independent geometrically distributed random variables is given by (1) and tends toward a uniform distribution on [1,j/p].

Similarly, if x1, x2, x3, … ,xj …,xn are geometrically distributed with mean = 0, then Sj is a random variable from a shifted negative binomial distribution with target number of successful trials = j and mean = 0. Since the mean is 0, the component probability densities can be visualised as below.



The probability densities Pr(X=Sj) and Pr(X∈(S1, S2, S3, …, Sj, … ,Sn)):

Pr(X=Sj) =  Γ(x+(1-p)*j/p+j)/(Γ(j) * Γ(x+(1-p)*j/p+1)) * p^j * (1-p)^((x+(1-p)*j/p))

Pr(X∈(S1, S2, S3, …, Sj, … ,Sn))
    = sum from 1 to n [Γ(x+(1-p)*j/p+j)/(Γ(j) * Γ(x+(1-p)*j/p+1)) * p^j * (1-p)^((x+(1-p)*j/p))]/n (2)

Although (2) does not exist in a closed form, it can be calculated and charts of the distribution for n=100, n=1000 and n=10000 for p=1e-06 are shown below. This distribution does not seem to tend toward a known distribution for large n or very small p.

[organofcorti]

The whole discussion about SMPPS is based on:

The assumption that this is true:

If the pool stays in a large negative buffer long enough, it is a likely that all miners will leave, and the pool will fail.

Which we ALREADY know to be FALSE.

You're arguing about the the conclusion which I posted here without reading the materiel upon which it was based. Otherwise you'd try to prove your point instead of presenting it as a magical fait accompli.

Please read the actual post, Kano. I think you mean well but if you aren't reading the post then I can't argue with you about it. You just come off seeming like a clever chatbot that repeats itself.

I've read the actual post twice already, ok 3 times now

The rest is based on the concept that over infinite time, during that infinite time, the value of the buffer can be any size - negative or positive - but taking that point to a specific case.

No, it's not. It's about how probabilities of the risk of an arbitrarily sized negative buffer occurring before n rounds can be calculated. It was to give miners a better idea of the risks involved with SMPPS.

Which of course is true.
(though you didn't seem to bother to point out that it can be both positive and negative in the same way )

True, but hardly useful when discussing the risks of losing shares. A positive buffer is not good for a pool in the way a negative buffer is bad for a pool. Maturity time won't get any less as the buffer becomes more positive, but maturity time will increase significantly as the buffer becomes more negative.

That is of course true of DGM also.

No, I don't think it is. I'll believe you if you can prove it using Meni's DGM functions though.

The DGM argument is that the pool will shutdown before that happens and the pool does not need to disclose the buffer.
I see that as a conflicting statement in itself.

As they say on wikipedia, "says who?".

Again you also make the assumption at the end

A strategic miner will only mine at an SMPPS pool when it has a positive buffer.

Which is in the least questionable as to it's relevance ...

It was very relevant. If some miners leave the pool when the buffer is negative, then the pool hashrate declines and while maturity time in terms of blocks will stay the same, with a reduced hashrate the number of days (or hours or weeks) until payment increases. The greater the increase, the fewer the number of miners that are prepared to wait so long for payment. Miners leave, the hashrate decreases again. And so on.

I'm sorry the post was misleading to you. I hope no one else interpreted it that way.

[organofcorti]

I just dont understand why some of these people voted - does not make sense to me so I had to share my opinion

I mean seriously 15% think Slush is not trustworthy? Come on...

this poll is FUBAR

To be fair, any poll in the Pools forum is pretty much just a test of how rabid the users of pools are.  The polls on this thread have been popularity contests (or how loud the haters were), rather than their trustworthiness (or lack thereof).  Even polls specifically for a pool on their own thread tend to give pretty awkward results that don't give an accurate representation of the overall pool users.

+1, pretty much. It's really difficult to get an accurate representative sample of users here.

+1 here too. A page back I found out that some people were voting randomly just to see the results. Plus grudge holders have a reason to vote negatively even if it's unfair, or if they don't like the pool and aren't voting on trust but just to get bad ratings for a pool. So I'm sorry to all who voted, I'm killing the poll until I can find a better way to get an idea about trust. Maybe I just have to only phrase it positively, like the first one. Ideas?

[Meni Rosenfeld]

Pr(X∈(S1, S2, S3, …, Sj, … ,Sn))
    = sum from 1 to n [Γ(x+j)/(Γ(j) * Γ(x+1)) * p^j * (1-p)^x]/n (1)

What is this quantity supposed to represent? It looks like you consider it to be meaningful but I just don't see it. Sn itself is the number of shares submitted in n rounds, isn't this what you're trying to analyze?

Also, if by the LHS you mean "probability that X is in the set {S1,..., Sn}", why would it be equal to the RHS?

[eleuthria]

+1 here too. A page back I found out that some people were voting randomly just to see the results. Plus grudge holders have a reason to vote negatively even if it's unfair, or if they don't like the pool and aren't voting on trust but just to get bad ratings for a pool. So I'm sorry to all who voted, I'm killing the poll until I can find a better way to get an idea about trust. Maybe I just have to only phrase it positively, like the first one. Ideas?

In my opinion, it doesn't matter how the poll is phrased because your sample size is a narrow subset of overall pool users.  You end up with the problem of vocal minorities being the primary respondents to your poll.  While in theory that shouldn't skew results, it's quite obvious in the last two polls [and this one especially] that you end up with slants that don't seem to reflect reality.

Similarly, all it takes is one pool mentioning the poll in their thread, their website, or their chatroom to completely swing the results of the poll in their favor.

[kano]

...

Which of course is true.
(though you didn't seem to bother to point out that it can be both positive and negative in the same way )

True, but hardly useful when discussing the risks of losing shares. A positive buffer is not good for a pool in the way a negative buffer is bad for a pool. Maturity time won't get any less as the buffer becomes more positive, but maturity time will increase significantly as the buffer becomes more negative.

That is of course true of DGM also.

No, I don't think it is. I'll believe you if you can prove it using Meni's DGM functions though.

The DGM argument is that the pool will shutdown before that happens and the pool does not need to disclose the buffer.
I see that as a conflicting statement in itself.

As they say on wikipedia, "says who?".
...

Meni.

I'll find the posts he said this later if I really need to?

[organofcorti]

+1 here too. A page back I found out that some people were voting randomly just to see the results. Plus grudge holders have a reason to vote negatively even if it's unfair, or if they don't like the pool and aren't voting on trust but just to get bad ratings for a pool. So I'm sorry to all who voted, I'm killing the poll until I can find a better way to get an idea about trust. Maybe I just have to only phrase it positively, like the first one. Ideas?

In my opinion, it doesn't matter how the poll is phrased because your sample size is a narrow subset of overall pool users.  You end up with the problem of vocal minorities being the primary respondents to your poll.

Not just vocal minorities, but (hopefully) miners and pool ops who are concerned with promoting reliability and accountability for pools. I had hoped there'd be fewer barrows to push, but oh well.