[1200 TH] EMC: 0 Fee DGM. Anonymous PPS. US & EU servers. No Registration!

[Inaba]

Simeonk83: Yep, I'm working on that.  The problem is to get an accurate estimate if the round ended right then, the amount of SQL query time is fairly substantial (has to add up all the rows) and it's not really feasible to do it on an interval on a per user basis, either... so I'm looking at the best way to handle it that puts the least load on the server.  Might be able to get it done today, I'm hoping.

GenTarkin: We are actually doing ok, running right at or just under difficulty which is good.  If you look at the blocks over all in the pool we are under difficulty by a decent margin so we are + luck as a whole.

[1713503409]

1713503409

[1713503409]

1713503409

[1713503409]

1713503409

[simonk83]

Simeonk83: Yep, I'm working on that.  The problem is to get an accurate estimate if the round ended right then, the amount of SQL query time is fairly substantial (has to add up all the rows) and it's not really feasible to do it on an interval on a per user basis, either... so I'm looking at the best way to handle it that puts the least load on the server.  Might be able to get it done today, I'm hoping.

No worries, I have no doubt it's not easy      What if you made it only update on request?   In that someone has to physically press a button to update the estimation.   That way the queries only have to run when requested, rather than every x minutes?  Not sure if that helps

[PoulGrym]

That'd be a good solution. Then also limit to one check every 10 min.. not to get flooded with req's all the time..

[rearwheels]

Simeonk83: Yep, I'm working on that.  The problem is to get an accurate estimate if the round ended right then, the amount of SQL query time is fairly substantial (has to add up all the rows) and it's not really feasible to do it on an interval on a per user basis, either... so I'm looking at the best way to handle it that puts the least load on the server.  Might be able to get it done today, I'm hoping.

No worries, I have no doubt it's not easy      What if you made it only update on request?   In that someone has to physically press a button to update the estimation.   That way the queries only have to run when requested, rather than every x minutes?  Not sure if that helps

I believe the SQL will take a fair bit of time to run, so if it is on-demand, after clicking, you will have to wait for a while before the information is displayed.

Secondly, if someone clicks on the link and the sql starts, but the user clicks to goto another page, the SQL will continue to run until it is completed or times out, replies to the web server which then does not display the information so the SQL processing time is wasted (similar to not having long poll).

[simonk83]

Fair enough, forget that then

[1bitc0inplz]

When we used to use Meni's scoring system, we were doing all the estimated in real-time. If implemented correctly (e.g., not doing the math in SQL) you guys shouldn't have a problem getting an estimate that is real-time as well.

[simonk83]

When we used to use Meni's scoring system, we were doing all the estimated in real-time. If implemented correctly (e.g., not doing the math in SQL) you guys shouldn't have a problem getting an estimate that is real-time as well.

What have you switched to (and why)?

[Inaba]

Can you elaborate on that?  And doing it correctly would be implementing it in SQL or in the case of EMC, most of it is in done on the getwork server.  Doing it in PHP is the absolute worst way to handle it.

[1bitc0inplz]

When we used to use Meni's scoring system, we were doing all the estimated in real-time. If implemented correctly (e.g., not doing the math in SQL) you guys shouldn't have a problem getting an estimate that is real-time as well.

What have you switched to (and why)?

We switched to ESMPPS, and the reason (frankly) was that we disliked the increased variance of scoring systems.

Can you elaborate on that?  And doing it correctly would be implementing it in SQL or in the case of EMC, most of it is in done on the getwork server.  Doing it in PHP is the absolute worst way to handle it.

Since we wrote our own pushpool replacement, we did all the scoring calculations there. You are correct, PHP wouldn't be the place to do them. Neither is SQL though. To do it right you'll want to do it in your pool backend, which is pushpool I'm assuming? You'll want to keep the shares in memory, and do your scoring computations off of the in-memory copy.

[GenTarkin]

When we used to use Meni's scoring system, we were doing all the estimated in real-time. If implemented correctly (e.g., not doing the math in SQL) you guys shouldn't have a problem getting an estimate that is real-time as well.

What have you switched to (and why)?

We switched to ESMPPS, and the reason (frankly) was that we disliked the increased variance of scoring systems.

Can you elaborate on that?  And doing it correctly would be implementing it in SQL or in the case of EMC, most of it is in done on the getwork server.  Doing it in PHP is the absolute worst way to handle it.

Since we wrote our own pushpool replacement, we did all the scoring calculations there. You are correct, PHP wouldn't be the place to do them. Neither is SQL though. To do it right you'll want to do it in your pool backend, which is pushpool I'm assuming? You'll want to keep the shares in memory, and do your scoring computations off of the in-memory copy.

Dude, do you run bitp.it? Cuz, thats similar to what they have. I so want a PPS type payout from emc....

[Meni Rosenfeld]

When we used to use Meni's scoring system, we were doing all the estimated in real-time. If implemented correctly (e.g., not doing the math in SQL) you guys shouldn't have a problem getting an estimate that is real-time as well.

What have you switched to (and why)?

We switched to ESMPPS, and the reason (frankly) was that we disliked the increased variance of scoring systems.

Did you make any attempt to adjust the parameters?

Anyway, you could have used PPLNS which has very little variance and isn't broken.

[1bitc0inplz]

When we used to use Meni's scoring system, we were doing all the estimated in real-time. If implemented correctly (e.g., not doing the math in SQL) you guys shouldn't have a problem getting an estimate that is real-time as well.

What have you switched to (and why)?

We switched to ESMPPS, and the reason (frankly) was that we disliked the increased variance of scoring systems.

Did you make any attempt to adjust the parameters?

Anyway, you could have used PPLNS which has very little variance and isn't broken.

Yes, we did attempt to adjust the parameters. But the idea of giving the pool operator variance also wasn't for us. No offense, but I think you know very little about the specifics of ESMPPS but you seem very determined to think that it is broken. I, frankly, think you are basing your assumptions on ESMPPS from what you think you know about SMPPS.

But, that is besides the point because your bias to your own scoring method is quite clear.

I was just simply trying to point out to Inaba that it was possible to get real-time stats from your scoring system. I feared that speaking in this thread might provoke you, so I will cease offering my assistance.

[Inaba]

Since we wrote our own pushpool replacement, we did all the scoring calculations there. You are correct, PHP wouldn't be the place to do them. Neither is SQL though. To do it right you'll want to do it in your pool backend, which is pushpool I'm assuming? You'll want to keep the shares in memory, and do your scoring computations off of the in-memory copy.

I'm a little unclear on how you'd achieve keeping the scoring information in memory in a long round without taking up an extraordinary amount of memory?  But regardless, how do you pass that information off to a web browser looking for it?  Additionally, how do you handle (or do you not) multiple getwork servers?  Each one would have differing information.

[Meni Rosenfeld]

When we used to use Meni's scoring system, we were doing all the estimated in real-time. If implemented correctly (e.g., not doing the math in SQL) you guys shouldn't have a problem getting an estimate that is real-time as well.

What have you switched to (and why)?

We switched to ESMPPS, and the reason (frankly) was that we disliked the increased variance of scoring systems.

Did you make any attempt to adjust the parameters?

Anyway, you could have used PPLNS which has very little variance and isn't broken.

Yes, we did attempt to adjust the parameters. But the idea of giving the pool operator variance also wasn't for us. No offense, but I think you know very little about the specifics of ESMPPS but you seem very determined to think that it is broken. I, frankly, think you are basing your assumptions on ESMPPS from what you think you know about SMPPS.

But, that is besides the point because your bias to your own scoring method is quite clear.

I was just simply trying to point out to Inaba that it was possible to get real-time stats from your scoring system. I feared that speaking in this thread might provoke you, so I will cease offering my assistance.

I apologize for calling ESMPPS "broken" which was too confrontational. I resent your accusations. If my advice isn't welcome I'll stop wasting my time.

[Inaba]

I read the linked post in regards to how ESMPPS works and I'm a little confused - it seems like unless you have a large run of good luck, the pool will always run in the red and be behind in paying out shares?  As time goes on, the pool would go further and further into the red.  Does ESMPPS count on a run of good luck to even things out?

[GenTarkin]

grr another long block ... lol... hows the PPS stuff comin inaba? =)

[1bitc0inplz]

Since we wrote our own pushpool replacement, we did all the scoring calculations there. You are correct, PHP wouldn't be the place to do them. Neither is SQL though. To do it right you'll want to do it in your pool backend, which is pushpool I'm assuming? You'll want to keep the shares in memory, and do your scoring computations off of the in-memory copy.

I'm a little unclear on how you'd achieve keeping the scoring information in memory in a long round without taking up an extraordinary amount of memory?  But regardless, how do you pass that information off to a web browser looking for it?  Additionally, how do you handle (or do you not) multiple getwork servers?  Each one would have differing information.

I do not know much about your backend, but it sounds like we are two entirely different architectures. On my pool we use Redis (a no-SQL datastore) and Node.js. Additionally we have a single "getwork server". So, since Redis stores everything in memory anyway, we were able to really quickly re-perform the scoring algorithm on every share submission. Unless you centralize your share data in someplace other than SQL, I do not think our solution will be easy for you.

I am curious, why do you have multiple servers? Is it to allow for users to be closer to a server (e.g., geographic distribution), or are you doing this for performance reasons? If at all possible try and consolidate your getworks to a single server, and I think that will open up more possibilities to you.

I read the linked post in regards to how ESMPPS works and I'm a little confused - it seems like unless you have a large run of good luck, the pool will always run in the red and be behind in paying out shares?  As time goes on, the pool would go further and further into the red.  Does ESMPPS count on a run of good luck to even things out?

I am not aware of any single post that fully describes ESMPPS, unfortunately. I'd love to write one, but I've just been pretty swamped lately.

ESMPPS does not depend on luck to even things out. In fact, that is one of the key differences between it and SMPPS. SMPPS pays out older shares first. So, if the pool gets behind on payment, then it neglects newer shares to pay back older shares. This creates a problem where new miners may have to wait several rounds without any payment before receiving any of their earnings. Obviously nobody wants to work for "free", so it would be a very disastrous thing for a SMPPS pool to go red.

ESMPPS solves this problem by putting all shares into "buckets", allocated by their percent paid.  A bucket is never paid more (percentage wise) than the least paid bucket. This allows for all shares to be paid the maximum amount possible, while not favoring older or new miners.

Take the scenario where a pool is always lucky. The pool would therefore always have a positive buffer, and therefore pay 100% of what a 0% "true" PPS would. However, you and I both realize two problems with this scenario. A, no pool will ever be lucky 100% of the time. And B, no "true" PPS can sustain a 0% fee.

So, let's look a a difference scenario. Say you pay out on invalid blocks, and have a 0.5% block withholder rate (e.g., 0.5% of your pools hashing power is "attacking" you by withholding any block solutions they may find). Assume the probability of a block becoming invalid is ~1%. Also, assume that "luck" nets out to 0 over a long period of time.

In this scenario, you will eventually reach a future where no bucket is paid past the 98.5% bucket. However, unlike *MPPS systems old and new shares are paid immediately up to the 98.5% range. This underpayment is remembered by the system, and if the buffer goes positive, the backpay is made up. Interestingly enough, even in this scenario the ESMPPS pool still behaves "normal" and outperforms any true PPS pool, provided that true PPS pool charges more than a 1.5% fee.

IMHO it's a pretty interesting dynamic, a pool that "self regulates" itself. This is, ideally, what a true PPS pool tries to accomplish when the pool operator charges a fee. However, in the case of ESMPPS it set by the probabilities of the situation itself, and if the situation takes a turn for the better then everybody wins. If the situation stays the same, it still outperforms true PPS.

[rearwheels]

block stats were updated really fast this time round.

[Inaba]

I do not know much about your backend, but it sounds like we are two entirely different architectures. On my pool we use Redis (a no-SQL datastore) and Node.js. Additionally we have a single "getwork server". So, since Redis stores everything in memory anyway, we were able to really quickly re-perform the scoring algorithm on every share submission. Unless you centralize your share data in someplace other than SQL, I do not think our solution will be easy for you.

Hmm, I'm not familiar with Redis - so are you saying you have direct access to the memory that your getwork backend is using from your webserver to gather those statistics?  That's mighty handy if so.

I am curious, why do you have multiple servers? Is it to allow for users to be closer to a server (e.g., geographic distribution), or are you doing this for performance reasons? If at all possible try and consolidate your getworks to a single server, and I think that will open up more possibilities to you.

I could easily consolidate to a single server, but you are correct in your geographical assumption.  There is simply no way to run an efficient server over oceanic links, the latency is to great.  Western Europe is at the very limit of acceptability, often falling into unacceptable latency, anything further east and it becomes untenable.  Thus multiple servers.

I am not aware of any single post that fully describes ESMPPS, unfortunately. I'd love to write one, but I've just been pretty swamped lately.

In this scenario, you will eventually reach a future where no bucket is paid past the 98.5% bucket. However, unlike *MPPS systems old and new shares are paid immediately up to the 98.5% range. This underpayment is remembered by the system, and if the buffer goes positive, the backpay is made up. Interestingly enough, even in this scenario the ESMPPS pool still behaves "normal" and outperforms any true PPS pool, provided that true PPS pool charges more than a 1.5% fee.

IMHO it's a pretty interesting dynamic, a pool that "self regulates" itself. This is, ideally, what a true PPS pool tries to accomplish when the pool operator charges a fee. However, in the case of ESMPPS it set by the probabilities of the situation itself, and if the situation takes a turn for the better then everybody wins. If the situation stays the same, it still outperforms true PPS.

Let me know if you do write something more definitive on it, I would be very interested in it.  Have you done any analysis on the timeframe for a pool to self regulate?  It seems that it might take a very long time to reach a steady state if you have a particularly bad run of blocks without a balancing good run extreme swing.  I can see how it would outperform a pure PPS, certainly, but it seems theres a lot of risk to the users, or operator if the operator wishes to shoulder that burden by paying out into the negative.

[Inaba]

grr another long block ... lol... hows the PPS stuff comin inaba? =)

I think I've come up with a good solution, but it's fairly complicated.  I need to complete a tenative design and I'll roll out a test server once I have something concrete in place.