[1423GH] ABCPool PPS - Proxy Pool For High & Steady Mining Rewards

[rTech]

Damn i noticed same thing but i thought it was my own sucky internet connection

Heh i even reseted by modem and setup'd it Damn from now on im reading firts
forum before i act on my own 5hrs of lost works is a lot for small miner like me.

[Jezzz]

The rate of invalids seems very high.  I have ~60,000 shres and ~2,000 invalids.   Is this normal?  What makes a share invalid?  I have not had this problem with other pools.

[Jack of Diamonds]

No, I don't think that's very normal. That is a over 3% rate.

I mined for about a week on this pool & had ~3000 invalids but only after half a million shares or so. So that's about 0.5%.

[MintCondition]

The rate of invalids seems very high.  I have ~60,000 shres and ~2,000 invalids.   Is this normal?  What makes a share invalid?  I have not had this problem with other pools.

This is not normal. Our users typically see less than ~0.5% invalids.

Shares are invalid if they fall under any of these categories:
* if their hash does not meet the difficulty target for a share (ie: computation error on client)
* if they are duplicates (ie: client resubmits; this does not affect your earnings)
* if they were handed out to a different worker than the one submitting a solution for them (ie: client programming error)
* if they were handed out more than 120 seconds ago (ie: expiration; happens when using slow GPUs or multiple workers per GPU)
* if they are based on a different timestamp than the one supplied (ie: 'time rolling' is not supported)

NB: Shares are considered 'Stale' if they would have been valid for the previous block and arrive within 6 seconds of noticing the new block.

Which client are you using? And do you use a proxy?

[Jezzz]

Cgminer, no proxy.

[MintCondition]

Our backend got stuck around 07:00 UTC, and has just been forcibly restarted. As a consequence, ABCPool has been out of operation for approx. 5 hours. We are doing a post-mortem on the incident now.

The root cause of the problem was found: A bug had triggered an infinite loop. This caused the backend to become stuck without crashing. A crash would have restarted the backend automatically, but being stuck is more difficult to recognize. We have since fixed the bug and are evaluating options for automatic forced restarts in case a similar bug should arise.

ABCPool is now fully operational again. Please verify that your miners have reconnected.

[MintCondition]

Cgminer, no proxy.

It could be an LP issue, since the amount of invalids is around the 3% people see when mining without LP.

I know of a bug in Phoenix, where it reports that it is doing longpolling but has actually lost the LP connection. Since it fails to time the connection out regularly, it doesn't know it will never receive new-block notifications through LP. This can be witnessed in its logs around the moments a new block is reported: There would be no message indicating Long Polling was responsible for pushing new work, but just a message that a new block was found. Restarting Phoenix solves the problem (until it happens again).

Maybe something like this happens with cgminer?

[PoulGrym]

Just wanted to tell I tried to change to 100% donation. It only worked with 9.99 >_<

Bye guys! might have a look here when PPS .. right now ABC is pretty sweet! less then 0.5 rejected. Inaba you should check what they do to keep it so low. Here i had about 3~4% Rejected shares.

They modify their getwork server to report less rejects/stales.  There's no mystery there. I can do the same here if it'll make you feel better, though.  I can, in fact, using their methods, give you 0 rejects.

They have the same reject rate as here, they just don't report it to you, so you will never know if you are having a problem or not and therefore causing a detriment to the pool as a whole.  On an individual level, it's not a big deal, but when you start getting a lot of miners, and if everyone has absolutely no idea if there's a problem, it can add up to a significant drain on the pools true hashing power (vs reported hashing power).  Unless they've done some mathematical gymnastics with stat reporting, their true hashrate is 1.5 - 3.5% lower than their reported hash rate.  That means you are losing out on 1.5 - 3.5% of your potential income vs variance when you make a decision to mine at a given pool.  (That is not to say your overall income is materially affected, only your decision making ability to assess whether one pool is better than another.  Although, if enough people have unreported problems it will also affect your overall income as well.  But you'll never know unless people self report a lower than expected income.)

They bill this as a "feature," and in so far as they pay you on stale shares it is definitely a bonus, but in reality it's basically dishonest math when measured against the accepted standard.  I've not found anywhere that they confirm or deny that they report stats based on stale shares as well.

So I wanted to see what you guys have to say in your defense?

[MintCondition]

Just wanted to tell I tried to change to 100% donation. It only worked with 9.99 >_<

Bye guys! might have a look here when PPS .. right now ABC is pretty sweet! less then 0.5 rejected. Inaba you should check what they do to keep it so low. Here i had about 3~4% Rejected shares.

They modify their getwork server to report less rejects/stales.  There's no mystery there. I can do the same here if it'll make you feel better, though.  I can, in fact, using their methods, give you 0 rejects.

They have the same reject rate as here, they just don't report it to you, so you will never know if you are having a problem or not and therefore causing a detriment to the pool as a whole.  

So I wanted to see what you guys have to say in your defense?

There are definitely pools that do this. Actually, *all* non-PPS pools that purport to be paying for stale shares, actually don't. I do understand the confusion, since it also took Us some time before I realized that pay-for-stales claims can not ever lead to extra earnings at non-PPS pools.

Slush's pool for example has no LP, so it can not possibly notify its workers of a new block. Practically this means those workers do stale work ~3% of the time, although Slush accepts most of these stale shares as valid shares. They therefore report a 0.3% stale rate. Total block reward will be the same 50 BTC in the end, and since everyone has approx. 3% extra shares, payout per user is still the same. Only everyone thinks they get payed for stales.

On the other hand, any PPS pool would shoot itself in the foot if it was reporting a lower stale share amount than was actually the case: It would mean they now also have to pay the promised per share amount for those 'fake valids', whereas normally those would be worthless. The aforementioned dilution effect does not exist at PPS pools, since there is no 50 BTC block reward to be distributed (just the pay per share).

For ABCPool, this goes even further: since we pay as much for stale shares as we do for valid shares, your earnings would be exactly the same if we report 0% stales, 0.5% stales or 100% stales.

For more info, see http://www.abcpool.co/stale-shares.php.

NB: SMPPS, ESMPPS, or other PPS-like reward methods are just as vulnerable to this issue as Prop or PPLNS pools. Even Solo mining has a stale rate, which is reflected in the number of orphan blocks found over time.

NB2: The 'secret' behind real low stale invalid block rates is having a highly connected bitcoind running so you are quickly notified of new blocks, wherever they may come from.

[Inaba]

There are definitely pools that do this. Actually, *all* non-PPS pools that purport to be paying for stale shares, actually don't. I do understand the confusion, since it also took Us some time before I realized that pay-for-stales claims can not ever lead to extra earnings at non-PPS pools.

Slush's pool for example has no LP, so it can not possibly notify its workers of a new block. Practically this means those workers do stale work ~3% of the time, although Slush accepts most of these stale shares as valid shares. They therefore report a 0.3% stale rate. Total block reward will be the same 50 BTC in the end, and since everyone has approx. 3% extra shares, payout per user is still the same. Only everyone thinks they get payed for stales.

On the other hand, any PPS pool would shoot itself in the foot if it was reporting a lower stale share amount than was actually the case: It would mean they now also have to pay the promised per share amount for those 'fake valids', whereas normally those would be worthless. The aforementioned dilution effect does not exist at PPS pools, since there is no 50 BTC block reward to be distributed (just the pay per share).

For ABCPool, this goes even further: since we pay as much for stale shares as we do for valid shares, your earnings would be exactly the same if we report 0% stales, 0.5% stales or 100% stales.

For more info, see http://www.abcpool.co/stale-shares.php.

NB: SMPPS, ESMPPS, or other PPS-like reward methods are just as vulnerable to this issue as Prop or PPLNS pools. Even Solo mining has a stale rate, which is reflected in the number of orphan blocks found over time.

NB2: The 'secret' behind real low stale rates is having a highly connected bitcoind running so you are quickly notified of new blocks, wherever they may come from.

After I wrote that a bit later I realized it might have sounded accusatory or advasarial and I apologize for that.  It was not my intent, but by that time it was already out there and editing it would have been pointless.  However, the contention in the message still stands and I'm not seeing where there's a rebuttal in so far as providing anything to refute the fact that you simply aren't reporting the stale/invalid shares even though they exist. 

I said as much that it's great that you are paying for stales and being a PPS pool, it's a definite advantage - I have absolutely no argument with that.  But whether or not you report it is immaterial to the fact that you are paying for stales - you pay for them either way, so it is to your advantage to NOT report them to give the illusion that you have a better (lesser) reject rate than other pools. 

I'm not sure you understand how LP works when talking about stales in relation to LP.  It's completely immaterial how connected your bitcoind is or how quickly your bitcoind gets notified of a new block - your bitcoind and pushpool (if you are using pushpool or something else, it doesn't really matter) are a little microcosm of the larger network.  All that matters for LP and stales is when pushpoold is notified of a block change, the LP is pushed out and the workers receive new shares for the block bitcoind is currently on.  The Bitcoind -> Work Distribution Server interaction are all that matters.  Bitcoind is going to be doling out the getworks to your getwork server based on what it thinks is the current block.  Your bitcoind could be 10 blocks behind and that will not have an effect on your stale count because your getwork server is serving up what bitcoind thinks is the latest block and submitting it back, which bitcoind tests against what it thinks is the current block.  If your bitcoind is behind and you find a hash that fits, it will then submit it to the network and get an orphaned block because it was behind.

So again, I still content that you are either not reporting the actual number of stales or you've found some magical new method to a) reduce latency between your getwork server and bitcoind by a factor of two or three and more importantly b) found a really magical method to notify all miners of new work with less latency by several orders of magnitude.

Since you have no control of the links between your getwork server and the miners, and you have no control over the miners and their systems, it would really be a magical method to reduce your stale shares beyond possibly maybe 2% maximum at a theoretical limit.  If you gain control over those system, then yes you could reduce that further, but barring controlling the entire system, with the current GPU miners out there, you simply can't reduce it beyond that threshold because of built in problems with the protocol in general.

[spiccioli]

Hi MintCondition,

I've tested your pool with poclbm and DiabloMiner and both have problems staying connected, while phoenix, on the same rig, works ok.

What is different between your pool and most others?

This is DiabloMiner

Code:

[8/18/11 10:22:18 AM] ERROR: Cannot connect to pool.abcpool.co: Bitcoin disconnected during response: 200 ok
[8/18/11 10:22:27 AM] pool.abcpool.co accepted block 342 from Cypress (#1)   
[8/18/11 10:22:28 AM] pool.abcpool.co accepted block 343 from Cypress (#2)   
[8/18/11 10:22:30 AM] pool.abcpool.co accepted block 344 from Cypress (#1)   
[8/18/11 10:22:30 AM] pool.abcpool.co accepted block 345 from Cypress (#1)   
[8/18/11 10:22:32 AM] pool.abcpool.co accepted block 346 from Cypress (#2)   
[8/18/11 10:22:46 AM] pool.abcpool.co accepted block 347 from Cypress (#1)   
[8/18/11 10:22:47 AM] pool.abcpool.co accepted block 348 from Cypress (#2)   
[8/18/11 10:23:08 AM] ERROR: Cannot connect to pool.abcpool.co: Bitcoin disconnected during response: 200 ok

and so on and so on till it goes out of sync and invalid shares in my account go to the roof.

poclbm keeps disconnecting, I don't have a log, but I can give one if you need it.

TIA

spiccioli.

[Jezzz]

Cgminer, no proxy.

It could be an LP issue, since the amount of invalids is around the 3% people see when mining without LP.

I know of a bug in Phoenix, where it reports that it is doing longpolling but has actually lost the LP connection. Since it fails to time the connection out regularly, it doesn't know it will never receive new-block notifications through LP. This can be witnessed in its logs around the moments a new block is reported: There would be no message indicating Long Polling was responsible for pushing new work, but just a message that a new block was found. Restarting Phoenix solves the problem (until it happens again).

Maybe something like this happens with cgminer?

Updated everything and still many invalids.  The ratio is much higher than can be attributed to LP issues.  I had to move my miners away for the time being, but my current stats are 126,914 shares and 24,479 invalids.

[MintCondition]

After I wrote that a bit later I realized it might have sounded accusatory or advasarial and I apologize for that.

Apoligy accepted

However, the contention in the message still stands and I'm not seeing where there's a rebuttal in so far as providing anything to refute the fact that you simply aren't reporting the stale/invalid shares even though they exist.  

I don't really understand what you mean by not reporting them. Sure, our statistics are pretty basic now but the valid/stale/invalid counts that we have been reporting are (1) reflecting the share submission results we returned to mining clients and (2) accurate. (1) can be proven by miners themselves, if they keep a detailed log of sharecounts. (2) is never provable I guess.

(...) whether or not you report it is immaterial to the fact that you are paying for stales - you pay for them either way, so it is to your advantage to NOT report them to give the illusion that you have a better (lesser) reject rate than other pools.  

That's right. There definitely is a reputation incentive, and not a financial one. Non-PPS pools have both.

I'm not sure you understand how LP works when talking about stales in relation to LP.

We rewrote every line of code that had to do with LP, because it was horribly inefficient, so I understand a thing or two about LP. However..

It's completely immaterial how connected your bitcoind is or how quickly your bitcoind gets notified of a new block

you're right about this. My mistake. In NB2 I should have written 'invalid block' where i wrote 'stale'.

(...) you've found some magical new method to a) reduce latency between your getwork server and bitcoind by a factor of two or three and more importantly b) found a really magical method to notify all miners of new work with less latency by several orders of magnitude.

There's nothing magical about it. Deepbit is also quite good at it as I understand.

On a general note, thanks for your inquisitiveness, it's good to ask the hard questions once in a while . I really do want to focus on improving ABCPool now, so I won't be giving this kind of detailed replies any more on this topic.

[MintCondition]

Cgminer, no proxy.

Updated everything and still many invalids.  The ratio is much higher than can be attributed to LP issues.  I had to move my miners away for the time being, but my current stats are 126,914 shares and 24,479 invalids.

We're sorry to see you go. We don't have the resources right now to look into this very deep. When we have the chance we will do some tests with cgminer ourselves to see where the invalids may be coming from.

It would help if you can share two more things with us:
1) The MH/s each of your cards normally produces
2) Whether the rate displayed in your account at ABCPool is lower than what you experience at other pools.

Regarding 2): Last week a user reported lots of invalids, but said it didn't affect his reported rate of valids. If your valid hash rate is affected, you should have seen a reported hashrate of +-16% lower than normal (ie 24479 / (24479+126914)).

[MintCondition]

I've tested your pool with poclbm and DiabloMiner and both have problems staying connected, while phoenix, on the same rig, works ok.

What is different between your pool and most others?

Thanks for letting us know about this. We did heavy modifications of pushpool to obtain low stale rates. We've only tested these mods with Phoenix, that's why we recommend it on our site. It now appears that some other clients may experience problems.

This is DiabloMiner

Code:

[8/18/11 10:22:18 AM] ERROR: Cannot connect to pool.abcpool.co: Bitcoin disconnected during response: 200 ok

I looked through the Diablo code, and I'm fairly certain this is caused by ABCPool timing out the LP connection with a '200 OK' after 50 seconds to get rid of disconnected clients. Most pools don't do this, but it is one of the reasons ABCPool can keep the stalerate low. Phoenix handles this excellently by silently reconnecting; just what we want. Diablominer might expect a different response than 200 OK. I've posted the issue in their forum.

poclbm keeps disconnecting, I don't have a log, but I can give one if you need it.

That would be most helpful. Can you PM them?

[Inaba]

I don't really understand what you mean by not reporting them. Sure, our statistics are pretty basic now but the valid/stale/invalid counts that we have been reporting are (1) reflecting the share submission results we returned to mining clients and (2) accurate. (1) can be proven by miners themselves, if they keep a detailed log of sharecounts. (2) is never provable I guess.

I should have been more concise in my contention.  You have done one of two things:

1. You've widened the window as to when a share is considered stale.
2. You've modified the code to not report stales that are associated with LP.

My point is that you have not and can not reduce the amount of ACTUAL stales beyond a certain threshold due to factors that are outside of your control.  ACTUAL stales, in this context, we are defining as shares submitted from a block that is no longer active (e.g. after a LP).  You may or may not be paying on stales generated through other avenues (such as slow or broken miners, malicious actions, etc...) and they aren't really within the scope of what we are talking about.

With that definition in place, your reported stale rate of 0.5% (at least by one of your users?) is literally impossible with the hashrate you have.  If you only had 5 or 10 miners, then yes, you could probably achieve that if they were close by on the network.  However, with a diverse population of miners (and I believe I read you are running on AWS hardware?) there is no possible way, no matter how much code optimization you do, that you can achieve that rate.  The physics of network latency and especially AWS latency prevent it, not to mention the variety of user hardware out there.  The ONLY way to achieve that is to do one of the two things I listed above.

The pitfall with this practice is that it gives a false sense of efficiency to the user and gives them no incentive to improve.  That in and of itself harms your pool, but given the fact that you ALSO pay on stale shares further harms you because you are paying for shares that are worthless to you but NOT to the miner, again giving them absolutely no incentive to improve and costing you money in the process.  The bottom line is that it affects your bottom line negatively.  You may be using it as a loss leader to gain market share and that's fine; it doesn't cost your users anything and that's all they are really concerned with.  I do question, however, the long term viability of such a plan.  You are running at a minimum of .5% all the way to a 5% (or more, there really is no upper limit) financial disadvantage to other pools on a per block basis, since you have to cover the fees of the shares that have zero chance of finding a block for you.  Given that you are also running a straight PPS pool (and I commend your cojones), that's going to cost you even more BTC especially on long blocks.

If it works for you, great.  If it works for you long term, even better... I have doubts, but I can certainly be wrong about your viability in the long term.  In the meantime, it's great for the users, since they get paid regardless... The only fear I'd have as a user were on the day that you decide to close up shop you aren't too far in the hole to pay everyone out.  Hopefully for your sake that day is a long day off and you won't have to worry about it.

[BurningToad]

I haven't read ALL of this discussion, but are you saying that a 0.5% stale rate isn't possible?  On my ~400GH/s pool with a fairly standard pushpool (with patch to avoid duplicate work), and bitcoind with JoelKatz (sp?) patches, many users get below 0.5%.  The pool average is usually around ~1%, but here is a "top list" of users with at least 100,000 shares, sorted by lowest stale %.

"id";"share_count";"stale_share_count";"stale_share_count / share_count"
"345";"138222";"387";"0.0028"
"784";"125035";"363";"0.0029"
"55";"152146";"454";"0.0030"
"1133";"278451";"847";"0.0030"
"637";"160893";"491";"0.0031"
"482";"296248";"932";"0.0031"
"318";"318306";"982";"0.0031"
"580";"108635";"349";"0.0032"
"498";"646696";"2056";"0.0032"
"480";"179959";"571";"0.0032"

Sorry if I misunderstood your assumption though.  If you are saying a total pool average of < 0.5% isn't possible, I would probably agree, unless you only had a very specialized group of miners who spent time getting their setup right.  Then again, I dunno.  I've seen my pool's average for the day around 0.8% before, and I haven't done anything in particular to bring that down.

Oh, and as a fellow pool op, I'm also curious about the viability of 0% fee PPS.  It certainly is a great way to grow a pool though, as you can see by how fast you have grown.  Also, once people tend to join a pool, they tend to stay unless there are large issues.

[Inaba]

Yes, I mean an aggregate pool rate, not on an individual miner basis.  I thought that was assumed from the fact that I mentioned miners being "close by" on the network or other factors that mitigate network and hardware latency. Which is why less miners would be more likely to have a lower stale rate than a lot of miners with varying hardware and connections.  The larger the pool, the higher the overall stale percentage, everything else being equal.

Some people are on better hardware/connections than others and will achieve a better stale rate than others.

[BurningToad]

After reading some more, it sounds like he was talking about 0.5% invalid (for reasons other than being stale.)

[c_k]

Poolowner: please test cgminer, it is getting stuck often where it submits a share, loses communication, gets communication, submits a share and loses communication again in a seemingly never-ending loop