Re: Mining pools list

[-ck]

For example, the time GHash.IO was under attack is clearly visible, and I'd be worried about EclipseMC.

Hi OOC

If you watch EMC's performance on blockchanges here http://poolbench.antminer.link/ you'll see they're consistently more than 15 seconds behind almost everyone, from the SPV pools to the full validation ones. That's gonna make their orphanage work overtime in addition to it being so small these days.

[organofcorti]

For example, the time GHash.IO was under attack is clearly visible, and I'd be worried about EclipseMC.

Hi OOC

If you watch EMC's performance on blockchanges here http://poolbench.antminer.link/ you'll see they're consistently more than 15 seconds behind almost everyone, from the SPV pools to the full validation ones. That's gonna make their orphanage work overtime in addition to it being so small these days.

Thanks -ck, that's interesting. They haven't reported any orphaned blocks so it seems to be mostly luck - although such bad luck that unreported orphaned blocks might be a better explanation for it.

[-ck]

For example, the time GHash.IO was under attack is clearly visible, and I'd be worried about EclipseMC.

Hi OOC

If you watch EMC's performance on blockchanges here http://poolbench.antminer.link/ you'll see they're consistently more than 15 seconds behind almost everyone, from the SPV pools to the full validation ones. That's gonna make their orphanage work overtime in addition to it being so small these days.

Thanks -ck, that's interesting. They haven't reported any orphaned blocks so it seems to be mostly luck - although such bad luck that unreported orphaned blocks might be a better explanation for it.

Unless of course they're actually submitted so late that even their own bitcoind rejects it which is far more likely than it ever being even recognised as an orphan. Their bitcoinds recognising block changes does NOT equate with whenever a new work template is sent out by their pool server(s) which could be much later.

[organofcorti]

For example, the time GHash.IO was under attack is clearly visible, and I'd be worried about EclipseMC.

Hi OOC

If you watch EMC's performance on blockchanges here http://poolbench.antminer.link/ you'll see they're consistently more than 15 seconds behind almost everyone, from the SPV pools to the full validation ones. That's gonna make their orphanage work overtime in addition to it being so small these days.

Thanks -ck, that's interesting. They haven't reported any orphaned blocks so it seems to be mostly luck - although such bad luck that unreported orphaned blocks might be a better explanation for it.

Unless of course they're actually submitted so late that even their own bitcoind rejects it which is far more likely than it ever being even recognised as an orphan. Their bitcoinds recognising block changes does NOT equate with whenever a new work template is sent out by their pool server(s) which could be much later.

I didn't know that was possible, but if it is then I wonder how many other pools are starting to have this issue? Slush has had some significantly poor luck lately.

BTW, who runs  http://poolbench.antminer.link ? You know if they have a freely available API of some sort?

[-ck]

I didn't know that was possible, but if it is then I wonder how many other pools are starting to have this issue? Slush has had some significantly poor luck lately.

BTW, who runs  http://poolbench.antminer.link ? You know if they have a freely available API of some sort?

This is why we've been trying to raise awareness that a lot of what is put down to "luck" is everything + luck since there's only the composite final endpoint of blocks and no way to measure the rest. There's no API for it, this is just cgminer set up on a vps in balance mode detecting block changes from many pools and simply laying them out on that website. The owner goes by the handle bitsolutions here I think.

[bitsolutions]

I didn't know that was possible, but if it is then I wonder how many other pools are starting to have this issue? Slush has had some significantly poor luck lately.

BTW, who runs  http://poolbench.antminer.link ? You know if they have a freely available API of some sort?

Yeah, there's not really an API right now since the code is a massive hack job to say the least, I'm planning a rewrite at some point but I'm already backlogged with other things so it's not a priority. It should be simple to parse out the data from the html though.

[organofcorti]

I didn't know that was possible, but if it is then I wonder how many other pools are starting to have this issue? Slush has had some significantly poor luck lately.

BTW, who runs  http://poolbench.antminer.link ? You know if they have a freely available API of some sort?

Yeah, there's not really an API right now since the code is a massive hack job to say the least, I'm planning a rewrite at some point but I'm already backlogged with other things so it's not a priority. It should be simple to parse out the data from the html though.

Great - mind posting an update when you're ready?

[bitsolutions]

Great - mind posting an update when you're ready?

Sure, btw what sort of data and format do you want from it? It would be pretty easy for me to create an API for the existing data on it if it's too hard to parse, one issue is that it's not really designed for long term historical logging right now.

[organofcorti]

This is why we've been trying to raise awareness that a lot of what is put down to "luck" is everything + luck since there's only the composite final endpoint of blocks and no way to measure the rest. There's no API for it, this is just cgminer set up on a vps in balance mode detecting block changes from many pools and simply laying them out on that website. The owner goes by the handle bitsolutions here I think.

Do you have a thread about it that I've missed?

From what you've written, the "luck" statistic (shares per block) can be influenced by orphans that aren't recognised as such (I guess some pools call them "stale" blocks?).

I'd like to make a list of things that can make luck worse, and just of the top of my head:
* incorrect reporting of orphaned blocks - either knowingly (not reporting orphaned blocks) or unknowingly (block is returned so late it is "stale" and bitcoind doesn't recognise it as a valid solution).
* various types of knowing and unknowing block withholding attacks
* a pool identifying both upper and lower case in a hash as different, so one hash can be submitted many times (eg GHash.IO)

What am I missing?

[organofcorti]

Great - mind posting an update when you're ready?

Sure, btw what sort of data and format do you want from it? It would be pretty easy for me to create an API for the existing data on it if it's too hard to parse, one issue is that it's not really designed for long term historical logging right now.

The data I'd like is what you have - the block distribution latencies per block, ideally from as many parts of the network as possible.

Your page is set up as a HTML table, so I can scrape that easily.

The downside of HTML scraping is that if you make any changes to the page format it often breaks scripts, so generally an API is better (CSV or JSON format is fine, the flatter the structure the better). But for now, no need for any API.

[bitsolutions]

The data I'd like is what you have - the block distribution latencies per block, ideally from as many parts of the network as possible.

Your page is set up as a HTML table, so I can scrape that easily.

The downside of HTML scraping is that if you make any changes to the page format it often breaks scripts, so generally an API is better (CSV or JSON format is fine, the flatter the structure the better). But for now, no need for any API.

Here's a simple api I threw together http://poolbench.antminer.link/api.php?range=5 you can specify how many blocks to go back by specifying the range parameter in the url, it will probably do funny things if you go back too far though. That's the raw non-analyzed timestamp data so that you can do your own analysis on it. You can ignore the poolnum parameter as that's only used internally.

Edit: I changed the formatting on the api so that it's now sorted and the time-stamps should now have proper precision.

[-ck]

This is why we've been trying to raise awareness that a lot of what is put down to "luck" is everything + luck since there's only the composite final endpoint of blocks and no way to measure the rest. There's no API for it, this is just cgminer set up on a vps in balance mode detecting block changes from many pools and simply laying them out on that website. The owner goes by the handle bitsolutions here I think.

Do you have a thread about it that I've missed?

From what you've written, the "luck" statistic (shares per block) can be influenced by orphans that aren't recognised as such (I guess some pools call them "stale" blocks?).

I'd like to make a list of things that can make luck worse, and just of the top of my head:
* incorrect reporting of orphaned blocks - either knowingly (not reporting orphaned blocks) or unknowingly (block is returned so late it is "stale" and bitcoind doesn't recognise it as a valid solution).
* various types of knowing and unknowing block withholding attacks
* a pool identifying both upper and lower case in a hash as different, so one hash can be submitted many times (eg GHash.IO)

What am I missing?

No thread, no. The message has been sparsely distributed across various threads at suitable times as we slowly try and raise awareness, but miners have a tendency to believe in simple messages like "0% fees, +X% merged mining" so it's hard to not feel like you're bashing your head against the wall when trying to educate people about it.

SPV pools could be working on false blocks and dead end forks as well as happened going V2->V3. You're on the right track about things that can make luck appear worse. The point is mainly how much work is wasted work - i.e. working on long since changed blocks or, much rarer but much worse, wrong blocks entirely. Slow pool software, poor scalability in chosen software/languages, badly performing bitcoinds, added overhead of waiting for accessory altcoin daemon block changes and so on all contribute to it. Merged mining is a classic as it creates the illusion of greater profit margins when whatever overheads they generate on bitcoin block changes almost certainly outweigh the pitiful profit of said shitcoins. There really is no way of quantifying these losses since they're all lost in the much larger noise of luck variance, and no pool is ever going to tell you what their average bitcoind block change latency is, what their share processing latency is, what their stratum template propagation latency is and so on.

[organofcorti]

Do you have a thread about it that I've missed?

From what you've written, the "luck" statistic (shares per block) can be influenced by orphans that aren't recognised as such (I guess some pools call them "stale" blocks?).

I'd like to make a list of things that can make luck worse, and just of the top of my head:
* incorrect reporting of orphaned blocks - either knowingly (not reporting orphaned blocks) or unknowingly (block is returned so late it is "stale" and bitcoind doesn't recognise it as a valid solution).
* various types of knowing and unknowing block withholding attacks
* a pool identifying both upper and lower case in a hash as different, so one hash can be submitted many times (eg GHash.IO)

What am I missing?

No thread, no. The message has been sparsely distributed across various threads at suitable times as we slowly try and raise awareness, but miners have a tendency to believe in simple messages like "0% fees, +X% merged mining" so it's hard to not feel like you're bashing your head against the wall when trying to educate people about it.

SPV pools could be working on false blocks and dead end forks as well as happened going V2->V3. You're on the right track about things that can make luck appear worse. The point is mainly how much work is wasted work - i.e. working on long since changed blocks or, much rarer but much worse, wrong blocks entirely. Slow pool software, poor scalability in chosen software/languages, badly performing bitcoinds, added overhead of waiting for accessory altcoin daemon block changes and so on all contribute to it. Merged mining is a classic as it creates the illusion of greater profit margins when whatever overheads they generate on bitcoin block changes almost certainly outweigh the pitiful profit of said shitcoins. There really is no way of quantifying these losses since they're all lost in the much larger noise of luck variance, and no pool is ever going to tell you what their average bitcoind block change latency is, what their share processing latency is, what their stratum template propagation latency is and so on.

So, your points about wasted work would be mostly to do with work lost on a change to a new block, and any found blocks being "stale" (too late to even be in an orphan race)? Say the time lost is on the order of ten seconds, that equates (on average) to a loss of ~ 1.7% of work, and about the same increase of necessary work to solve a block.

That could account for pools with slightly poorer luck that is only significant in the long term. Or could the percentage figure be much worse than that?

[Biodom]

Do you have a thread about it that I've missed?

From what you've written, the "luck" statistic (shares per block) can be influenced by orphans that aren't recognised as such (I guess some pools call them "stale" blocks?).

I'd like to make a list of things that can make luck worse, and just of the top of my head:
* incorrect reporting of orphaned blocks - either knowingly (not reporting orphaned blocks) or unknowingly (block is returned so late it is "stale" and bitcoind doesn't recognise it as a valid solution).
* various types of knowing and unknowing block withholding attacks
* a pool identifying both upper and lower case in a hash as different, so one hash can be submitted many times (eg GHash.IO)

What am I missing?

No thread, no. The message has been sparsely distributed across various threads at suitable times as we slowly try and raise awareness, but miners have a tendency to believe in simple messages like "0% fees, +X% merged mining" so it's hard to not feel like you're bashing your head against the wall when trying to educate people about it.

SPV pools could be working on false blocks and dead end forks as well as happened going V2->V3. You're on the right track about things that can make luck appear worse. The point is mainly how much work is wasted work - i.e. working on long since changed blocks or, much rarer but much worse, wrong blocks entirely. Slow pool software, poor scalability in chosen software/languages, badly performing bitcoinds, added overhead of waiting for accessory altcoin daemon block changes and so on all contribute to it. Merged mining is a classic as it creates the illusion of greater profit margins when whatever overheads they generate on bitcoin block changes almost certainly outweigh the pitiful profit of said shitcoins. There really is no way of quantifying these losses since they're all lost in the much larger noise of luck variance, and no pool is ever going to tell you what their average bitcoind block change latency is, what their share processing latency is, what their stratum template propagation latency is and so on.

So, your points about wasted work would be mostly to do with work lost on a change to a new block, and any found blocks being "stale" (too late to even be in an orphan race)? Say the time lost is on the order of ten seconds, that equates (on average) to a loss of ~ 1.7% of work, and about the same increase of necessary work to solve a block.

That could account for pools with slightly poorer luck that is only significant in the long term. Or could the percentage figure be much worse than that?

frankly, your own data (52 wk) has the answer if you look closely and considering ^^^.
my only regret is that i listened for far too long to explanations that included "poor luck".

[organofcorti]

frankly, your own data (52 wk) has the answer if you look closely and considering ^^^.
my only regret is that i listened for far too long to explanations that included "poor luck".

That's why I'm asking - could such luck be caused by block change inefficiencies, or something else? I think it's unlikely, tbh, and more likely something else is going on.

[-ck]

frankly, your own data (52 wk) has the answer if you look closely and considering ^^^.
my only regret is that i listened for far too long to explanations that included "poor luck".

That's why I'm asking - could such luck be caused by block change inefficiencies, or something else? I think it's unlikely, tbh, and more likely something else is going on.

Well, the worst cases I've seen of block changes, some pools have missed them entirely. Even so at most it would be say 30 seconds of lost work. You've listed another juicy possibility though in the block withhold idea, which we bring up about every 18 hours on these forums, and still have no answers for... so nothing new there

[organofcorti]

Well, the worst cases I've seen of block changes, some pools have missed them entirely. Even so at most it would be say 30 seconds of lost work. You've listed another juicy possibility though in the block withhold idea, which we bring up about every 18 hours on these forums, and still have no answers for... so nothing new there

... and there could be other vulns (like the GHash.IO one) that aren't directly block withholding, more like share multiplication. But yep, no answers, just lots of guesses and some FUD.

[Laviathon]

I just realised that it shows I don't pay tx fee but we do.  Could you fix that for me.

[organofcorti]

I just realised that it shows I don't pay tx fee but we do.  Could you fix that for me.

Done.

[Laviathon]

I just realised that it shows I don't pay tx fee but we do.  Could you fix that for me.

Done.

Thanks!