Unfortunately luck is luck. Doesn't matter if we're at 15PH or 50PH. All it would do is statistically reduce the length of time between a painful long block. However what is more important is, whether your pool infrastructure is ready for the massive growth? There is a reason why Kano wants to keep his pool within a certain network size. What is needed to support 20PH is not the same at 200PH. Look at Antpool, the refresh on the site is usually about 15-25 mins delay on each block. Quite frankly I'm not even certain if Kano was prepared for this explosive growth we had going from 5PH in Dec to 40PH in just 4 months. That's 800% growth!! It's hard to predict that type of growth. There is no perfect pool, I don't care what anyone says. There will always be features that 1 person want vs another and no pool can satisfy everyone. Quite frankly a manual payout system may work well when you're managing a small pool but it's definitely not the best system when you're much larger. I was definitely not happy when everyone's pay was held for so long due to some investigation delay on some portion of small crappy miners. Some may not care as their payout is small, but when you're running a big operation, you need consistency. With growth there will also be lots of other issues that comes with it (hardware upgrade, servers getting overloaded, network issues, DDOS attack etc). Planned growth is good, unexpected boom may cause more harm then good. Now I don't know what the staffing requirement is from 5PH to 40 to 100, etc. What I can see recently with all this growth is more frequent restarts for upgrades and code changes. Kano have to deal with crap miners that consumed his entire day, etc. You also see extra investments in more proxy servers, dealing with issues on disconnects, etc. It can be a lot of work and plenty of things can go wrong or is not planned for.
Based on current CPU, current CKDB code limit performance on the current hardware is probably about 6x the current pool i.e. about 180PHs
However, when we hit 40PH the CPU increase wasn't linear, is was less than that, so I'd expect the current CKDB code wouldn't be an issue until around 200PH or more.
As -ck mentioned above, that's unrelated to mining as such.
-ck's ckpool mining code could indeed handle EHs mining on the current hardware.
There are issues if CKDB got to that limit on the current code, but I also know exactly where those issues are and that is the next priority change.
I've had quite a few changes I've been working on since I last made the comment about when I'd work on that, but they have indeed been related to the security of the pool since it has grown, not the size of the pool in performance.
I certainly don't think the fact that I've detected a miner that has withheld blocks and earned 46BTC as trivial.
In fact you will find there is no other pool that has done that so early, before.
Slushs's withholding was picked up at the absolute minimum after throwing away 250BTC to the miner.
I suspect it was a lot more than that.
You also need to consider how long a miner takes to mine 10+ blocks ... to get an idea of at the least, how long it was going on before he detected it ...
The manual payout system is not like I sit there and type numbers into a spreadsheet or something like that.
It's simply that I manually run the pool security checks and payout generation scripts, that at the end, produce each payout transaction.
I then simply run each transaction script I've generated once the block is mature.
The security checks I run, at least daily, usually more often, I guarantee are well beyond anything you will find any other pool does.
The delay that occurred was to work out what was going on with the withholding, add code changes to CKDB and determine if I should put a hold on which accounts.
Yes the total saved ended up being less than 10BTC, but the changes are in place already so I can now check that easily.
... and of course if I just let it got on, that would have been, with the current belligerents, at least 10BTC a week ... until it grew due to no intervention.
I've processed more than 20TB of share history (yes I have every share logged ...) and generated a database history of all shares over 1G since February last year (I'll do the previous 5 months back to when the pool started some other time)
CKDB now records them all individually (>=1G) now searchable in the DB so I can run miner performance stats even on block ranges.
(of course CKDB has shift summarised stats of all shares, but this is all the actual individual shares >=1G share diff)
I use SQL to do this so that it doesn't affect CKDB, since the miner share history is, of course, now over 18 months of data - all in the DB, but CKDB doesn't load it all into RAM, only recent history, since it doesn't need to know more than a few weeks of history.
I have, as I've stated before, already had normal block withholding data available, just nothing at this low level to detect it way earlier, that no other pool has done before now.
This is why I say it's entirely "my opinion", no technical proof behind it, but it seems like the pool was operating much more efficient when it was around 25PH. When my partners and I were pushing the pool to 40PH with our 7500TH to get us thru them nasty long duration, Zach's tool would alert me 5mins on the block and I had to wait 5 mins until it was shown on Kano.is. It was never like that before as it was always instant.
Sorry, that's not quite correct ...
The web site on the pool shows blocks immediately after they are found and will always do that before anything else on the planet, except under one condition:
That one condition is during a CKDB restart.
That's currently about once every 3 days at the moment (and there'll be another one in the next 4-6 hours)
That has no effect on mining, only on web/API access.
There's only been one block recently that I recall being during a restart - and yes if the web site data isn't available at that time due to a CKDB restart, indeed you will have to wait for the restart to complete, to see it.
If Zach is using kano.is to get that data, you'll never see it there before here.
If he is using some other source to get that data by analysing a local bitcoind, then the only time he'll know about it before the kano.is web site will know about it is during a CKDB restart.
There's other things I noticed on certain delays and stuff, but it's not significant enough to cause concern. Luck is luck, doesn't matter if the pool is at 10 PH or 100PH, it all boils down to how long you're willing to wait for the nastiest long block duration which only gets worst with higher difficulties. However I rather take no growth then growth that is not well planned in advance. Outages, disconnects, or orphans can get very expensive. That orphan we recently had was super super expensive during that dry spell where every block count.
P.S I never take anything personal so don't worry about that. Everyone is entitled to their opinions. I just prefer to states the facts when I can and highlight where it's my own personal opinions.
That orphan we got was almost unavoidable.
The timing was a couple hundred milliseconds.
We have already recently won an orphan race where we were not first to release the block, but due to our connectivity we were able to get the larger pools to work off our block rather than the other pool's block.
In the recent orphan case we lost, I suspect it was a solo miner in china, or a small not yet well known pool in china, who pushed their block to the chinese pools so it took longer to get to us.
Remember, orphans are random just like everything else, and the pool's history shows it is doing way better than the other pools.
3 orphans (and 1 stale I flagged as an orphan) in 668 blocks.
One recent orphan block, that being less than 46BTC, and certainly less than 250BTC
... and FYI ... I don't mind you posting that and I'll not delete such posts, but I will make corrections as I've posted to state pool facts
