I'm waiting on a response once they believe they've fixed it (which they seem to have done, they tested a software patch on another pool and did solve a block).  My understanding is that the group of users were running custom gear, but I have asked them for more detail (chip manufacturer, software kits if any).  They did NOT have problems until the difficulty passed 4.2b, and that was proven by looking at their test runs when they used to mine on Eligius.  They moved to BTC Guild in early April because they had fewer connection issues with our servers.

It has been 4 weeks since they started mining on BTC Guild.  So unless there are other, yet unknown, hardware issues, only the last 1 month of luck (at most) is the fault of this group of miners.  And up until recently, there was NOT enough data to rule out variance.  

EDIT:  People can point out that action was taken after they started non-stop posting in the thread, and they can continue to do that all they want.  People will always try to assume a causality when something happens, ignoring the fact that action was going to be taken whether they said a word or not if something was actually broken.  I began talking other pool operators and asking if they had noticed any miners with unusual luck since we were all trending downward.  We started discussing possible causes (hardware releases, the 32-bit overflow point, etc.).  But at the time we had nothing firm, and there was not enough data to rule out "luck happens".  That was only 3 weeks into this group starting to mine on BTC Guild, and that was the first point where the 1-month luck interval began to take a significant dive.  Even at that point it was NOT clear if it was simple variance.  I've been running this pool for over 3 years now, and big swings were not unusual.

PPS was turned off March 15th.  3 WEEKS before this miner even started mining on BTC Guild.  PPS was turned off after discussing the IRS notice published about a week prior with tax accountants and lawyers about the classification of pool payments and reporting requirements that PPS may have required in contrast to distribution methods that are purely based on what miners have produced rather than a pay per unit of work method.

As you edited, yes, 522067 isn't in any way showing signs of a withholding attack/bug.  They started mining at much higher difficulty than many other miners, so obviously their solved blocks vs shares ratio is much worse than some of the other top users who were mining at difficulties significantly lower than it is when 522067 started.

As of right now, this doesn't appear to be affecting publicly sold miners, at least not to the same degree.



Yes, basically.  Luck is measured by shares vs network difficulty, but given constant speeds, the luck on each individual block would be comparable (6 hour block on one pool would be a 2 hour block on a pool that's 3x faster).  This is why faster pools level out to neutral luck faster.  They spend less time on bad blocks, but also less time on good ones.  As a result they simply have more blocks in a given time frame, which means they're more likely to be close to neutral in that time frame compared to a smaller pool.  Meanwhile, a smaller pool in that same time frame may be much lower, or much higher than neutral.

There's 2 possible break points.  There's ~2.1b and ~4.2b.  One is for signed 32-bit, one is for unsigned.  BTC Guild's luck took a sharp turn starting ~6 weeks ago.  That is right in line with the network difficulty passing 4.2b (March 24).

Luck for BTC Guild and Eligius both started trending negative a little earlier than that.  Roughly February.  However, this wasn't bad enough on either pool to simply rule out variance for that time frame.  For BTC Guild it was ~6% negative in March.  For Eligius it was ~5% negative.  However, February is when 2.1b break point (Jan 24th adjustment put diff over that amount) would have triggered.

Right now the largest users that had definite anomalies in their shares vs blocks solved have had accounts frozen, and one group has contacted me and is looking into it.  If they come back and it is a definite 'Yes, something was broken in our software/firmware/hardware', then at least the problem is identified and will be fixed.

If it was a custom proxy, yes.  As I posted, my personal believe is this problem would be in software or firmware.  Hardware is possible, but doesn't quite make sense given how barebones a mining ASIC *should* be.  My bias points me towards firmware/controller software rather than mining software since that is where it makes the most sense.  However, if somebody is running *custom* software or a custom proxy, that would be another possibility for where the flaw lies, assuming there is one somewhere.

Didn't say they were as efficient

1 million 1 MH/s miners are just as likely to solve a block as 1000 1 GH/s miners or a single 1 TH/s miner.

There's no reward to recalculate.  Nothing was found other than 5 Kittehcoin blocks (which were paid) in that time frame.

Yes, I remember something about it, I believe it was Meni.  The problem is, as you said, it would require a hardfork.  Even if we had proof that this type of attack was happening to explicitly kill off the ability for miners to pool together without incurring huge losses, it would ever be implemented.

If it becomes widespread, the obvious solution would be going to invitation based pooling where miners can be vetted before receiving payments.  The problem is this would completely kill the ability for small miners to join a pool since they would be too slow to prove that they were going to provide valid blocks.

If the problem is related to the difficulty going above the limits of a 32-bit variable, it may not have anything to do with *recent* hardware releases.  It could be OLD hardware.  It would likely be firmware or software in the controller for the hardware.  (See edit for more info).

This would explain why these accounts were not noted the last time I did a pass.  I was looking for active withholding previously.  Accounts with significantly low block submissions vs shares submitted.  Last night's pass was specifically targetting shares vs blocks over the last 6 weeks, eliminating the chance of past luck making up for recent shortfalls.


EDIT/CLARIFICATION: The hardware itself has no reason to actually know the result of its hashing outside of diff>=1 (or >=1024 in some newer hardware I believe).  It's probably not in the software since *most* ASICs do not use custom software, they use cgminer or bfgminer.  So the point in the middle that handles communication between the hardware and the software is the most likely culprit.

I froze a few accounts last night that had started showing unusually bad block solving rates over the last 6 weeks, even if they had previously not had bad luck, or did not have abnormally bad luck over the lifetime of their account.  One of those users has contacted me and we're now going to go over their HW/SW configuration to find out if something is busted.

The pool luck only took a drastic turn ~6 weeks ago.  And unfortunately, you can't tell how abnormally bad luck is until you have enough time to identify a clear pattern.  2-3 weeks at 15% of the network is not enough to call 90% luck bad enough to be suspicious, so it was not clear if it was abnormally bad or just bad until ~2 weeks ago, when we had a solid month at ~90%.  Even that is not enough to rule out variance for certain.  Since then, the 1 month time frame is in the 80s, which is even worse worse.

6 weeks ago is (roughly) when difficulty surpassed the limit of an unsigned 32-bit integer (4.2b).  It is very possible that this is the cause of it, but until I know hardware/software specifications of these users, it's uncertain.

I like how the proposed fixes are all related to minimum order sizes rather than the obvious fix:

STOP ROUNDING TO THE NEAREST CENT!  You already support 8 decimals of precision on BTC.  Do it for fiat, and simply round down a user's available balance to 2 decimals when they want to pull money out.  Professional sites have done similar things for years.

Hell, even inept Karpeles at MtGox knew to do it that way.

As usual, the second I go to bed something breaks, and not in a way that will trigger an alarm to wake me up.  Somehow the pool got stuck on LOT last night, but it was trying to solve a block in the past rather than the actual network block.  I've disabled LOT now.  I'm trying to see what actually happened, though the debug.log is not being particularly helpful.


EDIT/UPDATE:  The reason shifts got "stuck" on Calculating is the PPLNS display simply assumes any open shift with 0 blocks paid hasn't been calculated yet.

BTC Guild is smaller (relative to network size) than it was months ago.  The days of a public facing pool being 25%+ of the network are over, unless they have an obscenely large private farm propping it up.  Private mining entities now make up ~40% of the network, leaving only 60% left to fight over.  As a result, shifts are no longer setup around 15-20x difficulty, they're setup to be roughly 10x difficulty.  This means bad luck shifts are no longer 8-10 blocks.  A neutral shift right now is 8.75 blocks (meaning 9 blocks or more in a shift is positive luck).

This is one of the things I've been discussing with the other pool operators.  The bad luck trend for BTC Guild and Eligius seemed to start around the time difficulty crossed 2.1b.  This is the limit of a signed 32-bit integer, one of the most common numeric variables used for programmers.  It's quite rare that people need > 2.1b (4.2b for unsigned).  The difficulty, growing exponentially, ramped up to this number extremely quickly.  I realized that BTC Guild's internal reporting used a signed 32-bit int for displaying difficulty on a status screen.  I wasn't sure if the program would crash or simply loop around, so I did server maintenance around ~1b diff to update it to use a double variable instead.

The problem here is that we've been unable to identify any hardware that is broken in this manner so far.  It doesn't make sense that such an error would even happen at the hardware level, and most ASICs use cgminer/bfgminer, which we know is not the source if there is a problem of this nature.

Yes, fascinating that I deleted the posts that contributed nothing constructive, and decided to imply that I simply don't care, or am too lazy to do anything, and also include a dollar figure for income that was pulled directly out of your ass as well.



This.  Additionally, you can't send a "prove you're not evil" work packet to miners in stratum.  Every miner has a unique ExtraNonce1, so they will never overlap their hash results.

At least at first it will be DOGE and LTC only, but you will be able to run auto-switch between those two.  There's a chance another scrypt coin gets added, but for now those are the only two planned.  They're simply the only ones that are worth mining the majority of the time.

Don't recall seeing "low difficulty" anywhere.  The miner side difficulty has no influence on whether or not their hash will solve a block.


EDIT:  NVM, saw it.  My point above still stands.

Luck is not a zero sum game.  The difficulty can continue to rise even if the entire network was having bad luck, as long as the growth of new speed is greater than the deficit caused by poor luck.  Work audits are not possible with stratum, since each user has an independent ExtraNonce1, meaning their nonces will create radically different results than another user's, given the same template and ntime.  Additionally, a truly malicious withholding attack of large size is unblockable since they could simply make new accounts.  If they have enough speed to cause noticeable harm to a pool's production, and enough money to literally throw it away (since doing a withholding attack against PPLNS is costing them money for every block they withhold) they have more than enough money to proxy that attack through new IP ranges and accounts.