What address were you tracing and using to measure the packet loss.  I need something concrete to send to my host to figure out where this issue is coming from.  Servers are showing NONE of the usual idle signs.

US West just had a bit of packet loss resulting in some idles/disconnects.  At this time I can't say if it was the server or the ISP having the issue.  Absolutely nothing unusual on any of the pushpools or pfsense.

With how smooth things have been going today, I may squeeze in a little time to get the Hall of Fame back in, with Top 100 Donators and Top 100 Block Finders.

A swing that wide is not uncommon when comparing two pools over a 24 hour period.  The other day BTC Guild had -39% luck.  BTC Mine had an insane +90% luck.  You would have made significantly more at BTC Mine than BTC Guild, even with a 2% fee vs 0%.  Pools are NOT immune to short term variance.  They are simply more likely to provide stable variance over a reasonable time frame.  You can see at BTC Guild our variance over the last two difficulties was +0.2% and -0.4%.  But those numbers included swings from -30 to +30 if you were to look at any given 24 hour period.

BTC Guild switched to paying out all 8 decimals.

The funny thing about mining profitability is how electricity is priced (at least in most of the US).

Large scale miners pay more per kWh than hobbyist miners.  In MOST of the US (I have yet to hear of any state/area that it is different), you end up paying more for electricity as your usage grows to certain threshholds.  In CA, the base level is around 13-15 cents per kWh.  When I was mining with 7 rigs (14 video cards), almost all the electricity usage was classified in top tier due to high usage for the month.  Top tier pricing is 42 cents per kWh.

The overhead becomes significantly higher once you try to expand beyond a few cards in the computer you already had turned on 24/7.

This is why I had that notice on the site header for almost a full week after the leak.  If you used the same password: CHANGE IT!  If you were using the same password on multiple sites, odds are it wasn't as secure as you think (unless it was something you had to copy/paste...but thats a whole other discussion about poor password security).

Clear your cache.  The security token is a cross site form request exploit prevention to prohibit a malicious site from tricking you into requesting payouts, changing wallets, emails, or passwords.

I'm shopping around for a host that can offer hardware similar to what is in place in US West.  My goal is to consolidate us down to 2-3 servers with roughly the same setup, each one capable of holding well over 2.5 TH/sec, since we were around 3.2 TH and still growing before the DDoSes took us down.  The second server is being shipped to Texas tomorrow and will eventually replace US Central & East.  So far the only comparable EU servers I've found cost nearly twice as the US ones do, which is funny considering the old server hardware was significantly cheaper in EU.

For those afraid of losing East and Central - US West was a project that I've been preparing for since we hit 2 TH/sec a few weeks ago.  There is no way to iron out these bugs in a test environment, due to the nature of the problems (block solving, long-polling, and a massive number of unique IPs hitting a source-based load balancer).  The kinks have almost been completely ironed out based on the last couple hours (and block solves), and the new Central wont be replacing the old servers until I'm positive it can handle the load as well or better than the current servers.

Worker stats are now in alphabetical order as they were before the DDoS.  Not included on the timeline is the ability to have a resettable share/stale counter rather than just round/total.  Reset stats should be back on the 14th.

I just tweaked the load balancer (which is what caused the idles and disconnects), and when we found a block I only saw about 3 seconds idle time, from the pushpool instance that actually found the block.  This time it didn't kill off the LP connection/server connection, just a short idle spurt.  I'm working on finding out if its the LB or Pushpool that is causing the final small delay.  It may be pushpool hanging for a moment while it opens up SQL connections with the other servers to publish the new block.

Timeline for the rest of the week/weekend can be found at: http://www.btcguild.com/timeline.php

Stales are indeed GREATLY reduced (they're not gone, and I'm sure someone will come in complaining about 91.13% stales right after I go to bed).  The load balancer is now functioning far better on US West.  It still seems to be getting an idle when the server manages to find a block, but it only lasts about 10-15 seconds based on monitoring my personal miners.

Just changed the pushpool configurations on US West.  Spoke with jine eariler, and there is some unknown bug in the Longpoll Keep-alive patch, which could very well be the culprit for a number of the problems today.

The next step if this doesn't improve stability is rolling back to pushpool and bitcoind stock code rather than the RPC keepalive/built in long polling versions.  Running out of items to cross off the list of trouble makers.



UPDATE:  ~50 minuets later.  497 shares on my phoenix miners (the "worst" for stales), 0 rejected.  0 idles.  0 long poll disconnects.  Looking promising.

The idles when we find a block are the root of the invalid problem.  I thought it was related to the calculation script, but I've profiled the time it takes to complete the different steps and the entire thing only runs for about 5 seconds, only about 1 second per server.

I'm thinking this is related to a bitcoind patch or the long poll keepalive patch.

Unless it's broken, in which case that means I'd be losing about $200 every 5 minutes on invalid payouts...

I ran a small test run (added a 5th pool to the load balancer for about 3 minutes) using testnet.  The problem is either fixed, or NOT directly related to the publishing of new blocks.  Won't be able to verify until we find a block on US West.

Stales went way down on US West with the patch about an hour ago.  Still having a very odd issue where the load balancer is spiking in CPU usage when running a block calculation, which is why you're getting idles.

I'm working on figuring out what about the calculation script is creating the problem.  This is my absolute top priority, because we just got screwed out of blocks (2 invalids in a row) due to the server essentially going offline before it was able to report the blocks on the network.  Same thing happened with the 2 invalids a couple days ago.

Made some changes to the DB and the new block code to hopefully fix the idles at LP.  This is why a lot of you probably had idles in the last 5 minutes.

Adding some timing profile code to the end of round calculations to figure this one out.  The queries being run on the servers should not be tying up the resources long enough for idles, but looking at my logs last night it seems like all but one of my idles overnight was due to a round ending.

The stales improved quite a bit overall, but they're still higher than normal.  It seems to be related to the LP taking longer to push out than it should, even though the server has already moved on.  Still optimizing, but in general the new server is performing well (No crash overnight, the idles have been identified as clustered around end of round calculations, and stales are better with a trail to why they're higher than they were earlier this week).

Hopefully the latest round of updates fixed the stales.  This afternoon I went back and rolled in the patches to pushpool that I had been slacking on, something went awry on the rejects due to roll_ntime.  Disabled it as jine did on his pool and things are already looking much better.