They haven't been working properly due to no longer pulling stats live from the other servers.  Made some big changes to how last shares are handled which is why My Account/API load almost instantly, whereas before they had to pull the data live from all servers which meant the load times were 4-6 seconds.

They will be back shortly (within the next 24 hours).  I got called in to work today (I normally work 4 days/10 hours), which has pushed back my schedule a day for the fixes/updates this weekend.

Not yet, I've been a bit busy today with a new botnet that surfaced on the pool last night.  Its a different one from the last two.

Looked into the invalid after waking up, it was a standard run of the mill invalid.  Our first "real" invalid in a while (the other 4 most recent were from when US West had stability issues during LPs).  Came from US East, no odd signs in the logs showing anything other than somebody else on the network came up with a solution slightly before US East sent out its solution.


Regarding instant payouts:  Yes, technically INSTANT payouts will be gone if there is a 1 hour delay is added.  It still doesn't have to wait for 120 confirms, and it still allows you to receive rewards from invalids.

Fixed the bug.  Optimized the database to significantly reduce size and speedup SELECT queries.  Ended up changing block_number (not block ID) to an unsigned value momentarily, then set it back to unsigned after remembering block_number is set to -1 on invalids.

The worker stats froze because the sync script was failing to connect to US Central.  The connection issues on Central are outside of my control, they're a network wide issue, not a server specific issue.  Once I get the reset shares/last shares restored I'll split the sync script for worker stats so an individual server won't stop the rest of the workers from updating.

I will likely implement the stat delay, since the response to possibly changing payout scheme from proportional is quite mixed.  Essentially "This Round" would be removed from worker stats.  You'd have:

Est. Speed, Total Shares (Since Reset), Total Shares (All time), and Last Share Received

Reset shares and last share received are being fixed tonight/tomorrow.  I will hold off on implementing a stat delay until those are back in place to allow reasonable worker monitoring.

Received an email that finally gave me figures that show a significant level of pool hoppers in the pool could in fact be detrimental to the rewards of regular users.  Unless its a very large portion of users hopping the effect is very small (enough to be called luck noise), but it is there.  However, as mining is evolving, it may only be a matter of time before a client comes along with this capability built in.  If that were to happen, then it could create a noticeable impact.

I'm evaluating the two options I see that would "solve" the issue.  1 hour delay on stats, or SMPPS.  I was very anti-SMPPS at first, but the email I received included a comparison of how the payouts would have looked over a 72 hour period where pool hoppers were involved, using proportional vs SMPPS.  I'll be giving it some thought this weekend after I get miner idle emails back up and running.

The provider of our server uses Softlayer.  Softlayer was under attack by a very large DDoS and a LARGE number of servers that use Softlayer in that area were effectively offline (significant packet loss causing slow/unreliable connections).  As for your miners not reconnecting, that's something you'll have to ask the miner authors about.  The pool doesn't do any funny things with connections, its very standard pushpool in terms of authorization/getwork routing.

Our pool is purely proportional payouts, meaning you'll get paid (Your Shares / Total Shares) * 50 BTC.  The timing of your share submissions, or turning your miner on/off have no negative effects other than the fact that you're not mining.

FYI, the idles I'm sure almost everybody experienced just now were the result of overall bitcoin network finding 4 blocks in 1 minute and 13 seconds.  That many long polls that frequently can do a number on bitcoind's getwork code, even with all the optimizations we've had added, not to mention the volume of network traffic.


EDIT:  I take it back, apparently Super Server (US West) had no issue on that spam of LPs!

FYI:  US Central's problems are not a software/hardware problem on our end.  According to IRC there's a major DDoS hitting Softlayer right now, and MANY websites are being effectively put offline.

Jine:  Isn't it funny how everybody starts questioning the functionality of the pool during bad luck, but never bother to ask if the pool is "really solving all these blocks" when having great luck? 

The point isn't that pool hopping isn't more money.  It is.  Putting your shares into pools with the shortest round is going to make you more money.

But my point is that in terms of BTC/minute, your payouts from BTC Guild are the same no matter what pool hoppers are doing:  Staying in the pool the whole time, jumping to another pool, or not being in the pool at all.

Technically, pool hopping does hurt the pool because long rounds take even longer than they should, so a higher percentage of your pool's time is functioning at a low BTC/minute from bad luck rounds.

However, this argument is only valid IF the pool hoppers were going to stay on the pool in the first place.  My argument is that nobody that has the setup to pool hop would bother being on the pool in the first place if we had a stat delay/score system.  There are many other proportional/real-time pools out there to hop on.  As such, pool hoppers not being on the pool at all will actually mean the long rounds do last slightly longer than if the pool hoppers were on them for the first couple of minutes.

Please point out why another pool's payouts during that time affect the normal BTC Guild user.  You quoted one of three scenarios.  In the end, the non hoppers make the same BTC/minute whether the hoppers were involved for all of the round, part of the round, or none of the round.

I'm listening for arguments/flaws in my scenarios above.  If I can get concrete proof that pool hopping actually -hurts- the pool payouts for non hoppers, I will introduce a stat delay (I refuse to implement any score based/SMPPS system, they are not user friendly).

As far as I can see, the effect of pool hopping is it increases the duration of long rounds.  But as I pointed out above, the duration of long rounds is still shorter -thanks- to the pool hoppers being there for a portion of the round.  If the pool were not hopper friendly, the unlucky round would have been even longer because the hoppers would simply ignore the pool and go somewhere that they can hop without penalty.

Here's a theoretical scenario.  I'm using simple math, just to make it easy to follow:

This scenario assumes the following:
  10% of the pool is hopping
  The total pool speed WITH the hoppers is 100k shares per minute. (So 10k/minute from hoppers, 90k/minute for "legit users")
  The hoppers are swapping at 1 million shares (AKA: 10 minutes/100k shares)
  The hoppers are efficient enough to join rounds/leave rounds at the exact start of a new round/1m share mark on a long round.

Scenario 1)
We solve our first block at 500k shares.  Pool hoppers get 5 BTC (10%), rest of the pool gets 45 BTC.  So pool hoppers get 1 BTC/minute, rest of the pool gets 9 BTC/minute.

We then have a long block, 3 million shares.  The pool hoppers contributed 100k, or 3.3%.  Hoppers get 1.66 BTC, the rest of the pool received 48.34.  Because the last 2 million shares were running at 90% normal speed, it took 22.22 minutes to finish the block after they left, or 32.22 minutes total. The pool hoppers got 0.166 BTC/minute (10 minutes of work).  The rest of the pool got 1.50 BTC/minute (32.22 minutes of work).  What the pool hoppers earned on other pools during the other 22.22 minutes is irrelevant to BTC Guild's users payouts.


Scenario 2)
Now lets say I implement a system that makes it impossible to hop.  For example a 24 hour stat delay, meaning they could NEVER know if we were having long rounds (unless we were so unlucky it lasted 24+ hours).  There are plenty of pools out there to hop on.  If a pool hopper is looking for max rewards, they simply WONT USE our pool.  As I stated in the IRC chat:  "Hoppers gonna hop.  If they can minimize the effect of bad luck, they will ignore a pool that doesn't allow them that opportunity."


So in the 500k share block, we take 5.55 minutes to complete the block, giving the normal users 9 BTC/minute for their time.
In the 3m share block, we take 33.33 minutes to complete, giving normal users 1.50 BTC/minute for their time.


Scenario 3)
Pool hoppers say:  "I guess I can't hop, I'll just STAY in BTC Guild".  That 3 million round then takes 30 minutes.  Pool hoppers contributed 10% and got 5 BTC, the rest of the pool got 45 BTC.  In this case, hoppers made 1.666/minute on the 3 mil round, and non hoppers made 1.5 BTC/minute, exactly the same as before.


As you can see, the NON hopping users will make the same BTC/minute:  
  Scenario 1 vs 2 - 500k block) Hoppers in a pool for a whole a round vs not in the pool at all.  Non hoppers make 9 BTC/minute either way for that round.
  Scenario 1 vs 2 - 3 mil block) Hoppers in the pool for part of a round vs not in a pool at all.  Non hoppers make 1.5 BTC/minute either way for that round.
  Scenario 1 vs 3 - 3 mil block) Hoppers in the pool for a whole round vs only part of a round.  Non hoppers make 1.5 BTC/minute either way for that round.


In the end, the worst case scenario is that pool hopping makes a bad round take longer to complete.  However, a pool hopper would not be participating at all if the pool does not have the ability to be hopped into/out of.  In which case, NOT having the pool hoppers will make that round take even longer than it would with the hoppers.

How did you get into my house to find my miner control switch!

Update on the idles earlier today:  Host has modified the DDoS filtering to allow a higher packet flow before it trips on our mining server's IP.  The problem seems to have stemmed from when all 4 pools triggered a long poll within 1-2 seconds (remember that bitcoind = p2p network, and has to write to disk when a new block is found, causing minor variations in each bitcoind knowing the block chain has moved).

When all 4 subservers triggered the LP within about 1 second of each other, it launched out the new information to all miners at once.  The server is averaging about 3.5k workers right now.  Some workers are actually 2-10 miner instances, some have even more for the people who setup computers at their office to mine while idle.  Each one has its own long poll connection, and its own work queue.  The LP pushes the new work to everybody, and people re-established connections.  Many miners will request at least one additional piece of work at the same time.  This is on top of the usual getwork/share submission traffic.

The end result was a rare chance of the server triggering the DDoS filter which adapts to the traffic flooding in.  Those who were around for the old US West know that this can get much worse with how frequently miners try to reconnect when a connection is rejected.  This is why some people were still mining happily and others were disconnected.  The DDoS filter knew certain connections were "good" because they weren't spamming for connections.  The people that got disconnected however were repeatedly hitting the server, and the filter was keeping them locked out during the "attack".

I've been pressing my button to toggle the luck switch but I swear it's not working.  I guess those guys didn't install the switch properly!  I'll have to drive there myself to flip it >_<

The problem is somewhere in the ISP's network based on traceroute information.  The traceroute dies before it even reaches the server.  It is possible that US West has enough IPs pointed at it that the long poll is causing enough packets to flood in/out in a very short timespan and its tripping the DDoS filtering.  Or it could simply be an unrelated hardware issue at the ISP that needs fixing.  Will know more once I get a response.

I know for the miners out there it sucks to see these dips of efficiency due to idles (luckily not too long), but the good news is that this time the problem is not in server hardware or software configuration.  If this is an accidental trigger of the DDoS filters,  it may be as simple as the ISP whitelisting our IP from the DDoS filters unless we ask for it to be turned on.

Just forwarded a whole set of traceroutes to the host for US West.