They're located in Chicago, IL.  Colocated with ServerCentral (nLayer).

Looks like this load balancing setup just doesn't work well enough.  I'll be making some changes to the servers this afternoon after fixing the earnings from last night.  Remove the round robin, and move back to having 2 servers [primary + 2 backups].  Users will be able to manually point at any of the 3 (2 of which should be as effective as a private server due to how many users use the default).

Please update your post again in a while with fresh stats.  These types of problems really shouldn't be happening especially if you're connceting directly to the IP.  Other than the obvious problems that can occur when 2 LPs hit in a short window.

For anybody that has an unusual number of idles/connection errors [not counting this morning when the servers rebooted], try the following:

Change your miner from btcguild.com:8332 to one of the following (this is the recommended order to try):

mine3.btcguild.com:8332 ([removed] by direct IP)
mine2.btcguild.com:8332 ([removed] by direct IP)

Should be fixed in the next 60 seconds.  One kernel tweak didn't carry over after rebooting the servers.

I'm going to have to do some work to calculate payout miners for some of that time.  It won't be 100% perfect, but anybody that is a dedicated miner should come out AHEAD of their normal 24 hour earnings.  I've made changes to the server logs now that we're on a stronger server [with significantly more HDD space], so if it happens again I can do a complete recalculation for the entire period with exact payment to the share.

Current plan is the following, unless I can pull some extra information from the partial logs:
  1) Calculate the average shares per hour submitted by each miner for the 3 hours before the problem.  Multiple that hourly average by 8 (downtime was very close to 8 hours).
  2) Take the shares submitted by each miner during that 8 hour time frame the previous day.

  Whichever number is LARGER, will be added to your earnings, at a 0% fee.

Pools are back up.  Something went wrong between the pool software and bitcoind last night, resulting in RPC errors between PoolServerJ and bitcoind.  This meant the pool was still generating work for an old block.  At the same time, the pool could not grab the current difficulty from bitcoind, so the shares weren't being logged to the payout table (which requires current difficulty to properly record the earnings).

I'll be working on a fix for the missing earnings today, and have it implemented shortly after returning from work this afternoon.

Just woke up, something big happened last night and I'm trying to find the cause right away.  Mining will be restored shortly.

Update:  Got the load balancer to forward HTTPS requests if you do not include 'www' in the URL.

Make sure you're including www. in the address (https://www.btcguild.com  not http://btcguild.com).  The load balancer is not currently able to redirect HTTPS requests [still working on a way to do that properly].

Can you give me more information? Miner software + version, location, speed of the individual miner.  I've been continuing to run my 6970s on the pool all weekend and the only time cgminer complains (and it's the most apt to complain due to how aggressive it asks for work at a long poll) is when there's two very close LPs.

And even when it complains, its still hashing on fresh work (it gets work with the LP, then complains about the time to get work because it asks for more IMMEDIATELY after, even though it gets the response well before it exhausts what came with the LP).

Our blocks now have the /P2SH/ in the coinbase.  The problem was the custom coinbasing in PoolServerJ to embed messages [we tag our blocks with "BTC Guild"].

Glad to hear its working better now.  Ideally everything was automatic, but when moving servers from Dallas to Chicago, things can show up you don't expect, given the number of active miners/configurations pointing at the pool.


I've found out why the pool wasn't properly voting for p2sh.  Due to using PoolServerJ which allows custom coinbasing, it was not automatically including /P2SH/ which the default bitcoind coinbase would.  There will be a small blip in your miners that shouldn't last more than 30 seconds, when I restart the pools to add /P2SH/ to our custom coinbase.

Thank you.  I was very happy with the move overall this time (compared to the multiple hours of downtime last time due to bitcoind taking ages to load up).  A bit disappointed in the load balance issues this morning, but I guess I was a little rusty since we were only running on one server for so long.

I finished tweaking the website/DB issues this morning, and it helped tremendously.  Page loads have been near instant all morning regardless of how much data is getting pulled.  That's good news, because I'm hoping to get to work on adding the PPLNS option later this month so miners will be able to pick between 5% PPS or 2% PPLNS (and swap between them with a single button).

Hah, thats why I get for typing out "total of 1 GH/s" then changing it to "400 MH/s each" without changing the G to M.  I wish I had 1200 GH/s of my own.

Just looked into it on the BTC Guild side.  I've upgraded the bitcoind, but it looks like PoolServerJ's internal workmaker is the problem (Bitlc also uses PoolServerJ).  It's making its own work and not including /P2SH/.  I'll have to get in touch with shadders to find out how to make it carry forward (I'm not the best with Java).

If you haven't done so already, restart the miner just to make sure you're on the proper DNS settings.  Also, a few fixes were done to improve performance in the last half hour.  Since then I've had 0 errors/delays/stales on my miners (3 miners, 2 on VPNs routing through different parts of the country, 400 GH/s each).

Working on a small update to the pool server since the DB server is not on the same machine as the pool anymore.  You may see a few disconnects (only about 1/3 users will be affected due to load balancing).

UPDATE (10:12 AM PST):  Pool changes are done for now.  Will be keeping an eye on performance (running a handful of miners over VPNs so I can monitor performance from multiple locations) for the rest of the day.

Website access needs to be https://www.btcguild.com (note the WWW).  Still trying to find some way for the load balancer to forward an HTTPS request to the webserver without that error.

Thanks!  I've been planning it for quite a few weeks, but you never know how a big move like this will work out in the real world.  All miners are on the new servers as of right now.  BTC shares that were submitted to the old server during the move have already been added back to the new server, and NMC shares will be added back in the next few minutes.  Only valid shares submitted during the move are being copied over (since those are all that matter for payout).

Payouts will be re-enabled in the next half hour.  Hall of Fame, performance charts, and auto payouts will kick back in around that time.  Unfortunately, the timezone issues mean the performance charts got really screwed up during the move, so they will be starting fresh.

EDIT/UPDATE:  Our bitcoind was updated during the move, and we are now voting in favor of BIP16.