Tomorrow afternoon I will be switching pools over to difficulty 2 shares.  I will also be changing the way the pools are handled until I can work getting multiple servers in a single datacenter to do -real- load balancing (2+ pool servers behind a frontend that relays connections to the pools).

I'm posting this on the website as well:

  PLEASE POINT YOUR MINERS TO ONE OF THE FOLLOWING:
    uswest.btcguild.com
    useast.btcguild.com
    nl.btcguild.com
    uk.btcguild.com

Miners pointed at btcguild.com will be forced into one of the new servers I'm putting up.  Miners pointed at eu or us.btcguild.com will be forced to another server.  This will likely overload a node until people move to a proper URL.  After 48 hours I will be removing the eu and us sub-domains, and eventually the generic btcguild.com will point to a web-only server and not work as a pool host.

The pool will remain this way until a proper server cluster is setup at one of our hosts that will allow the setup I spoke about at the start of this post.

I knew the kitten would keep out the complaints/questions long enough to get a good night of sleep.

The pool is going to be receiving a few updates soon, possibly tonight, but more likely tomorrow:
   1) New custom DNS server that will cause users to "stick" to a specific pool server when using generic btcguild.com
     1b)  The custom DNS server may also be able to keep loads truly balanced if it can handle the initial traffic we put onto it.

   2) Difficulty 2 shares.  I'm working with the IRC channel to make sure we properly modify pushpoold to run on a difficulty 2 share system.  The net benefit of this is it will cut the SQL load in half, and roughly cut the getwork load in half, which will result in much higher capacity per server.
      2b)  This will give a slightly higher variance in rewards, but it would even out very quickly over a few rounds.  Each share will count as '2', so that people could compare our shares per block to the average without doing any extra math.

   3)  US Central and NL2 servers will be up by next weekend.  With the switch to difficulty 2, this should mean our pool's network could handle roughly 4 TH/sec given an even distribution among the the servers.

An attempt was made at using HAProxy as a sticky load balancer / failover redirect this evening, but it was quickly overloaded.  The volume of connections that the pool makes, in addition to the geographic spread of the servers was causing the latency to freeze up HAProxy.

More work being done tomorrow, and hopefully the US Central server will be up as well.


In other news, BTC Guild now has a mascot that showed up at my door this afternoon: https://i.imgur.com/0MOIB.jpg

New server to act as a load balancer has just been made available.  Hoping to get it deployed this evening.  Current priorities are:

1)  Real load balancing (not Round-Robin DNS)
2)  Moving the website off of US West's pool server
3)  Setting up US Central (Chicago)

If the Chicago server is online by the time I get off work, it will be added to the available pool servers tonight.  The next step is getting US West (pool) separated from the web interface.  Then getting a load balancing server US side, and once that is proven to work properly, an additional load balancing server for the EU side.

Quicker?  I completely changed the design of pushpool to allow a multiple server system, and put 3 new servers online in one week.  There was absolutely no way to prepare for a following spike of a 900 GH/sec GROWTH in under a week.  This pool has been around for barely over one month, and has only recently started to produce enough money to actually pay for anything.  Unlike Tycho, this pool hasn't allowed me the privilege of quitting my day job.

US Central is expected to come up soon.  A new server is also being setup to handle the web traffic and get that load off of US West.

I am also looking into getting two cheaper servers specifically for load balancing US and EU, to get the speeds split more evenly between the 4, soon to be 5 or 6 servers.

US Central (Chicago) will hopefully be online tomorrow or the day after.  Also obtaining a separate server to handle the website to get that load off of the US West pool server.

I'm still hoping to get that feature in this weekend.  I was sick most of the weekend, and my attention was focused on the server sync issues with the UK server.  Right now I'm working on getting a lot of the computational/SQL load for the frontend to be distributed to the slave servers, to speed up the My Account/API pages, as well as help the US West server from having idles.

Toning down the aggression can help.  Additionally, the IRC chat has universally agreed that phoenix's stale rate is completely terrible right now, since it seems to ignore clearing out its work queue after receiving a long poll.


Pool Status Update:
US West is getting bogged down compared to the other servers.  I'm working on shifting some of the frequent MySQL tasks to the slave pools to help it handle the load.  Eventually the frontend will get moved to a separate server so that the pool itself on US West is not significantly affected by website load.

I am very sorry for this lack of notice on this.  I migrated our DNS servers to DNS Made Easy, and that entry was lost in the shuffle.

Last nights rounds have now been properly calculated.  Apache was locking up overnight which runs the front-end calculations.  Nothing was lost at all, and thanks to all the security checks I added for server sync, all the round data is in-tact, including accurate timestamps for each block's solve time.

To keep up with my track record, the technical cause of the issue was that the pool's massive expansion (we were capped at 550-600 GH/sec as of Monday, it is now at nearly 1.5 TH/sec on Sunday) finally found something new to break.  The Apache server had been modified only slightly to give the MaxClients value to the max of 256 (before adding a ServerLimit change, which I did this morning).  With the pool's massive growth, and the variety of apps/widgets out there which poll the API once per minute, the limit was just too low.  Luckily, the Apache side is hosted on US West, which has massive amounts of RAM and a much better processor than the slave servers, so increasing this value was not a problem.

I spotted some of the final sync issues that were being caused between the new UK server and the other servers.  Share totals are now being reported properly and not doubling up incorrectly on the My Account/API.  Last share time is also fixed on both, meaning miner idle emails are going to be started up again this evening.

All 4 servers had a pushpool update run today to keep them sync'd up properly in the event of a MySQL connection failure.

Block 473-474 were affected by the sync issue, and I'm working on rerunning the allocations on those blocks, giving users who were under-reported their actual rewards, but not taking away rewards from those who were not affected.  This reallocation will occur later tonight or early tomorrow.

UK server bug has been fixed.  I did not log out and back in after updating the limits.conf to allow the pushpool user to run more than 1024 connections.

UK Server is now online and running.  I recommend everybody use btcguild.com as their miner address to eventually cause the most even distribution of work among the servers.  However, if your performance is suffering, the following addresses are available:

=== Preferred (By Me) ===
btcguild.com

=== 2nd Best ===
us.btcguild.com  :  Points to US West / US East (and eventually US Central)
eu.btcguild.com  :  Points to EU (NL) / EU (UK)

=== Last Priority ===
useast.btcguild.com : US East (Florida)
uswest.btcguild.com : US West (California)
uk.btcguild.com : UK
nl.btcguild.com : Netherlands

Please only use the last priority series of servers if you are having problems with a specific server.

UK server coming online soon.  Also ordered a US Central server.  This isn't the most effective way, but right now with our rate of growth there simply isn't enough time to make completely custom pool software.

eu.btcguild.com  will auto balance between the NL and UK server, so the current overloading of EU server should die off soon after the new server comes up.

Long polls happening over and over are NEW blocks.  They are not happening "over and over" just because the server feels like sending them to you.  You would have seen the exact same LPs from DeepBit.

Long polls are good things.


New server is already being provisioned in the UK now since EU server seems to have the most load being pointed at it.  There is no way for me to know exactly what the new servers can handle, but they are weaker than the primary server.  Right now all I can do is throw more servers at the problem until I have a chance to possibly rewrite the entire pool to be custom like slush & Deepbit, rather than pushpool.

Fixing block allocations now, I knew the change to how the backups are made would cause me some trouble before going to sleep.


UPDATE:
  Blocks from the night have been fixed.  Time stamps are a little off on them, but share counts and allocation are good.  Part of the housekeeping for the optimization involved a heavy modification to how the worker share counts are stored from now on.  As such, the historical DB of _WORKER_ share counts has been reset prior to block 424.  The share counts for USERS are still present and visible in the block listing.

TECHNICAL EXPLANATION:
The problem was related to the cross server synchronization during block calculations.   Block calculations are done by a script which checks for new blocks once per minute.  At one point tonight, we had two blocks solved between the script running due to a 40-second block.  This caused a problem in the synchronization code, causing it to exit.

When the calculations are performed, the code sets a "lock" in the database telling it that a reward is being calculated, so that if it were to get stuck due to interserver connection problems, it would not have another instance of the script run and potentially duplicate the work.  Since the code exited due to a synchronization problem, this lock did not deactivate, and the automatic block allocations were halted until I manually restarted them.

This is a problem I will work on fixing this weekend.  It's a result of the code becoming much larger due to the integration of 3 servers with the pool, something it was not originally designed for. 

Part of my optimization to the servers involved deleting old data from the slave servers once they were confirmed to be sync'd with the master server, it may have caused some funny reset share/stale numbers.  Resetting again should solve this problem.

I applied a fix to the US East and EU servers.  Since they're not running on SSDs, the backups from RAM to the HD were causing short locks on the table.  Since applying the fix, the idles on my miner pointed at US East have halted.

A 4th server is being prepared in advance to keep all the servers well under their maximum capacity.