"Shares Since Last" - A confusing stat which wasn't all that useful has been replaced by "Estimated Shares This Round".  It is an approximation of the shares that are a part of the next announced round.

Website move is complete about 45 minutes ahead of schedule, the prep work paid off.  DNS entries have been changed.  The TTL was set to just 30 minutes, so most users should be able to access the website within the next 30 minutes unless their ISP/computer are not honoring the TTL.

I ban botnets when I have enough information about the size of the botnet and a good idea that I've obtained the complete list of their IPs.  That reserved server IP is for large scale CPU miners.  In the case of botnets, they get sent to that server as well, then filtered out once I have a solid list of all the IPs that are infected.

Small change to the website interface.  Moved the "quick stats" box from the upper right of the page and created a statistics bar fixed to the top.

For now its just pool stats.  I have the following 3 priorities right now:

1) Move the webserver [in about 9 hours]
2) Automatic Payouts
3) Replace US West's load balancer with the setup used on US Central [next weekend]
4) Mining teams (more info on this later)
5) AJAX option for stats bar at top/personal stats.

I've been doing some cleanup on the code before preparing to move the website, so a few small anomalies might appear for a few minutes here and there.  Mostly working on the server synchronization and making it far more aggressive, now that we're only running two major servers.  A lot of the scripts are now executing almost instantly thanks to only needing to open up 2 remote SQL connections instead of 6-8.

Heads up regarding maintenance tomorrow:

Tomorrow at noon (PST) I will be moving the website to a different server.  The pools will remain functional during the move, and your shares/rounds will be allocated once the new website is in place.  Estimated downtime to move the server is approximately 1 hour.  It may take a little longer for your DNS cache to update to the new IP address.

An update on last night's stats problem:

The host for the OLD US Central server is physically moving locations.  This information was not available in any obvious posting/alert, it was hidden in their support desk about 2 weeks ago.

When the stat scripts / new block scripts run, they abort if a server connection cannot be made (otherwise the block would be calculated missing a server of shares).

Luckily we're no longer with that hosting company, I'm just waiting for it to come back online so I can transfer the BTC to our main wallet.  I've already manually set the US Central (old) blocks to 120 confirmations since they were stuck around 30 confirms remaining.

Fixed the issue.  Stats script/new block script were failing to connect to the OLD US Central server, which was cancelled.  It shouldn't have gone offline though (its supposed to stay active til August 10th).  The host for the old Central is pretty laggy, so its possible they're getting attacked again [unrelated to us].

Jine's points aren't about stopping a DDoS (whitelisting won't work AT ALL and load balancing will only work if you have big enough pipes and enough entry points to distribute the load without failure).  His points are how the larger pools have been able to handle the load of the large scale CPU miners (some of which are botnets, some of which are not).

There have been a few restarts on the Central server to correct the scaling issues as they show up, which boost the reject rate for SOME people (depends on if you land back on the same node or not).

Since the last round of setting adjustments, at 1.4 TH/sec the server is not having long poll related issues currently, even when 3 of them hit in a 5 minute window.

Just because I got an email about it:

US West is fine right now.  The reason the speed is declining on West is I'm stress testing US Central to get an idea of how the new load balancer can hold up.  All of the "unofficial" DNS entries are currently pointed at US Central.

Fixed a bug with US West where new workers weren't properly being sync'd to the pool server.

Our new US Central server is now online.  DNS entries for uscentral have been moved over, along with the unsupported [old] entries (de, de2, nl, nl2, eu).  Closely monitoring the server, but so far the signs are VERY good that the new configuration and load balancer are going to run much smoother.

The new server is similar to US West, where it has multiple loadbalanced nodes.  If one goes down, you will get moved to a working node within 30 seconds.  A big change in the load balancing method was made, which will hopefully prevent the (occasional) idles US West gets at long polls.

The blocks are out of order due to the sync error that caused a 6 hour backlog of blocks last night.  The block allocation script doesn't search for the oldest block to apply it to the oldest round, it grabs them by server (So a few rounds were US West, then it moved to US Central, then East), causing them to show up out of order.

Well, technically the orange line could go above the blue.  Each line is running at different increments.  Right now they won't because you're looking at a 60 day average, where difficulty has massively changed.  If a 60 day period ended up being a steady/declining difficulty, eventually the BTC earned line would be above the speed estimate line.

Won't happen unless difficulty plummets.  The days of 1 GH/s being >= 1 BTC/day are over.

Just to chime in and be "that guy":

While I completely agree and have said the same thing in IRC quite a bit about a lot of the smaller pools crying "DDoS", technically they ARE being DDoS'd.

The difference is, they're not being DDoS'd in a HOSTILE manner (ie: When BTC Guild was being hammered offline for DAYS by hundreds of thousands of computers, maxing out the pipes of all our servers).  The smaller pools are being DDoS'd by legit traffic that can't be handled without pushpoold and bitcoind patchs, not to mention separate frontends (run on a separate line from the pool itself).

Technically a botnet miner taking your pool offline IS a DDoS.  It is a distributed load that is denying service due to being unable to respond adequately.  Trust me, and Tycho can chime in too (slush likely as well):  If you get targetted by a hostile DDoS like what hit us a few weeks ago, you'll know.  You'll know because most (if not all) of the smaller pools out there are not running on a host that will tolerate the attack and will swiftly nullroute your IP address for 24 hours automatically.

We're at positive luck since the difficulty change, even with the 5.8m block shoved in there.  That's after we were at -7% luck this difficulty last night.  I'd say the past 24 hours have been flowing pretty well, enough to makeup for the bad start.

Sync issue on US East again last night from a 22 second round, caused our blocks to not announce for a few hours.  Just fixed it and the blocks have been announced.  Our luck had a -massive- swing last night from around -7% to +8.3% since this difficulty kicked in.

There's no such blocking.  Send me a PM with the details you're using and I'll take a look (probably won't be able to respond until tomorrow morning).


I've been working on the new server, which is set to replace US Central and US East.  I'm working on a new configuration for the internal network and load balancer, which should eliminate some of the issues that US West has around long polls.  As usual, there's no way to be sure that the limited testing will scale properly when put into production.

The new server will likely be linked up with the website and other pushpool instances tomorrow evening and opened up as a 4th server.  If there are no issues overnight, I will begin moving the DNS entries for US East and Central to the new server, and close them off entirely Sunday evening (assuming the new server isn't choking).

After a good period of testing (~1 week under load), I will attempt to point US West at the new server for a few hours to give it a solid stress test.  If it doesn't choke during the stress test, US West will have a brief maintenance window (approx 1 hour) to upgrade the server load balancing configuration to what the new server is using.