New registrations have been disabled until our new server is in Chicago, or a fix to pushpool's scaling is found, whichever comes first.  Estimated time of the new server coming online is July 11th-15th (shipping it this week).

I'll be bringing back NL1 and NL2 tomorrow (and pointing their DNS entries back) to alleviate the stress.  They're much weaker than the other servers, but its better than nothing, even if it does slow down API/My Account pages.

When the new server is ready, all the US pool DNS entries will be pointed towards the new pool.  24 hours after that happens, I will shut off the other pools (US East and US Central).  I will post the IP for manual pointing if your computer has an error with resolving the name (DNS caching gone awry).


Regarding today's huge number of issues: I just finished filtering out large portions of the networks that were attacking the servers today.  Mostly they were hitting DE1/DE2, but the rules are now on all servers.  If you ended up getting caught in the ban, send me a PM or a support email with your IP address and I'll whitelist you.  

I know these problems were terrible today, but this is the same kind of attack DeepBit had to deal with a few weeks ago.  The problem with DDoS prevention is that mining looks a LOT like a DDoS, especially when a server is having connection trouble.  Standard DDoS filtering techniques will end up having an obscene number of false positive bans on a pool server.

Servers are looking pretty stable again.  This whole issue at least allowed the removal of the pool from the web server, which means a pool issue won't bring the website crashing down as it has in the past.  The sync code between servers also received an update after taking down US West, NL1, and NL2, far less aggressive sync attempts (meaning one server having connection issues won't take down another server with it).  The downside is it means that rounds may extend a few seconds past their actual solve time (up to a minute).  The upside is that if one round is extended due to a sync issue, the following round will be that much shorter.

The new server hardware will hopefully be online around July 11.  It's IMPOSSIBLE to effectively stress test this kind of machine to replicate the real world conditions, but the underlying software configuration has been tested with the help of the IRC chatroom.  Failover when a sub-pool crashes takes about 4-6 seconds to kick in, meaning you can point your miners at one IP and it will automatically put them at an internal pool.  There will still be an EU alternate set of servers, so if a connection issue occurs between you and the US pool, there are still fallbacks on different ISPs.

DE1 and US East are getting DDoS'd (just to add more work to my weekend).

I'm sorry, the next time we have unplanned server issues I'll make sure to warn everybody in advance.

I'm trying to get these servers to last the next 5-10 days, at which point I can put our new colo'd hardware online which has enough power to match our current US Servers, while being loadbalanced with internal failover.  The hardware is mostly setup, and it'll be shipped out to a colo facility in Chicago on Tuesday.

Moving the pool off the US West web server after putting it off for other issues.  The miner reconnects are triggering some DDoS filters at the ISP.  I've moved the DNS over to another server temporarily to get miners off of US West.

DE, DE2, Central, and East are fine right now.  Shares still counting, work still being issued.

US Central has been up for the past 12 hours or so.  Running just fine.

Server crashes from trying to get a bitcoind that will actually utilize the full specs of the servers caused some bad performance.  I've reverted the changes to the basic bitcoind modification (RPC threading) by JoelKatz for now.  He's got some new fixes for the getwork patches, but I am giving the pool some time to recover for now before testing the patch on one of the servers again.

This server performance issue was compounded by overall bad luck in the last 24-36 hours.  We're still having good luck since the latest difficulty change, but those multiple days of +10-20% luck caught up to us.

DE2 server is online.  NL1 and NL2 DNS records have been moved to DE2, so no DNS records point towards the NL servers.  I will close off those pools completely once their speed approaches < 10 GH/s.

US Central froze.  There were no shares being solved by that server during that time.

There is no "fixing".  The server that was offline/crashing during those blocks.  If you even managed to get a single getwork response during that time it was a miracle.


DE2 is being setup TODAY.  I'm also changing the server sync code to be more flexible when two servers fail to communicate a round change with each other.

Block script got stuck due to US Central's freezing of bitcoind.  Been using that server to run JoelKatz' patches since it tends to have the most issues.  Sent him a report of roughly what happened, though its hard to tell since it froze, rather than crashed.

Block updater script has gone through the blocks from last night.  Working on a script that will kill a bitcoind process and restart it if a server isn't reporting any shares for 5 minutes


The new server was delivered to my house yesterday, and I've been setting it up to replace all the current US servers.  It will be shipped out on Tuesday, so it could potentially be online for next weekend.

US Central is connecting fine from central California.

Also, fixed some payouts.  A few payouts were requested from US West while the bitcoind's were being switched over.  I voided those payouts so the funds are back in your account to request again.

Idles/stales are not the reason you went from 3.5 to 2.8.  LUCK is.  We were functioning at +10-20% luck for the last two days.  Last 24 hours was -10%

Just reverted the change back on servers, bug in the new code was still causing some lockups.  JoelKatz is working on a new patch.  I got trigger happy after the patch seemed to be working for a half hour on NL2 without issues.  Next update will only go live on one server until we've had a few block solves.

Patch is now live on all servers.

Full decimal payouts had issues on older bitcoin clients, and it wasn't fixed until 0.3.21 (I believe that was the version, may have been +/- 0.0.1).  The amounts would not show up in your balance, and would often end up being included as transaction fees without your knowledge.

Once Automatic Payouts are implemented (this weekend unless something catastrophic happens with the servers), there will be a setting you can use which will allow full precision payouts.  It will default to 'off' and warn you when enabling that you should be using the most recent bitcoin client if you want to receive all 8 decimals of precision.

I've just deployed JoelKatz's latest experimental bitcoind patch on NL2, which makes some major changes to the getwork functions.  If he's correct on his analysis of the problem, this patch could potentially remove the miner idles unless the process actually manages to cap CPU usage on the server.

I've got a good idea of the problem.  There will be a quick restart on US Central (~0.5 seconds) in the next few hours to run a potential fix.

Yes.  It should be delivered today or tomorrow if its on time.  But that still means that we are 2-3 weeks away from it being deployed live.