Alright, everything is rolled back to the old software.  There's a definite bug in this new setup where it will occasionally stop accepting new connections.  I thought that was fixed after the patch this morning, but it just showed up again.

I won't be touching the pool software for the next week while doing a line-by-line review to try to figure out what is causing this to happen.

That's how most do it.  The issues today only affecting 2 of the 4 public servers, so many users did not experience any disconnect/outage at all.

There's nothing a pool can do when it comes to poor hardware design.  Most ASICs are simply designed poorly, they're hacked together in a way that "works" and immediately shipped.  As a result, there have ALWAYS been problems with just about every ASIC hardware manufacturer where they randomly screw up when a server disconnect occurs.  Normally they work fine, but sometimes they just die and require a power cycle.

This is the one nice thing about ASICs which use USB, they're controlled by decent (and update-able) software that can normally get them to behave properly and handle things like disconnects without dying.

You should've clicked reset stats so it had a proper average window instead of including the downtime/outages.  I'm looking at your account now and it's almost at 37 GH/s.

Only a few people are affected because it's only affecting fringe cases where the stratum commands are being sent in awkward ways that normal mining software doesn't do.  The problem mdude77 was having was related to how the stratum proxy sends it's auth+subscribe requests (it sends them in one packet), ONLY when using the '-cu' argument.

I believe I've got the problem figured out.  Once again the slush stratum proxy never fails to disappoint in its ordering/formatting of his own damn protocol.  Restarting with a fix in place.


mdude77:  Can you try restarting pointing at the stratum.btcguild.com:3333 address?

Alright, I think I've been able to duplicate the issue, time to run some packet capture to see what the hell the stratum proxy is doing that is non-standard.  Next time you supply logs, please dont' use '-q'.  Either leave that off or use '-v' so it actually outputs information.

Your problem, based on your logs, is something completely out of my control.  As far as I can tell, you can't even connect to stratum.btcguild.com:3333, period, let alone authenticate.

I know it's not just visual, but your hash rate is PROBABLY recovered by now, the problem is BTC Guild uses a 1 hour window, and if your'e not mining for part of that window due to shitty hardware/software, your hash rate will show up as bad for an hour, even if it's CURRENTLY hashing at normal rates.

Stratum proxies, Bitfury, and KNC Jupiters are all known to have EXTREMELY bad reactions to server restarts/disconnects.  There's nothing a pool can do on their end to fix shitty software/firmware.  I can't leave the servers running on the same outdated software for the next 2 years just because a few pieces of hardware react poorly.

Try resetting your stats if your speed is showing up very low.  I know Bitfury is one of the many ASICs that is complete garbage in its firmware, which means it's not good at handling server disconnects.  There have been a few server restarts over the last few hours (switching to new software, new software crashing and restarting, and then switching back to the old software after it had crashed).

PM me the worker name you're using....because I'm completely unable to duplicate anything you're saying.  It's working the same as it has all week...because NOTHING has changed on that server all week.

I'm unable to duplicate any of this.  I'm pointing a stratum proxy at the only server running the new backend and getting 78ms-100ms responses, which is the same as my ping to that server.

Just flipped back to the old software after encountering a crash on the server end.

HellDriverUK:  That still makes no sense that you're submitting a few shares and it quits.  It's not like the servers send different information after a few shares.  I need pastebin log files to have any chance at figuring out what is going wrong on your end.

Trying to get a working version of cgminer 3.10 working, but since cgminer likes to fuck with drivers and make itself incompatible with bfgminer it's taking a long time.  I'm not seeing anybody else complaining about a similar issue though...I'm still thinking it's a config problem.

Absolutely nothing has changed on the validation server in the last week.  Any connection attempts have been connecting to the validation server all week.  Additionally, the communication between the server and client is identical to what it used to be.

My guess is you messed up your argument on the worker name/port or something similar.  Works fine on bfgminer 3.9 and 3.10 here.

At 4:27 AM this morning, the validation server finally froze up (this is a good thing).  I copied the log files out and restarted it just a few minutes later.

Now I've had a chance to go through the logs to find out exactly what was happening inside the server at the time of the freeze, and I believe I've got a workaround implemented.  Two of the public backend servers have been restarted with the new code with the workaround implemented.  Sorry to the users that experienced a disconnect when the swap happened.

Dwolla was a fun ride.  They were the fastest and cheapest way to get money out of MtGox for a long time.  It basically only got off the ground due to Bitcoin, as much as they refuse to admit it.  I remember when they were putting out PR releases about the growth of their service over the last year (this was mid-2011), and had a chart and timeline about events that lead to their growth.  They didn't include Bitcoin.  

Want to know what mirrored their graph?  The USD volume on MtGox.  Their *entire* growth was from Bitcoin related transactions.

Just wanted to thank the forum-goers today.  This outage lasted a lot longer than I would've liked (well, hard to have any outage that is shorter than 0 minutes), and lasted longer than my original projections due to just how long the resync took.

However, for the first time in quite a while, the thread remained clear of useless posts, no demands, no crying foul.  Even my email inbox was *mostly* empty during this event.  The ability to concentrate solely on the task at hand (and providing updates) really made it easier to focus than having to pull double-duty as public relations.

All of the pending manual payouts have now been processed.  A few more manually triggered automatic payouts will happen over the next 1-2 hours in order to catch up with the backlog of accounts that have reached the threshold.

New payout server is finally synchronized with the blockchain.  Doing a few checks and then I'll start letting the payouts flow.


UPDATE #1 (6:02 PM):  New payout server is properly reporting blocks to the website and recording/paying them.
UPDATE #2 (6:04 PM):  First batch of small payouts (< 0.10 auto payouts) has been successfully sent and recorded.
UPDATE #3 (6:25 PM):  5 batches of small auto payouts sent, two batches of large (>= 0.10) auto payouts now sent.

Taking it slow to prevent something that has happened in the past, where payouts were using the unconfirmed outputs of other payouts, causing a severe delay in how fast payments were getting confirmations.