I didn't update the little "New News" counter, but the pause was extended to 01:00 GMT.  I think it will be ready by then.  Old server is at block 266k, new server at 237k.  I think at least one of them should be caught up before the hour is up.


UPDATE:  I think I should've gone outside to watch some grass grow.  This process gets exponentially slower the closer it gets.  New server is only 2k blocks behind the old one now, ~9000 blocks behind the live blockchain.  I've got the debug.log file scrolling on half my screen so I can start processing payouts the moment it finally catches up.

I was adding those as I wrote that response, they should be showing already.

Old server is reindexed up to block 259k.  New server had a corrupt data file and is doing a reindex as well (at 160k).  Now it's a race to the finish.  Coins are already in transit getting confirmations for payouts to resume.  Will update again as soon as one server is ready.


EDIT:
15 minutes later.  Old server at 260.5k, new server at 225k.

A lot of failovers from Eligius due to some of their issues today.  At least it's nice to see a good chunk of their miners have failovers on multiple pools, I know a lot of users do not.


An update on the progress for the payouts:  I'm currently indexing the blockchain on two servers now.  The original payout server which has the blockchain locally, but is on slow hardware, and a secondary server which has ~260k blocks locally and is trying to download the remaining 21k to synchronize with the network.  The 2nd server is running on an SSD and has a much better processor as well.

Whichever one finishes first will be setup with the hot wallet in order to start processing payouts.  Sorry it's taking so long, but it looks like at least one server should be caught up within the hour.

In the meantime, I'm continuing to watch blockchain.info and add block solves manually so rewards are applied to their proper shifts as blocks are solved.

No plans on that front yet.  I tend to assume most users have smartphones capable of receiving email alerts.  Once things settle down with the heavier lifting (backend getting debugged/deployed and ScryptGuild), I may add a secondary email field specifically for alerts where users could set it to one of those SMS-emails that the various providers have.

BTC Guild has been my primary source of income for a little over a year and a half now.  That's one thing that's a bit different with BTC Guild compared to larger pools.  I quit my day job/career to pursue the pool in April 2012.  Additionally, I keep no staff.  It's not due to cost, but due to trust/efficiency.  Bitcoin is so dangerous when it comes to how readily available the money is, almost like a petty cash drawer in a small office working on the honor system, but instead of $100-200, it's $500,000-1,000,000 and it constantly gets refilled.  An 8 hour lapse of time (like for example, sleeping) with somebody else with any access to core functionality is literally a million dollar or greater liability.

As a result, I continue to work on additional monitoring, efficiency improvements, etc., to allow the pool to continue to scale to more users and more servers without the additional requirement of more staff.

Manually adding the recently solved block to the appropriate shifts since the process of reindexing the blockchain is taking a bit more time than I expected.  Reindex up to block 230k of 281k now.

The actual pool servers and database server all run on SSDs.  The payout server is a bit outdated since it's not really doing anything that requires much HDD/CPU under normal circumstances.  It'll eventually get moved to an SSD system at some point.

Payout server is re-indexing the blockchain with the barebones hot wallet file (just the block generation address) loaded.  Currently at block 210k of 281k.  The pool won't reflect any block solves until it's done reindexing.  Since it already *has* the data, this shouldn't take too much longer (just a lot of HDD thrashing).

Once that catches up, I'll be sending the old hot wallet contents back to the payment server.  Payouts will resume once that transaction has a few confirmations.

For users of your intelligence level, I recommend dogecoins.  Memes are probably closer to your native language.

Yeah, manual payouts are the only ones that got stuck so far, I've yet to encounter a single auto payout that got stuck.  Which makes sense, because manual payouts use full 8 decimals for precision, leading to a *lot* more work being done by bitcoind in order to pick which inputs to use.

Almost done cleaning the hot wallet out.  Also cleared up about half of the payouts that were showing incomplete data.  So far only 3 payouts were found that actually *failed* to process.  The rest were just lacking the txid association.

At current rate, payouts should be running again in under 2 hours.  A couple automatic payout runs will be executed manually in order to catch up to any backlog.

It's that time of year, where the payout wallet is getting so fragmented it is occasionally timing out when trying to process a payout.  At 21:00 GMT, payouts will be paused for 2-3 hours (depending on network block rate for confirmations).  Earnings will continue to accumulate, it is only the actual payment processing that will be paused during this time.


EDIT:  Moved up to 20:00 GMT after finding a number of failed payouts in the last few hours.

Depending on size and manual vs auto, it is either every 30 minutes or once an hour.

That news post isn't related to other pools, it was a BTC Guild specific issue.  The problem was figured out (wallet file was too fragmented causing it to respond extremely slowly whenever new blocks are found on the network).  I thought that type of slowdown had been mostly corrected in bitcoin 0.8, but was wrong.

Well, looked like it wasn't quite ready for prime time.  Rolled back to the old software.  A bug that I thought was fixed happened again, where the libevent listener completely stops responding to connections/accepting new ones.  Always fun to debug something that you can't figure out the exact cause of since it isn't crashing anything.


EDIT/UPDATE:
Added a *ton* of logging to the US validation server, hopefully I'll get woken up around 4 AM to it freezing and the logs will tell me where this timeout is occurring.

In the meantime, time to start planning out ScryptGuild while waiting for the bug to resurface on the validation server.  All backends are back to the known stable code, sorry again to any users who were disconnected as a result.

The beginning of a new era:  https://blockchain.info/tx-index/6be69ab8b3faad4ee99c2323a25402413f00409efa0bc4641e84cd416dedab13

The first live block solved with the new software.  It has been deployed on 2 of the 4 public servers now.  I'm also comparing two slightly different variations of the backend on those servers to compare CPU load on the primary thread (network i/o processing).

These new servers are wired in very heavily to the alarm system so that they will alert me if they become unresponsive (not just crash).  After a few days, the remaining backend servers will be updated once the benchmarks between the two variations have enough data available.

The new stratum server has now been deployed on *one* of the public servers, with heavy monitoring in place.  I am sorry to the quarter of users who experienced a disconnect as the backend was switched over.

Again, it is on your end if you are seeing issues right now. All servers are functioning normally and nothing has been changed on the back ends in weeks.

I can only suggest inspecting your setup. No server is having any issues and the pool speed is not reflecting any regional connectivity problems.