The person who found a block was being checked improperly.  It was looking at who owned the last timestamped share in the block.  The problem is, sometimes multiple shares were submitted that same second.

I ran a fix this morning which only gave credit to the person who submitted the actual solving share.  Sorry if you were one of the people who lost their block found count because it was given to you in error previously .

The round duration/block statistics were corrected this morning.  I wasn't able to stay awake long enough to monitor when we found the block to make sure everything worked perfectly in the live system.

Block allocations were done properly, it just didn't properly cut off the round timer/post the new block in statistics.  Fix was pushed onto the server this morning and everything is running smooth again.  As has been the case with all the bugs that have come out so far, the back end calculations have been working flawlessly, it's just the display that has had errors.

Upgrade took a little longer than anticipated.  Some of the optimizations were ported from being part of the website (PHP) to being part of the pushpool service itself.  End of round calculations no longer modify any part of the shares database.

The once per minute pool stats update is also modified greatly, cutting the run time from 3-6 seconds (depending on how many transactions were hitting the DB during the stat update) to around 0.2 seconds.

Pool is back online.  No additional downtime is expected to occur, unless a problem is found when the current round ends.

Pool will be going offline in 2 hours and 10 minutes (2:30 PM PST).  Estimated downtime is 1-2 hours.  Once it is back up, the delays, rpc errors, and miner idles caused by the end of a round should no longer occur.  Additionally, the site should be even more responsive upon coming back up.  Part of the change that went live has already cut page generation times to < 20ms for most pages.

Later today (12-14 hours from now), the pool will be going down for a server upgrade.  It will be down for approximately 45 minutes to 1 hour, as this will also be when the work done the past 24 hours for optimizing the DB for block calculations will be implemented.  I have decided to keep the pool down when implementing this to have a single chunk of downtime rather than intermittent RPC comm error/miner idle messages for 1-2 hours while attempting to implement the change while the system is live and running.

Once the hardware arrives, I will make a post on the forum and a red banner will appear on the website, giving 2 hours advance warning.

It was bound to happen, our lucky was way too good this week

Yes.  Those errors should not be seen again, at least until we find another block.  I'm reworking the block calculation code right now so it won't lock up the database during calculations.  Depending on which happens first (completing the rewrite or a block being found), we may have another set of RPC/Miner idle errors for about 2 minutes when we find our next block.

Recent (< 30 minutes ago) RPC & Miner idle errors are part of moving the old shares from confirmed blocks to a new database, and taking them out of the current one.  They should have been very short durations (< 5 seconds).  This is a necessary delay in order to let the pool scale with higher workloads and not be bogged down with speed calculations and share calculations.

I've scaled back the aggression the server was using for how many rows it was accessing (copying + deleting) per cycle.

The 0.001 is no more.  The pool now calculates rewards out to the 8th decimal point, the limit of the Bitcoin protocol.  Payouts are cutoff to 0.XX, but the remaining amounts roll forward into your next payout, exactly the way they are handled at slush or Deepbit.

I believe any miner connection problems are being caused by end of round delays.  The site currently calculates all block rewards in one large chunk of code, which also updates the shares database to associate shares with the proper block for audit purposes.

Because of the size of the share database, and the number of shares being updated in one query, it can cause a delay for access to the database from miners, causing the server to respond with errors.

This issue will be resolved some time today.

Regarding our 0 share, multi-year long block, as pointed out previously it did calculate rewards properly, that display issue will be fixed in the next few minutes.

The pool is no longer rounding rewards down at the thousandth.  You will now receive rewards all the way down to 0.00000001 increments.  If BTC values ever reach the point where 0.00000001 BTC has realistic value, the pool will be updated again to include additional precision for rewards which will roll forward until they eventually add up to a 0.00000001 increment.

Currently visiting family this evening, I"ll be returning home in another hour or so at which point I'll be doing a few minor updates, mostly visual.  Email alerts will likely be in before this time tomorrow.

Can you clarify what you mean by "Current Block Status" ?  Current block stats (ie: Duration & share count) is at the top of every page.

Worker status is part of the API (go to "My API Settings" for your URL).  My pool was added to the sidebar gadget you linked this morning .  I'll send a message to the guys that make the miner status app to see if they can get BTC Guild included.

Exactly what vuce said.  Slush's pool had some downtime, and people shifted to Deepbit to continue receiving reliable payouts.  That huge surge made Deepbit THE place to be for steady payouts, so even after slush's pool recovered, not everybody switched back.

Given the rise in BTC price, which is a greater percentage hike than the difficulty, that certainly isn't the cause.  

I'm inclined to say it's a portion of the OCN miners that came in as a result of the forum thread there losing interested.  Most the people on those boards are high school / college kids, and came here thinking Bitcoin was a way to make fast easy money with their gaming rigs.  Once the thread got locked, it stopped getting promoted as much and now the surge is dying out.  Just my personal theory for PART of the decline.

I'll take a look when I get home from work today.  I just checked on my account and my worker speed changes every minute in line with the pool stats update.

Pool hash rate is in gH/sec, since it will never drop below 1 gH/sec unless something breaks.  Worker hash rates are mH/sec.  I set them that way for easy printing for people who didn't want to convert speeds based on size.  I will swap them both to mH/sec though for consistency.  Anybody who is parsing out a JSON API should have the ability to divide by 1000 as needed .

I'll convert time since last share to HH:MM:SS as well.


UPDATE:  The changes mentioned above are now in effect.  Time since last share is now H:MM:SS, pool speed is sent as mH/sec to stay consistent with worker speeds.

Thank you for all the support, especially the generous donators giving above and beyond the 2.5% amount I've set as the "invalid protection+instant payout" threshold.

When I started the pool, I did not expect to see anywhere near this level of generosity, which is why I decided to consider the pool 0% fee (nothing off the top), and just rounding the user rewards down at the 0.000X level.  The donations being received completely dwarf the rounding kept by the pool, which was already very insignificant [0.024%].  As a result, I will be changing the pool to round down to the 8th decimal, the limit of what BitCoin supports.

This change will be going live this evening, and be applied to all future blocks.

Have you tried stopping/starting?  We're still running strong with over 25 gH/sec right now.