Total Earnings / 0.97 = Amount you've made before fees.  Difference is the fee.

This argument would make sense if I was doing something to try to lure users in.  Reduced fees (or 0%), promotional events, referrals, etc.  BTC Guild's entire business is based on providing superior service, and in return it charges what I would consider a reasonable fee.  BTC Guild has been shrinking, though that's mostly due to ghash.io's private farm which is roughly 10% of the network if not a little more.  I'm not trying to claw it back in any way, and have made changes in the past to purposely push hashing power off of the pool when it was getting too large.

Cex.io charges extremely high fees for "maintenance", and they also probably sell a lot of that hash rate into the order book since they definitely own more than what is on the book.  However, ghash.io also allows miners to point their own hardware at the pool at 0% fee.  They're the same company.  The "throwing money away argument" is based on that part.  They are spending time and money servicing external miners, and are not making a single cent off the time/money spent on those users.

My only concern with the network is the current uneven distribution of hash rate in PRIVATE hands, and 0% pools.  Obviously I'm going to be biased, but BTC Guild makes a good amount of money.  I have an incentive to stay honest, work hard, and keep the pool going.  Any kind of attack from BTC Guild makes no sense because a successful attack, while potentially profitable, will kill the business.  My concern comes from pools where there is no incentive, such as ghash.io.  We already know they control a private farm in excess of 1 PH/s.  Additionally, they are offering a 0% fee pool.

That makes no sense, they are spending money to get other hash rate to work with them.  Similarly, they lose nothing if they do something that the rest of the network would consider bad.  It doesn't have to be an actual attack, it could be something as simple as refusing to vote/upgrade the next time the protocol needs a super-majority to be modified.

Like I said, I'm biased on that regard, they're competition technically.  Though it's not much of a competition when one side is producing a profit and the other is literally throwing money away.  It's up to users to make the judgment call about their motives, but it makes meuneasy to see an entity that has no incentive to stay honest to its users.


Expanding on the first bit:  My concern with private hands is the relatively low cost for an ASIC manufacturer to obtain 51% in the current environment.  There is not enough competition in manufacturers, and none of them have invested much money in mass production.

I'd recommend reading up on things before coming to conclusions and pointing fingers.  What you posted is in no way unusual.  

Additionally, selfish mining doesn't apply to BTC Guild.  As posted above, Selfish Mining is GUARANTEED to noticeably increase the number or orphans on the network.  This not only makes it obvious its being done, but BTC Guild *pays for orphans* when its blocks are orphaned.  BTC Guild would actually become unprofitable if I ever attempted to do selfish mining because of that simple fact.

Shifts have been taking roughly the same time for the last 24 hours.  That would confirm my own observations:  The servers are not having problems.

If any servers are misbehaving, it's the ones whose sole purpose is trying to differentiate DDoS and Botnet traffic from legit miners.  Those servers are only encountered when you first start mining or restart.  If you have been mining consistently, there have been no problems.  If you're having problems involving disconnections, they're on your end.

Rewards are back to being applied within 1 minute of the block solve.  Since the NMC chains have not frozen up since the initial hardfork, I believe whatever caused the original problem was an isolated incident.  This coincidentally would have prevented the above problem from ever happening, although the fix to that script will remain in place so if the delay has to be re-implemented it will not cause a block to miss the being credited.

Doing a rescan of the bitcoind transaction list to make sure no other blocks are missing.  This was caused because the script which polls bitcoind for block solves does so by getting a (fairly large) list of transactions and checking for generation transactions.  This block was solved near the X:57 auto payouts, which were recently increased to handle over 200 payouts in one sweep.  As a result it pushed the generation transaction out of that list.


UPDATE:  This was the only one found.  I've updated the script to now search back for the last 1,000 transactions so it can't be repeated.  Not that it matters much, the real reason this happened is the delay in showing blocks until the blockchain has increased by one due to the NMC bug.  That hotfix will be turned off this evening since the NMC blockchain freeze has not reoccurred.

Check the actual thread for a response.

I am thankful that this at least shows some people are keeping a watch on things.  The vast majority of the day to day pool operation is hardware side and networking.  The scripts handle everything of importance, and there's always the chance for a situation which causes them to behave in a way that is not expected as seen above.

A response/explanation of what happened, why it happened, and how it has been fixed is here: https://bitcointalk.org/index.php?topic=49417.msg4245745#msg4245745

Looking into the missing block right now, I have a hunch of what happened but need to confirm.

Backend servers weren't affected by any outages, so unless you were starting up new hardware/restarting miners there shouldn't have been anything wrong.  If there was, it was not on the pool's end.  BTC Guild uses a frontend filtering server which is used for new connections, but once proven to be valid miners you are no longer connected to those servers.

I do see a drop in the bandwidth on both the EU and US frontend filtering servers this morning.  It may have been a DNS issue with cloudflare, or possibly an attack at the datacenter where those filters are run.  Neither server appears to have been the actual target of any attacks however (no massive flood of connections/bandwidth usage).

Pools change a value in the coinbase message for each worker/piece of work they assign.  Unless the pool wrote their own software and did it HORRIBLY wrong, there is absolutely no way that any two workers will be hashing the same values.

Only Deepbit (3% prop) and BitMinter (if you pay at least a 2.5% fee).

If you want the same features, BitMinter is the same fee (Orphans+Idle Warnings require a 2% "donation" on top of the 1% fee).  Slush is effectively higher fee (orphaned blocks + namecoins are worth much more than the 1% fee difference).  Leaving you with Eligius and GHash.io as your only alternatives if you don't want to deal with extremely small pools.


EDIT:  Forgot Deepbit doesn't pay transaction fees.

The EU filtering server was suffering during the outages.  This was fixed shortly after the website was restored.  As of this post, all servers (including the DDoS/Botnet filters) are running optimally.

Alright, everything has been running smooth for about 30 minutes now.  I think I've killed off all the offending queries, and there's some extra work being done in the background to hopefully prevent it from happening again.

== Below This Line May Be Useless If You're Not Familiar with Databases ==

The problem stemmed from a JOIN based query used in looking up PPLNS Stats.  This query was not a problem until just recently, and was caused by the growth of the pool making the result set from this query become too large for a memory table.  As a result, the query was forcing the result set to be written to a temporary disk based table.  This query executes a LOT, and it's unique per user because it's the query which displays a user's stats for recent shifts on both the Dashboard and PPLNS Stats page.  This ended up causing table deadlocks if too many instances were accessing the table at once.

It took a while to diagnose because I was operating under the assumption the writes to that table were causing it (block rewards, shifts being closed, etc).  However, in this case it was the reads causing the root problem.  The writes were just the catalyst to cause the eventual table deadlock.

Once I confirmed the specific queries that were causing it, I was able to get in and rewrite those queries to use smaller result sets and multiple queries, rather than a single query to get all of the data at once.

== End Technical Explanation ==


So far it's looking great, but I'm leaving the monitors up for the rest of the day to check for any abnormally long queries, so hopefully if it is not fully fixed, I will be able to pause the website again before it creates an actual deadlock.  I'm also adding a new external alarm to set off my phone if a similar error happens in the future, since this particular failure did not trigger any of the already existing alarms.

The API is still running during the website maintenance/update, just an FYI.

Mining is confirmed to still be operational.  Future fixes should not require any additional restarts of the mining server.  Unless something goes wrong, the only thing that will have intermittent service is the website itself.  Shutting off the website provides a SIGNIFICANT boost to how fast operations can be done on the database, mostly because the website is essentially under 24/7 attack these days so there is a lot of database load all the time.


UPDATE:  Problem has definitely been found, now working on fixing it.  Pool servers are still online and functional, just keeping the website off a little bit longer.

The joy of errors that can't be reproduced in a test environment.  Going to be spending the rest of the day staring at scripts and turning them on one by one.  Website will be back in a few minutes once I confirm we didn't have another repeat of the multiple block payout.

And now everything is back to normal.  All scripts running again, the error in payouts has been fixed, and all the servers appear to be running smooth from this end.  Sorry for the eventful Sunday morning.

One block was paid out multiple times due to the script looping.  Automatic payouts temporarily offline again while the reward calculations are fixed for shifts 13478 - 13487.  It's been a LONG time since this has happened, so for users not aware, here's the breakdown:

A block reward was given out to shifts 13478-13487 multiple times.  As a result, those shifts were credited roughly 10x the amount they should have been.  If you requested a payout, or had an automatic payout trigger during those shifts, you have been paid more than you should have.

Once the payout for those shifts is corrected, your balance will adjust to what it *should* have been.  If this amount is negative, your balance will grow back with each block solve until it's back positive.  The negative balance IS correct, because you were paid more than you earned due to the error.


Luckily this was only ~8 hours worth of "extra" payout, so any negative balance accounts would not even take one day to go back to a positive balance.  Hopefully most users remain honest and only a limited number make new accounts to get around paying back the debt.