As a precaution, we strongly urge you to change your password if it's not very strong. The possibility exists that weak passwords have been brute-forced by the intruder through the abuse of the password-change functionality.[/b] Also, never re-use the account password for your workers: the worker passwords are stored & sent unencrypted.

UPDATE: Payouts enabled, Address-lock and HTTPS now available for extra security.

The new stuff: Payouts are enabled again; maximum auto-pay trigger has been limited to 5 BTC to discourage large balances. We've also just deployed a payout-address locking facility, accessible from the 'My Account' section.

The theme for us for the past few days has been Security, security, security:

History of events
Last Thursday we discovered a security flaw which was being exploited. An attacker was transferring the balances of multiple accounts to his own Bitcoin addresses without consent or awareness of the respective account holders. This happened over a period of approximately one week. Some accounts listed multiple fraudulent transactions over several days. Initially this went on unnoticed, but then Thursday two users notified us of suspicious payments in short succession. This triggered us to halt all payments immediately and launch an investigation. After some nifty digital forensics we concluded that session spoofing was the attack vector. As an aside, doing forensics is quite fun but a lot less so when it concerns your own systems!

Vulnerability patched
The leak was patched soon after locating it, and we started inventorying the scale of the damage (affected accounts, time period, amounts). We are now asking everybody that has been affected to PM us the details, so we can work out a way of compensation.

SSL/HTTPS now available
The event also inspired us to adopt some extra security measures. We added HTTPS support to the site earlier this weekend, encrypting your communications with the pool through a self-signed certificate (see our FAQ for details).

Payment address locking now available
And now today we've added the address locking facility, which allows you to permanently lock your payment address. Payouts from locked accounts can only be made out to the address specified. The address can not be changed if a hacker were to somehow compromise your account, or even perform a SQL injection. The permanence is both the upside and the downside of the lock: An intruder cannot change the destination address, but neither can you.

ABCPool back to full strength
With this trifecta of improvements we are confident that ABCPool now offers the security needed to resume payouts. Therefore we'd like invite those that understandably took a break from ABCPool: You may point your miners to pool.ABCPool:8332 once again!

NB: About the PIN mechanism: We feel PIN in its current incarnation never added a lot of security. We might remove it down the road, or rework it into something better.

In approx. 90 minutes we'll deploy payout-address locking.  At that moment payouts will be enabled again.

Our intentions were to create a stable pool with a respectable size. We want to introduce fees only as soon as we are confident that our miners get value for their money when mining with us. With the recent improvements we feel that moment is not far away now.
The PIN-mechanism was inherited from SimpleCoin, and we were never completely satisfied with it. It will either receive a makeover like you suggested or be replaced by a better mechanism. Thanks for your suggestions!

MC

Brian, Chunglam, LoupGaroux & Hotdog453: Thanks for your kind words and your generous attitude! As you may have guessed ABCPool is a spare-time-and-money project for us. The support from people like you encourages us to keep improving ABCPool despite whatever setbacks we encounter. So thanks!

Reimbursement
We have a pretty good idea of the people and amounts that went missing. To be sure, we'd like to invite those that have lost funds and have not yet responded through the forum, to PM us with the details.

Chlorine & I have been thinking about how to handle reimbursement in a way that everybody will be happy, and I think we may have found a solution that could have your support. I'll come back to that that later this week. First I'd like to talk about some updates on the security front!

SECURITY UPDATE: Moments ago we've activated HTTPS access through https://www.abcpool.co, allowing encrypted access to ABCPool. We did not want to spend the resources for a third-party SSL Certificate so it's self-signed for the time being. That means you'll get a warning, which will disappear when you add our CA to your trusted authorities store. For details: http://www.abcpool.co/faq.php#toc4. On windows the process is real easy, for other systems.. I have no idea. If you figure it out for your device, don't forget to share it with the rest of us.

Another security feature that will be launched shortly is a permanent payout-address lock.

And as promised earlier, payouts will continue this sunday evening/night.

MC

UPDATE: Cause found, payouts will continue sunday-evening at the latest.

To all our users, thanks for your continued patience while we were getting to the bottom of this.

Earlier today we verified the exact details of how the theft took place, which was through session spoofing. Multiple accounts were compromised, resulting in unwanted payouts the bulk of which occurred between october 29 and november 3. We have deployed measures that prevent this type of session spoofing on ABCPool in the future.

What was potentially compromised:
* The attacker did not need your passwords for the intrusion
* No passwords have been leaked directly, since passwords are only stored as a hash.
* Weak passwords MAY have been guessed by brute-force abuse of the 'change password' function.
* The attacker COULD log in to any account through the ABCPool site and act as though they were that user
* PIN has been guessed (or brute-forced) in at least several cases

Steps we have taken to mitigate the issue thus far:
* We have fixed the session handling code
* We have reset the payment address for all our users, because it might have been set by the attacker to his own address.
* We have expired all current sessions
* We have introduced additional logging code

Steps still to be taken:
* Introduce additional security measures
* Re-activate payouts (this will happen sunday evening at the latest)
* Come to an agreement with you guys on how to handle the missing BTC.

What will change for you:
* For now, you'll need to (re)enter you payment address. You may take a look at past payouts and copy the address from there, but be sure to verify that it is actually your own address.
* It's always a good security practice to use difficult and unique passwords, and to change them regularly.

Right now, for the duration of a week.

The requirement has been lifted for the next 7 days.

An update on the investigation: The traces left in our logs indicate that the transaction has almost certainly been initiated through the web interface (possibly scripted to guess the PIN numbers). A SQL-Injection is highly unlikely because it would have left a different pattern of traces. In addition, a code re-review did not reveal any open SQL-injection vectors.

The attacker probably did not have access to all accounts, otherwise he could have just as easily taken a lot more while he remained undetected.

In the mean time, we advise everybody to make sure they are not reusing their passwords for other pools or services at ABCPool; please choose a new & difficult password if that's the case. It's easy to guess usernames based on the MtGox list and the forum accounts, and the Bitcoin community isn't that big.

We'll leave the payout disabled for at least another day until we can introduce additional measures to protect our miners from any unwanted withdrawals. For example, enabling you to permanently lock the payout address will surely help.

Now it's time for me to get some sleep!

'not yet redeemed' means that the receiver of your BTC has not yet spent it.
FYI: All payments, both manual and automatic, have been disabled while we're investigating how this happened.
MC

Hi Ciuciu,

That's a pretty serious amount; We've immediately halted all payments while we investigate this matter to avoid additional losses. It could very well be that someone has gained unauthorized access to our systems.

I'm investigating the matter now, and will keep you posted as I learn more.

After you first mentioned it, I reviewed the PoolServerJ source code and was impressed by its high quality. Its pluggable design is also admirable; seems like a top notch backend all-in-all, certainly worthy of a donation!

So thanks for bringing it to our attention

I'm sure your valid share rate at other pools is as you say, and that it is lower at ABCPool. My point is that it is irrelevant. What is relevant is the *absolute number* of valid shares you produce in a day at each pool, *not the percentage*. Let's say you produce 80000 valid shares + 400 invalids at some pool, giving you a 99.5% valid rate. Now you come to ABCPool and generate 80000 valid + 3333 invalids, giving you a 96% valid rate but with exactly the same payout. And that's part of what's been happening lately at ABCPool; many invalids are retries of shares that were already accepted as valid. As you see, it does not matter if you also generate thousands of invalid shares due to to various reasons; as long as the number of valid shares is what you'd expect, your payout will be the same.

Therefore the better question to ask is: how do your valid share counts, and consequently your payouts, compare between pools? You certainly need longer than a few hours to measure those.

We would be glad to welcome you back for a longer test-run!
MC

I see; then my statement was in error. I applaud yourbtc.net for trying to communicate how they are handling shares, not every pool is that open about it. Now if only that linked comment had also been a news item on the front page, I probably would have seen it

My point remains (and it applies to all pools, including our own) that a pool owner can never prove to its miners that the reported stats are correct, without giving everyone complete access to his systems. While the whole Bitcoin concept is based on proof: proof of work, proof of transaction, etc., Pool stats cannot be proven correct without full access.

Those are some impressive graphs, at first glance! However.. I notice they're from november 1, and you were mining on yourbtc.net. A quote from their main page:
So it seems you've been a victim of faked stats.

This is an excellent example of masking invalid shares. From your screenshot it seems the pool has good rates. While in fact there's no net improvement at all: nothing has changed in the income of the pool and nothing has changed in the payouts of the pool. The rewards are just redistributed, now favoring those who would otherwise have more stales. Of course, this takes earnings away from people who are actually mining efficiently. But they conveniently forget to mention that fact..

There's an additional problem with masking stale shares: It becomes harder to tweak your rigs/setup because you don't get any useful feedback on stale shares anymore.

Thanks for your well-written input Bitlane. We have to admit that since introducing timerolling at ABCPool, invalid rates have gone up. While initial results looked promising they has not kept up, as you have witnessed. We are working on getting this ironed out by applying extra instrumentation in our backend so we can pinpoint where things go awry.
To be clear: The predictability claim is about the reward scheme. Whereas somewhere else you don't know what a share will earn you, at ABCPool you know in advance (barring invalid shares).

For pools with PPLNS, Prop, or other non-*PPS reward schemes it is easy to report inflated counts of valid shares. For example, by allowing late submissions to count as valid for a couple of seconds. This does not improve earnings in any way, but will look better to the untrained eye. I remember reading one pool even admitting as much, using it as an advertising gimmick. This is of course only a cosmetic advantage: the same block reward simply has to be distributed among more shares, and everyone will still earn roughly the same. Bottomline: valid/stale rates for non-*PPS pools can never be fully trusted.

Conversely, *PPS pools cannot inflate valid share counts without taking a huge hit on their buffer, because there is no amortization going on: Every share is payed equally. Inflating valid count would mean increasing payouts without increasing income from solved blocks.

So what metric should a pool miner use then? Earnings! The earnings in the long run tell you what the best pool is. Sadly, the 'long' in long run really means long: a couple of days of data is surely not enough, due to wildly varying pool luck.

Nice catch What you saw is normal behavior. It has to do with how we distinguish between stales and invalids. Let me explain, It's quite simple, and fair (we think). When a new block comes in, we need to tell our miners as soon as possible so they won't waste any cycles on solving the old block. Our stance is that our miners should not be penalized for doing useless work, but only as long as we've not yet notified them of the new block. That's what Long Polling is for, so of course we'll trigger all LP waiters as soon as a new block comes in. That usually takes some time, and all shares that come in during that time are registered as 'stale'. Once all LP's are triggered however, it now becomes the responsibility of your mining software to act on it as soon as possible. Any old solutions that come in after we've notified *everyone* are classified as invalid. In addition there are some other types of submissions that are treated as invalid: shares for work older than 120 seconds, duplicates, and those with invalid timestamps are among those.

Factors contributing to high invalid rates are network latency and aggressively re-submitting miners such as cgminer, which will cause spurious duplicates. Therefore you shouldn't put too much emphasis on the invalid rate, and more on your valid+stale rate, reflected in your hashrate. You can have a 300MH/s rig and 50% invalids, but as long as the 50% valid+stale shares count up to a hashrate of 300MH/s, the invalid rate does not matter.

We actually had a bug way back in august that caused such a scenario, without impacting earning; for details: https://bitcointalk.org/index.php?topic=33586.msg457441#msg457441.

Sorry your experience with ABCPool was not all it could be; I honestly think more pools should be distinguishing between stale and invalid work, as it can model the division responsibility between miner and pool more accurately.

That being said, we think that there's some issues on our side that may sometimes negatively affect the invalid rate. Although it can never reach zero, we think it can be below 0.5% again. Please bear with us!

We have no plans to change our current reward mechanism to include merged mining. We do intend to keep striving to offer you the highest and most predictable Bitcoin mining rewards on the market.

That's right. You may be thinking of SMPPS and friends. For straight PPS (which ABCPool uses), the pool buffer does not control when payout occurs. At ABCPool, your shares are credited within two minutes of submitting them, and you can always withdraw that balance immediately.

MC

Hi Gigasvps,

ABCPool runs on a heavily modified pushpool. We've seen some idling on our own miners as well sometimes, we'll investigate this.

MC

Hi ShadowAlexey, I have reactivated your account.