Bitcoin Forum

The topic of this post could also be "How my carelessness cost me 75 BTC".
As some of you will (obviously - i'm expecting it) retort - "carelessness" can also be "foolishness", "stupidity", or some even harsher words.
But the fact is:
You only think you're invulnerable to mistakes until you make one.

TLDR version:
I left my firewall vulnerable to the bitcoin daemon RPC with severely unsafe settings, and 75 BTCs vanished.

I will not dwell on a big whining - the simple fact is, mistakes cost me my wallet.
Or, as someone who I shall not quote just told me:
I was even raising my % a bit higher, but like I said, this is a quote. :)

With that said, regardless of fault %s, i hope this just serves as a big warning - do you know exactly if your settings are as safe as they can/should be? If you didn't say "yes" in less than half a second, then i urge you to revise your settings.

The facts:
There are no excuses, really. There is a plethora of facts that lead me to this, YES, but that does not serve as justification.
"spill your guts anyway!" i hear in the back.
Ok, then, the loss of 75 BTCs is surely worse than the shame, so if you insist..:

I opened my firewall ports during lunch to show a friend some node.js things I have been working on - a realtime dashboard for P2Pool stats.
This wouldn't be too severe, if my RPCport settings were not too permissive. Which they were since I was abroad last month, and forgot to revert to secure settings.

Working 18 hours a day is not an excuse. Forgetting the RPCport settings is not an excuse. Leaving the firewall open when I got home and only wanted to sleep is also not an excuse.
Just a big sum of recklessness, that had a bitter taste in the end.

The attacker easily accessed my open RPC, brute-forced my user and pass (yes yes, which could also be more complex) and emptied my wallet.

The result:


http://blockexplorer.com/address/18GQdbRCF1f7fjkx3rMdWbuqqR8XFxhQgM (http://blockexplorer.com/address/18GQdbRCF1f7fjkx3rMdWbuqqR8XFxhQgM)

http://blockexplorer.com/tx/1cbcb30e26a00b81dfd03f3cf4b1d8ded8005a19493050b588d3f752a982b913#i4155767 (http://blockexplorer.com/tx/1cbcb30e26a00b81dfd03f3cf4b1d8ded8005a19493050b588d3f752a982b913#i4155767)

The only thing i will whine about, is.. "on my birthday? really? that was harsh."

Also, there is a clear need for more security measures in place.
To defend the (dumb/reckless/whatever) miner. Yeah because in the whole BTC universe, even dumb miners... mine.
The whole BTC universe, as a whole, is a sum of its parts. Even the dumb ones.
And today, i was "just another dumb miner". Which was, and still is, a part of the whole.

Maybe bitcoind should log ip addresses.
Maybe the RPC port should have some anti-bruteforcing logic attached to it. A real, effective one, not just telling the attacker the password is short, like it happens now.
Maybe. Just sayin'.

Troll away. But only after you double-checked all your settings. :)

I want to know more about "a realtime dashboard for P2Pool stats."

Wow, it looks like there are now bots that are always looking for open bitcoin rpc ports.  It took about 4 hours for you to lose 75 BTC.

:(

Not finished, but my idea was to put it on Github when it got to a stage where it could be developed further by other people - at this moment, it has a lot of "hardcoded local settings".
node.js, with some smoothiecharts, updating in realtime  (of course, thus node.js) being fed by the p2pool log.

http://imageshack.us/photo/my-images/32/screenshot20120126at404.png/ (http://imageshack.us/photo/my-images/32/screenshot20120126at404.png/)

Or I might just give it away as it is, at this time I lost the mood or will to do anything regarding this.

Known him for 15 years,  and he doesn't know what bitcoins are.... He liked the realtime data,  but didn't know what data it was.
I find it strange, too. Just a few hours after my flaw this happens... 
After hitting the wall (with my head of course) my first thought was a crawler script just doing portscans based on peer IPs coming from a tail of the logs.
Hmm... I see this as SO doable, it's really freaky...

unencrypted wallet, I take it?

I've read about this attack recently. Your other quote is probably dead on on the %.

Username:Password should force a certain level of entropy before being accepted. I can only imagine how many 12345678 pw there are out there.

the reason i'm not on IRC anymore and the reason why my bitcoin node is not exposed there with bitcoin client 'noirc' flag enabled.
to me it's just being an easy target as anyone can get IP lists from IRC and start probing different ports, default router l/p's and what not.

No, he said on OP, open RPC (well, maybe the wallet was unencrypted too, but it doesn't matter, that's not how it was stolen). Summarizing, it is as if his bitcoind node was accessible by anyone on the internet that happened to know his password, and apparently the password wasn't that strong since it was bruteforced. The attacker just requested the victim's bitcoind to send him money, and it sent.

The interesting part is for such a theft to happen, the thief needed to know that there was an accessible bitcoind on that IP. So, either it is someone close to OP who's stealing him, or there are hackers with crawlers searching for such vulnerable nodes. The latter sounds quite possible, what would mean people using bitcoind RPC should really pay attention to their access rules.

don't want my local network and my system compromised in any way regardless of bitcoin wallet

aren't private keys encrypted, therefore even with open RPC one would still have to decrypt them before a transaction could be made? In other words, an attacker would have to know rpc username/password and the wallet password?

No, RPC is there to allow control of bitcoind by other programs. Like, imagine you have a website that needs to perform payments automatically. Your web server contacts bitcoind and requests the payment. If authorized, bitcoind performs the payment. It doesn't matter if the keys are encrypted or not, as it is the bitcoin software itself that's signing and sending the transaction. It can decrypt the keys if needed.
The hacker did not steal a private key. It managed to access bitcoind and control it, requesting the payment thought the RPC interface. Bitcoind treated it as a legitimate request.
Normally this control interface should not be publicly accessible, but in this particular case it was.

Do you see the difference?

OK I get it. I assumed one would still have to input the wallet password, but it wouldn't make much sense using RPC if it couldn't do anything by itself, thus making wallet password moot.

if the wallet is encrypted you have to enter the wallet password at least once - otherwise the software shouldn't be able to get the private keys
AFAIK once you do enter the password there is an option to set a time out - after that time you will have to re-enter the wallet password to access the keys

Every node on the network knows the IP addresses of every other node.  More or less.  And the port is well known.

wasn't there 2 passwords here? 

the RPC bitcoind user:pass
the wallet encrypted pass

Indeed. Wise words. I slept 2 times what i normally do, and after letting all this "settle in", i feel exactly as you said.
Just take a punch, raise chin and keep fighting, so to speak.


I don't. As ridiculous as this might be (not "might", it IS) - part of my professional life is spent securing other people's systems. Telling them how their security is flawed. Shouting at users who have weak passwords. I totally slacked on my own.


I've been reading... BIP16/17, P2SH... honestly, i feel that, for the enduser, despite some divergences in opinion between Gavin/Luke and others, they ultimately want to do something that is good for everyone, so, eventually, the decision that will be made shall be positive.
What REALLY bothers me is a pool (DeepBit, *cof*) having the power it has right now. THAT is a problem for everyone.

All those BIP16/BIP17 (types of pay-to-script) and BIP11 are doing the same thing, which can allow you to require confirming your TXes from your mobile phone or some other separate device.
(It's already possible technically, but no client currently exist with appropriate functions, and this TX would be "strange"(non-standard))

The point of voting is to a) select the most suitable implementation, b) deploy it safely.

except rpc port which could be changed freely. -rpcport=<port>

I'm changing my RPC ports to a higher area (10000+) to keep my wallets safe. It's set to allow *.*.*.* with very simple username and password.


This incident(accident) is not the first in cryptocurrency area. several weeks ago somebody lost all(2850) his fairbrix. (https://bitcointalk.org/index.php?topic=53221.0) Also because of open RPC.

After setting up a small script to tail the log and do a portscan of 8332 on peers, it took me as much as 6 minutes to find someone with an open RPC port.
No, I did not check if the user:pass was weak or not, i'm not out for revenge.
But, still... 6 minutes only.

You do know that it is trivial to scan all 65535 ports to find the bitcoin RPC interface, right?  And now that you've posted this, I'd be willing to bet that whoever wrote the brute force tool is modifying it to portscan right now.

With known IP that's true..... and the client itself is a great source. RPC interface is indeed easy to discover then.

Maybe I should try finding some insecure peers in irc too.....

First off, that sucks.

Do you actually have evidence that there was a brute force attack, or are you simply inferring all this because the bitcoins disappeared while your firewall was open?

debug.log shows an abnormal number of connections made starting 3 minutes before the transfer was made, so i assumed that's how long it took for a brute force to be successful. I might be wrong, but it looks that way.

Yeah, that sounds like an attack.  You should post the log in a pastebin or something.

besides nolisten... does this make sense?

-rpcallowip=127.0.0.1

22MB, i'd have to trim it.

# cat debug.log | grep 'incorrect pass' | wc -l
951

And forget the 3 minutes i talked about, looking more in-depth at the log, first incorrect attempt was at 05:12, last was at 05:23.
So, 951 attempts in 11 minutes.

Only 951?
That doesn't sound like a brute force attack.
Your password has a length of 2 characters?

it was probably in a dictionary of commonly used passwords

It's the latter. I remember seeing a post like this just a few days ago.

Please tell us what the password was...  not secure anyway.

Yeah what was the password?  "password"?

951 attempts is negligible even compared to a common/weak password list which has a million or more passwords.

I mean getting nailed in 951 attempts is the world saying your password was in the top 0.1% of stupidest/weakest passwords on the planet.


However 951 attempts does raise a useful countermeasure.  Have bitcoind refuse connections for 30 sec after 3 failed password attempts and then after every failed password attempt after that.   So 951 attempts would require 951 - 3 = 948 *0.5 = 8 hours.  If the password was even slightly less weak (but still horribly weak) and was say 100,000th password on a brute force list it would take 30 days.

Let me count the ways:

1. You must explicitly choose a username and password; you must have enough tech know-how to find your bitcoin.conf directory or run with -rpcpasswrod= options.

2. If you choose a short password, then every failed access attempt DOES trigger a timeout.

3. You must explicitly tell bitcoin to listen for connections from IP addresses other than localhost, using the rpcallowip= option.

4. You must open a hole in your firewall that lets any arbitrary IP address through to your rpcport.

I'm sorry you lost 75 bitcoins, but you really made a LOT of mistakes. Adding more layers of protection to the RPC interface isn't high on the development priority list, but if anybody wants to volunteer to keep track of number of failed RPC authentication attempts over time then be my guest and write a patch. Just be sure it isn't vulnerable to denial-of-service attacks by people deliberately generating failed login attempts.

I suggested (https://bitcointalk.org/index.php?topic=61217.0) a second logfile for security matters so that tools like fail2ban could monitor something with less volume than debug.log.  Adding a second logfile seems much easier than implementing the lockout yourself.

RPC hacks = encrypted wallet feature is useless. What if the attacker gains root access to the PC?

Not. Cool.

I guess someone could release a non-RPC capable version.  Then again if attacker has root access then they already installed a keylogger and your wallet has already been stolen.

So, yes, i did 4 out of 4. I had no idea mistakes were prune to ratings.
So if I or anyone does 5 mistakes, what then, you think our house should blow up?


You said "layerS".
However, if ONE layer is well implemented, it should be enough, covering 4 out 4 ratings, or even 10 out of 10.
Out of all people, i think you would see these mistakes as a simple "something needs to be done". But if you're just following the easy path out of this issue by just throwing me a donkey hat, sure, i'll take it. From my first post, I did not make excuses, and admitted my guilt. You're giving me no novelty. Sorry about that.

It's just a disappointment to see development does not care about its 'flock' - we, who make mistakes as any human does, are on our own.
Fair enough, from your pedestal point of view, of course.

 

If the wallet is encrypted (is isn't by default), then there are 2 passwords needed. The first is the RPC password, which allows access to the RPC interface. Actually, the RPC interface also supports SSL, but that is almost never used. The second password is the encryption password. Once you have access to the RPC interface, you must then pass commands including the encryption password in order to unlock the wallet for access.

If configured correctly, it could remain open on the public internet, without too much fear. However, I would only use it over a VPN anyway.

True enough.

But if the RPC interface was hardened, at least this would buy some time.

As is, with root access the attacker could be in & out in a few seconds with a scripted hit.

A better way would be to start with an even smaller timeout, and double it upon each failed logon from the same IP.  This gives a human plenty of chances to retry the password, but quickly makes brute forcing impractical.

And in the process actually leaks information about the length of the password, though admittadly not much it's still kind of embarrassing since it lets the attacker know when their attack is likely to be profitable or not.

I have a patch in progress which improves logging and does the rate limiting. Being sure it doesn't create a dos and isn't gratuitously incompatible with the threaded RPC is why it's not already a pull request.  (also, I'm kinda uncomfortable about timeouts in the code since it's currently single threaded... any simple implementation of delays is an automatic DOS).

(the improved logging is important because it makes it slightly more likely that the attacker will get caught, but mostly because right now the vague logging causes people to open up allowip more than required because it's not clear why their requests are being rejected, especially as we have pretty weird wildcarding syntax)

And yes— m3ta had to do a lot of things to create this vulnerability.   But it isn't like he did them because he was _trying_ to get robbed, he did them because various tools and capabilities in our ecosystem put him in a position where the mistake was easy to make, and anytime that happens _someone_ will make the mistake eventually— it just happened to be him.  The fact that he was exploited was also because lots of other people are vulnerable too, otherwise the attackers wouldn't exist.

(And if you saw this thread and didn't go check your RPC and firewall settings, just to be sure, you're a fool— in my view)

If you look at industrial accidents there is a constant pattern in most of them— some stupendous feat of human error managed to bypass several safeties designed to prevent human error, all at the same time.   If you focus on the magnitude of the human error, you'll do nothing to reduce the incidence of failure in the future. Human error is a natural force as unstoppable as gravity.  We should build systems which have every worthwhile immunity, because the human failure is not going to go away. (It's going to get a lot worse, in fact, as more unsophisticated users adopt bitcoin)

In addition to some belts an suspenders in resisting basic brute-force attacks there are other things that could be done. There are _other_ daemons with similar RPC ports that don't have this problem because the daemon itself generates a default key and puts it in the file. e.g. Chrony.

Quick question: Can I use CIDR format in the allowip setting, and if not can this be implemented? I find it easier than using ranges and wildcards, but maybe that's just me.

Also, could rate-limiting be specific to the IP that is submitting invalid passwords? I realize that an attacker could use many endpoints to bypass this limitation, but it might be a step towards solving the denial of service concerns.

I apologize for coming across as harsh, it has been a hard couple of weeks.

And the reason it has been a hard couple of weeks is because I'm working really hard to try to get support for multisignature wallets... which would have prevented your loss (assuming you were using a multisignature-protected wallet).

The hacker would have been able to ask your bitcoind to send him your coins, but you would have got a confirmation on your cell phone (or SMS or email with a one-time-PIN or whatever) and realize immediately that you'd been hacked.

Or, in other words:  I've been working really hard trying to get a solution for the bigger problem of insecure computers, because I really do care about the "flock."

CIDR could be implemented, as could lists... might make sense to do as part of IPv6 support.

The rate limiting I coded up is per-IP, and localhost is exempted. I also started with per-group (e.g. /16 for ipv4) but thought it too risky.

Anti-brute-force is really a protection of last resort— the attacker doesn't have to be impatient he could use dozens of IPs (e.g. tor) and scan you for months in slow motion, even if its fairly aggressive it does not have that much protective value, e.g. even it it only allowed one bad attempt per IP ever, getting 1000 IPs is not _that_ hard and would have been sufficient here. So this sort of feature has to be carefully weighed against the DOS vulnerabilities it might create.

The real hope of this kind of protection is to reduce the odds of success and increase the cost of attacking enough so that few people bother attacking in the first place... creating a kind of protective halo of disinterest around the few users where all of the other better protections have failed.

You did come a bit harsh, by beating a dead horse, but no one is here to judge (specially not me, the horse) - it can happen to everyone.
TBH, we know you care.  Thank you for that.

Just PLEASE don't get the idea there was a 'demand' for an instant solution from you.
You're not anyone's errand-boy - you don't need to be too defensive regarding that, and if you let anyone, for a second, make you think you are, beat them with a stick. :)

Cheers.

Yes, this only really applies when someone bangs a '*' into the allowip and calls it a day. I hope to see your changes for CIDR merged soon ;)

Since bitcoin is touted as a candidate for global adoption by the masses, I'd have thought that its being considered that 80% of PC's on the planet run an insecure OS. Root access is only a script-kiddie away.

Then again, perhaps those 80% households will run the bitcoin client only on mobile platforms. Problem solved.  :P

You can run on freenode using tor to mask your ip.

noirc was not sufficient to prevent you address from being relayed— bad guys couldn't get it from IRC but they could get it from the regular node to node address exchanges.

noirc + nolisten will be enough in the next version of bitcoin (https://github.com/bitcoin/bitcoin/pull/792), however.

how do u do this?

That guy's a real asshole... or else a lot of people just sent him tons of BTC... idk for sure.

Anyway, as a not-so-expert-when-it-comes-to-security-guy, how should a firewall be configured on a bitcoin-only machine?

To allow Bitcoin through the firewall.

The original poster's problem was that he created a bitcoin configuration file specifically turning on the RPC remote control of Bitcoin (default off), setting an extremely simple user/password combination (default disabled), and adding an option to allow worldwide IP access (default local machine only). It would be the equivalent of leaving the keys in the ignition, leaving the car running, putting the signed-over vehicle title in the window, and posting a sign that said "free car".

Also he did not encrypt his prikeys.

So, was the wallet encrypted or unencrypted?

That was irrelevant. The RPC was open, and the authentication password was weak.
EDIT: To be honest I actually don't know if, when starting bitcoind as a RPC server, you have to open your encrypted wallet by typing the password. I'm assuming that's the case, what renders wallet encryption irrelevant.

I'm just trying to understand the RPC thing. When you do a transaction in Bitcoin-QT, you need the password to unencrypt your wallet. If you connect to RPC and authenticate you do not need to unencrypt the wallet to do transactions?

I don't think so, otherwise every RPC-command should contain the encryption password as a parameter. I believe you unlock the wallet while starting the service. Or the RPC-authentication password is also the wallet encryption password (don't think so either because RPC is much older than wallet encryption).

You may check https://en.bitcoin.it/wiki/API_reference_(JSON-RPC) for better info.

EDIT: Just checked and there are explicit RPC calls to lock and unlock the wallet. The wallet was then likely not encrypted, or encrypted with a weak password.

No, but if you screw up any worse, it'll probably blow up on it's own...

;-)

Sucks this happened man... But with gold in the ports, you KNOW it's being done... It's not a deterrent from a possible attack, it's a required defense against a known, perpetual, constant engagement.

You must take air with you if you're going underwater.... You will drown if you don't. It's not a maybe, it absolutely will happen.

Not meaning to toss any more gasoline on you. Just emphasizing the difference between a deterrent from a potential attack, and a required shield against a guaranteed ass-rape... This is guaranteed to happen unless you do something about it.

If it didn't happen, that would be the surprise.

That's correct. One has to first send the RPG command to unlock an encrypted wallet even when using the RPC commands.

So the thief brute forced two passwords, correct?