Bitcoin Forum

On Friday night GMT I had a little over 7000 XPM stolen from a wallet that was encrypted.  My entire holding of XPM.  Just to add salt to the wound, I had been mining heavily for the 2 days prior and had over 1800 coins maturing. Over the weekend the thief continued to steal those remaining coins as they matured.   I'm now trying to piece together how this has happened.

Firstly I shut down all of my VPS's and personal machines.  Since maturing coins were still being stolen it meant that they must have had a copy of my wallet.dat rather than using my RPC.  I turned on a single miner and set it running a script to sendtoaddress 10 XPM and ran it as fast as I could in an effort to beat the thief.  Many thanks to spekk and a few others @ mcxnow for their help and quick thinking with this solution on Friday night.  The thief was obviously doing the same with a script attempting to sendtoaddress 10, as we "battled" all weekend to beat each other to send the matured amount.  I increased the number of miners running the script and with this method I managed to salvage 1100 coins over the weekend whilst the thief got 800.  So all up I have lost a little over 7800 coins.

Here is an example of one of the many transactions the thief made:

The thief sent all coins to this address Aca1dndvLHK1BLWEGsJE2Ci35Wg4azZy2F.  Pool owners, please could you check to see whether this key is in one of your wallets and PM me if it is.  If not, is there anyway to check the transaction history/debug.log for incoming transactions from this address?  It's a long shot, but I'm determined to do go down every avenue to track down this person.  The only pool owner I know is RealSolid@mcxNow.  Please could people forward this to other owners so they can also check?

The wallet was encrypted, but I had been using it since the early days of XPM so there is a slight possibility that there was an unencrypted version on a VPS drive somewhere (that I'm no longer using).  I should state though that I have NOT been mining for several weeks until the 2 days last week when I tried out something new with Amazon ec2's - those instances were 100% using an encrypted wallet.  This can only indicate the wallet was stolen earlier.

I have only ever copied the wallet using scp so it does not exist on public dropbox or anything like that.

I have checked for a keylogger / trojan and don't think I have one, but who can be sure without a reinstall these days?

I have used VPS's from Digital Ocean, Amazon, Azure, and GoGrid.  Other than the new ec2's last week, all other VPS's were shut down several weeks ago.

I'm at a loss as to how this could have happened but I welcome any suggestions so that I can ensure it doesn't happen again.  As you can imagine I am absolutely devastated.  I am not a rich person and don't hold a lot of coins.  XPM was the first time that I had gotten in early and figured out how to scale cloud mining successfully and I manged to mine around 10k before I felt it was no longer profitable.  I sold some a few weeks ago to buy some mcxNow fee shares, 3k last week to pay off my early VPS fees, and the remaining 7k was my long term investment.  So other than the fee shares I had taken no profit at all out of what I had mined.  I'm now left with a very large bill for the ec2's I used last week and only the 1100 XPM I salvage to pay it with.

To the thief:- you probably think that stealing crypto is easy and inconsequential.  It might have been easy, but it is certainly not without consequence.  You're not the one who has to explain to their wife where this money has gone or why we have a large amazon bill to pay.  You're not the one feeling sick at the thought of having such significant amounts of money stolen.  You're not the one who has lost confidence in crypto.  But there is a very small chance you have a conscience - if you do please return my money to me at AKmhQzmDAPK8DCT97aVps87pN565kHzS1v and this will be forgotten about.

To everyone else, I urge you do make sure your wallets are encrypted and you are taking every precaution possible to secure your setups.

Thanks, a very gutted
paulthetafy aka paulscreen

Ouch! Over $5000 worth. You must have had a pretty mean VPS set up. Sorry to hear about your loss, hopefully some coins are returned. Something similar happened to me a while back as well.

Perhaps he brute forced the root password and got into SSH/SFTP? If so, Fail2ban could have prevented that and i recommend every server user to install it. Those VPS mining guides, while useful, did not take malicious intent into account.
Sucks man...

Sad story =/ Hope you to recover somehow

Why? They could have asked for unconfirmed transactions over RPC, dump the privkey of the block and import into their own wallet. This way, once the block matures, they can spend it.

I know this works because I do this very thing to group all my immature mined blocks into one wallet.

Sorry about your loss.

I have sent you small (0.15btc) donation to help you with your losses to the address in your signature.

txid: 0a393ac298ca893567b1746eea0455f916a3b2d979d4640db2bf0143522b0167

thanks usahero, that was both unnecessary but warmly welcomed.

Damm that sucks paul.  That really sucks.

I guess the lesson to be learned here is not to hold your funds in a wallet that exists in many many places.  It would be better practice to be routinely sending those funds to a cold wallet which you have stored in a very safe place.

If it is like you say, a VPS vulnerability, i wonder if we will be seeing more reports about this in the near future, as there were a lot of other people mining on those same VPS's.

Best of luck,
crendore

Damn that's allot hope you will find out who it was and how he did it! to ease your pain a little  (i know you are not begging) i send 10 xpm to your address, i know it's a small amount but maybe if 780 people do the same ;) i enjoy crypto and like it very much. it's not only the crypto but the crypto community what make's it worth liking it so please don't lose faith.


Regards.

thanks nfuse, I'm certainly not begging, but it is much appreciated.

Yeah this is not good, i wonder how it would be possible to give wallets or funds even more security?

BTW - can you explain more how this works

"  I turned on a single miner and set it running a script to sendtoaddress 10 XPM and ran it as fast as I could in an effort to beat the thief"

Have you got a copy of this script that runs on ubuntu 64 bit?

I don't have any coin i guess worth anything like 7k  however would be good to know there is a way to thwart people stealing your coins to some degree. I don't see how it would work ? why was he only trying to draw 10xpm at one time, why did he not extract the coins in one large transaction?

I would guess because the block value is a little over 10 XPM right now.

Really sorry to hear what happened to you, Paul. I had my Bter account hacked a couple of months back, and while my losses pales in comparison to yours, they were quite significant to me and hurt a lot.

Mail the shit out of all the exchanges and tell them not to accept any transactions from that address. Also keep a lookout to see where the coins move.

Indeed 7068 were taken in one hit, then 10 XPM at a time when each block matured.  The blocks were all around 10.5 in value to be on the safe side and account for fees he was sending 10 at a time.  The script was simply...

Have you checked your ~/.bash_history file?
Everything you type to the console will be recorded there, including your plaintext wallet passwords, if you are not explicitly excluding them.

So it would be easy for an attacker who has access to the machine to steal the wallet and the bash_history file.
Are you aware of this security hole?

Wow I wasn't actually. But I had never unlocked a wallet on a linux machine until after the first coins were taken and I had to unlock in order to run the sendoaddress script.  Good to know for future though!

Woah dude, this sucks.

Hope you find out who it is and recover your coins.

What step can I take to ensure the security of my wallet??

Make sure it is encrypted from day 1 and make sure you dont have a keylogger!

Ive encrypted it from day one, I have the wallet.dat files.

What the safest way to store them? I guess leaving them on the desktop on the server isnt a good idea ?

EDIT:

Ive been thinking...

The thing is this thief has been successful and probably feeling pretty smug right now, he will most likely continue.

OP I've emailed ypool support to see if they can help and dropped a link tp this thread in the freenode XPM channel, Ypool are the only functional XPM pool that I know of.

I'll also drop links to this thread where I can to make people aware. I suggest that if anyone else know a good place to drop a link then to do so and keep this thread updated so we dont pester admins etc with dupe requests.

I have a cold wallet that I keep on a flash drive.

I made a new encrypted wallet.dat and stored it on a flash drive, writing down the address.
This is my savings account flash drive. I have a cold wallet of every coin I take as payments.
They are all in the same password protected rar. I even have a copy of every wallet/client on that flash drive as well.
There is no way conceivable for someone to hack my cold wallet.  ;)

I plan on eventually making it 2x flash drives and keeping one in my safety deposit box. This way If anything happens to the first flash drive I'm not completely boned.

Another way to store a cold wallet for long term is to put it in a password protected rar and email it to yourself. As long as Gmail is still alive and kicking you have a copy.
For shits and giggles, and so its "stealthy", Name the rar something like "Christmas Pics". If someone gets into your email they wont even think twice about trying to brute force your "Chrismas Pics" lol

Sounds great, Im a bit new...what a cold wallet?

Is it one that does stay online and up-to-date with the blockchain?

Exactly the opposite  :)

Sorry for your loss. In the future, run Ubuntu from CD, install Qt, disconnect from internet physically, never mind syncing blocks, create new wallet, extract private key and save on paper or somerhing, switch of power to computer. Now when you do mining, send all proceeds to this cold storage and do it at once you get the coins.

Sorry to hear about your loss. Use multiple wallets and strong passwords next time.

I am sorry for your loss as well.

In an effort to help prevent this from happening to others I have a question:
- You mentioned "those instances were 100% using an encrypted wallet." I thought you couldn't mine with an encrypted wallet?
- On EC2 where you running windows or linux instances? I know with linux instances you can only log in with your keypair (pem) and all ports are blocked unless you open them with a custom security group config. Not sure on windows (I believe you can set a custom administrator password and clone with the same windows login ID and RPD easily to it)

As you mentioned this likely happened earlier though... This is why I do not use shared wallets. Or store my central wallets on windows =/


The problem with cold wallets is, by design, you have no access to it. Which makes it hard to "sell" coins to recoup expenses (and opens it up to being stolen via a compromised system)

Fail2ban is vulnerable to DOS of the entire server via log poisoning.

Deactivate password login and only use key-based authentication, problem solved.

I wonder if these guys can help:

http://bitcoinprbuzz.com/worlds-first-stolen-bitcoin-tracing-service-and-bitcoin-data-recovery-high-profile-digital-forensic-services-company-sytech-embraces-bitcoin/

Thats a lot of XPM, hope my donation helps cover a small piece of the costs. Please keep us updated on your findings. You could make a good guide about securely using your wallet on different machines, which in itself could be worth some nice donations.

Good luck and I hope the XPM will come back to you.

Ah, thanks i understand now. Sorry to hear this.

"Mail the shit out of all the exchanges and tell them not to accept any transactions from that address"   - would that work?  could they just not create another wallet on another machine and clean them through that?
I wonder in the future if the actual coins can be identified and black listed if proven to be stolen. Probably not i guess since that would have been done, and who would decide if they were really stolen or not hmmm

Sadly the sort of person that would hack into your machine and take every single coin is probably not going to give any back. :(

Yes mining with an encrypted wallet is fine.  You can't use sendtoaddress while encrypted though.
I was using Linux instances with all incoming ports closed except 22 for SSH.

I've learned my lesson with shared wallets.  Even though it is significantly easier to manage across lots of machines, I'll never do it again!

Thanks, that's very kind of you

Typo, meant Doesn't

Cheers! :)

UPDATEAdmin from ypool has responded.


:jh00: I have already contacted paulthetafy. Sadly we have no transaction that involves the thief's XPM address.

Hope some other avenue opens up.

You keep it in the freezer.  :P

The cold wallet does not stay synced to the block chain. You just backup your wallet.dat Once you have the address you can send coins to it and all you have to do to use thoes coins is to import the wallet.dat into a wallet/client.

I'm not sure how clear i'm being. lol

I think Ive got it.

Im gonna do something else which I hope is just as good.

Create a bootable USB with Ubuntu or similar on it and have a wllaet on there and keep my saving on it.

I just plug it in every now and again to update the wallet with coins Ive sent.

Is there a expiry time for transfere?

e.g.

I send coins to my USB wallet on Saturday and only update on the Friday. Will they always show up?

Sorry to OP for slightly derailing the thread but I'm sure others will find this info useful or at least make them think about their own security.

Your wallet could be offline for 10 years and still get the transaction when you bring it online.

If you go with the bootable linux. Make sure you make a copy, usb flash drives dont fail often but THEY DO FAIL so be careful.

Another option is to print out a paper wallet and send coins to it. Just don't lose it.  :D

are you the guy who accused me of stealing 7000 xpms on mcxnow chat? i'm sorry for your loss man but i kinda take offense to being accused of it out of the blue. i understand you were running j-coin on the same server iirc, hence your suspicions since i made j-coin. if someone could strip j-coin down and look it over for signs of a wallet stealer i'd really appreciate it. I don't need this bad publicity with everone already accusing me of being a scammer over the Gascoin debacle

Oh I absolutely didn't accuse - I think there was a misunderstanding there.  Someone asked what else I had installed on that machine and I said that the only thing was the j-coin wallet.  No implication there and I didn't for a minute suggest that you/it were to blame.  I've said all along I thought the wallet was probably copied weeks ago.  Sorry if it came across wrongly. 

ah okay, i just wanted to clear that up since i caught the tail end of the conversation. you had already moved on from the conversation and the trolls were all over me giving me the what for. again, i'm sorry for your loss. i hope they catch the guy. all scammers must pay

for future use of vpn i urge everyone to do this



start a text file one your home pc and start beating up your keyboard , use the shift key and numbers to get special chars.  about 20-50 chars long should do.

it should like like

!W45ygbw4%BN56j8u46m7mki578,o0,5mrn6Uw4b5vy1q34tv13%By2n456@$5y2v#$%t1cf34Tg2v345t24%BY@$YH#%6unh5&U#bv45c@#$!#!#RE$T!#$VQ#$

save your text file!!!! and don't lose it.

make two of these passwords, so you can add a secure password to your new user u added.


next you should also disable root access


change the permit root login option to no , save the file.


log out as root, and log back in using your new username and password in your text file. Obviously you will just copy / paste the password, right click on your ssh console to paste it in.

This should prevent anyone from getting into your vps.

Incidents like this make me wonder how long it's gonna be before crypto gets a proper organization to avoid and resolve such activities. With so many coins losses from theft, scams and password/wallet loss, even guys like us who have dedicated a lot of their time, effort and money into crypto are going to give up on it one day. A $5,000 loss and a huge Amazon bill would be quite depressing and should it happen again it WILL kill the trust in crypto for both Paul and those close to him.

I'm really sorry this happened to you mate, I don't understand why bad things happen to good people, this universe sucks!

I will be sending few coins across today, and I urge everyone who can to do the same. I know Paul quite well and I know that he'd do the same should this happen to me or anyone in the community.

Hope you'll catch him mate...

So without rpcallowip=127.0.0.1, could someone bruteforce the rpc password and do a sendtoaddress? But he said he ony had port 22 open.
And how about upnp?

I'm very sorry to hear about this.

I guess the time span and amount of instances you have been running, you probably have not kept any or may disk images or logfiles.
It would be interesting to find out how your wallet or hosts have been compromised. My guesses would be via a disk image or via ssh and weak credentials.

Some suggestions to miners about what can you do with ssh to improve security.

• Open up ssh only to what is necessary. 0.0.0.0 is alot of addresses, not to mention IPv6 addresses. If you are using ec2, you can update the firewall from the web interface when your IP when you need to log in from elsewhere. If you are on DHCP (your ip address changes), you can use a subnet (eg, x.x.x.x/24) which will still greatly reduce your exposure to attack
• use ssh keys, not passwords
• keep your ssh private key secure (keep it only on the machines you need use to connect to instances, a backup on USB )
• you can encrypt your key with a passphrase, look into this and also ssh-agent
• If you must use windows, use anti virus software, scan regularly, keep it up to date. MacOS and Linux are not immune either
• If you muse use a password, use a strong one. http://en.wikipedia.org/wiki/Strong_password
• Check the documentation for sshd, and edit your ssh_config file, especially these options
• AllowUsers - use this option in sshd_config to whitelist the usernames you need to access your system
• PermitRootLogin - set this to no or without-password, use sudo to become root if you need to
• Don't run a sshd on the machine you use to connect to your instances if you don't need to. If you must, then secure that too.

ssh ports are being probed for weak passwords all the time. If you run sshd without a firewall, just look at the logs.
By running the primecoind daemon (or any coin for that matter) you publicize your IP number to others (and the fact that you probably have a wallet that might even cointain some coins in it) 

Regarding wallets, I won't go into protecting your wallet or the best way to move your funds around but leaving copies of wallets lying around is not a good idea. Ideally a provider zeroes the disk when a customer stops using it but it may not always be the case, deleting a physical disk takes time.

When you are done using a machine and no longer need it's wallet file should can delete your wallet or even better,  "wipe" (apt-get install wipe,  wipe FILENAME) or write zeroes over it (dd if=/dev/zero of=PATHTOYOURWALLET bs=1024 count=100). WIth SSDs, you can't be sure that everything is ever deleted, but if the file is wiped or zeroed, you can be fairly sure that the file cannot be recovered without special tools and physical access to the disk, or without administrative access to the disk system in a larger provider. When you "rm" a file, usually, it is just the directory entry and list of block a file is using that is deleted. If not wiped or written over, it is possible for a file to be reconstructed.

Treat your wallet backups as you would treat your wallet, if someone finds your backup, it's almost as good as your wallet.dat

Ideally the ssh settings and user accounts are set on your first miner which you then clone.
Wiping the wallet could be made part of the shutdown script, make your you have a safe backup.

just my 2 cents but most vps will keep a log of ip addresses for incomeing connections if you could get this list it would help narrow it down you could then do a trace route/whois and backtrace to general area and isp from there you could contact the isp with the info and go from there i have to do this form time to time on ppl who jump bail when i can make contact with them online it might be a longshot but worth a try im recently new to mining so im afrain i cant donate coins but i have years of experience in tracking people and things down so will donate my time ive never tracked down a wallet addy before but theres a first time for everything :)

for someone who knows more about it than me heeres the transaction details

Array
(
    [hex] => 0100000001bf21d2bf33ad48cb59ef160ae204dda5d7bbf5e6be9d8f75d5fb95d620098edd00000 0004a4930460221008e1c6aa6c28c10456a608fbc1c77e6fd8816a3499a942cb71135669fdd9a5c 7d022100fc7ef05f56270f91a59da7ec421c28eb1966bb050daf275491ffc235bc4d569f01fffff fff0240aeeb02000000001976a914c9bd871c1b681e43aec2bba71ec46ac9dc82713788ac00ca9a 3b000000001976a914e41c51584730272c896b6ddd15af7f6a14dcea2c88ac00000000
    [txid] => c0bcfde4fa1ac44d96edeb448bd5d7fa3ecf73f525e69058d69a01cf695c0400
    [version] => 1
    [locktime] => 0
    [vin] => Array
        (
           

• => Array

                (
                    [txid] => dd8e0920d695fbd5758f9dbee6f5bbd7a5dd04e20a16ef59cb48ad33bfd221bf
                    [vout] => 0
                    [scriptSig] => Array
                        (
                            [asm] => 30460221008e1c6aa6c28c10456a608fbc1c77e6fd8816a3499a942cb71135669fdd9a5c7d02210 0fc7ef05f56270f91a59da7ec421c28eb1966bb050daf275491ffc235bc4d569f01
                            [hex] => 4930460221008e1c6aa6c28c10456a608fbc1c77e6fd8816a3499a942cb71135669fdd9a5c7d022 100fc7ef05f56270f91a59da7ec421c28eb1966bb050daf275491ffc235bc4d569f01
                        )

                    [sequence] => 4294967295
                )

        )

    [vout] => Array
        (
           

• => Array

                (
                    [value] => 0.49
                    [n] => 0
                    [scriptPubKey] => Array
                        (
                            [asm] => OP_DUP OP_HASH160 c9bd871c1b681e43aec2bba71ec46ac9dc827137 OP_EQUALVERIFY OP_CHECKSIG
                            [hex] => 76a914c9bd871c1b681e43aec2bba71ec46ac9dc82713788ac
                            [reqSigs] => 1
                            [type] => pubkeyhash
                            [addresses] => Array
                                (
                                   

• => AaAaWgCpqebejux1wyw1H8yQvuPCizy6yy

                                )

                        )

                )

            [1] => Array
                (
                    [value] => 10
                    [n] => 1
                    [scriptPubKey] => Array
                        (
                            [asm] => OP_DUP OP_HASH160 e41c51584730272c896b6ddd15af7f6a14dcea2c OP_EQUALVERIFY OP_CHECKSIG
                            [hex] => 76a914e41c51584730272c896b6ddd15af7f6a14dcea2c88ac
                            [reqSigs] => 1
                            [type] => pubkeyhash
                            [addresses] => Array
                                (
                                   

• => Aca1dndvLHK1BLWEGsJE2Ci35Wg4azZy2F

                                )

                        )

                )

        )

    [blockhash] => 7db340b42e39e14554b4ac7ee831a646df1dbbcb547cd4ebba2aa413a0de3858
    [confirmations] => 1830
    [time] => 1376860711
    [blocktime] => 1376860711
)
   

Thanks everyone for all your help and suggestions.  I still haven't gotten to the bottom of this but, thanks to the posts here, I do realise how easily I could have been compromised.  All of the VPS images I had used have been deleted already so I'm not able to check those for signs of malicious access.  But my best guess is that one of the early ones I used was compromised, before I started using an encrypted version of the wallet. 

Once I find some time I will try to collate all of the security-improving suggestions and post them into a new thread for everyone to benefit from.

Thanks
Paul