BMF has lost access to its wallet

[usagi]

Hello. This is the letter I never expected I would have to write, primarily because I never saw this coming.

On MayMarch 29th, 2014, I received the last e-mail which passed through tsukino.ca and the website was shut down by BitVPS.

The official story is that a RAID disk was pulled and was no longer in use. How this could have happened to a live server, esp. one of the more expensive plans (KVM-6) was never explained. The problem was not noticed by BitVPS but two weeks after, I sent them a letter asking about my server. I was assured someone would look into it.

How the remote backups which I had running via cron disappeared was never explained or commented on.

Over the next month, there were few emails sent to support. The responses were typically "someone is looking into it" or abouts; I didn't pay it much attention. In the back of my mind I had always assumed someone was looking into it and that the problem would be fixed soon. In the last two weeks (or so) I have stepped up efforts; On the 20th I ended up opening a ticket because email and IRC was something of a dead-end. Not because I couldn't talk to people like arij and Starsoccer and a few others, but because nothing was ever done.

On the 23rd my ticket expired unanswered so I reopened it. It remains unanswered and just expired again.

On the 27th I sent arij (the president of BitVPS) a detailed e-mail letter explaining what was going on and he assured me on IRC that he would open a new server for me "right away" while they worked on the backups so that I could at least resume development and have a webpage.

Of course, two days later, nothing has been done. Unfortunately I still don't know precisely why this hasn't been fixed and due to the exceedingly odd circumstances surrounding it, I am forced to disclose this as material information about BMF; BMF's webpage and it's wallet were hosted on that server (primarily because I was developing a web wallet service called hotwallet.ca since a couple of years ago). This wallet and all the bitcoins in it have been lost. The total estimated damage is about $10,000 in realized costs. There is a much larger amount possibly lost in losing the ability to prove we are GLBSE claimants. Not a small amount of money by any means, but not a huge and ugly number like "millions" (I pity some web businesses). However it means we can never prove a legal claim to money owed to us by HashKing, AmazingRando, Ian Bakewell, BitcoinRS, etc.

I have to admit, I have been assured by tech support that I would be taken care of and that a new server would be set up for us. However, it has not been done. Two months is a long time; if it gets done later I'll post an update.

This has set us back about 2 years. Essentially, our only asset now is our mining income stream. We have 36 mHash/share which is a significant increase over our last reported number, but please excuse me for not calculating the gain in percentage terms because this setback is deeply troubling to me.

I may have slightly panicked; they have assured me a new server can be up in about a day. Fingers crossed!

[abbeytim]

you didn't have any backup wallet backed up locally???

[twentyseventy]

Hello. This is the letter I never expected I would have to write, primarily because I never saw this coming.

On March 29th, 2014, I received the last e-mail which passed through tsukino.ca and the website was shut down by BitVPS.

The official story is that a RAID disk was pulled and was no longer in use. How this could have happened to a live server, esp. one of the more expensive plans (KVM-6) was never explained. The problem was not noticed by BitVPS but two weeks after, I sent them a letter asking about my server. I was assured someone would look into it.

How the remote backups which I had running via cron disappeared was never explained or commented on.

Over the next month, there were few emails sent to support. The responses were typically "someone is looking into it" or abouts; I didn't pay it much attention. In the back of my mind I had always assumed someone was looking into it and that the problem would be fixed soon. In the last two weeks (or so) I have stepped up efforts; On the 20th I ended up opening a ticket because email and IRC was something of a dead-end. Not because I couldn't talk to people like arij and Starsoccer and a few others, but because nothing was ever done.

On the 23rd my ticket expired unanswered so I reopened it. It remains unanswered and just expired again.

On the 27th I sent arij (the president of BitVPS) a detailed e-mail letter explaining what was going on and he assured me on IRC that he would open a new server for me "right away" while they worked on the backups so that I could at least resume development and have a webpage.

Of course, two days later, nothing has been done. Unfortunately I still don't know precisely why this hasn't been fixed and due to the exceedingly odd circumstances surrounding it, I am forced to disclose this as material information about BMF; BMF's webpage and it's wallet were hosted on that server (primarily because I was developing a web wallet service called hotwallet.ca since a couple of years ago). This wallet and all the bitcoins in it have been lost. The total estimated damage is about $10,000 in realized costs. There is a much larger amount possibly lost in losing the ability to prove we are GLBSE claimants. Not a small amount of money by any means, but not a huge and ugly number like "millions" (I pity some web businesses). However it means we can never prove a legal claim to money owed to us by HashKing, AmazingRando, Ian Bakewell, BitcoinRS, etc.

I have to admit, I have been assured by tech support that I would be taken care of and that a new server would be set up for us. However, it has not been done. Two months is a long time; if it gets done later I'll post an update.

This has set us back about 2 years. Essentially, our only asset now is our mining income stream. We have 36 mHash/share which is a significant increase over our last reported number, but please excuse me for not calculating the gain in percentage terms because this setback is deeply troubling to me.

This is a self-moderated topic, so I assume that Usagi will be deleting all post critical of him. I'll be surprised if he leaves this one up.

This is all very convenient, isn't it?

[usagi]

you didn't have any backup wallet backed up locally???

No, I had paid for remote backups on my KVM.

Hello. This is the letter I never expected I would have to write, primarily because I never saw this coming.
[snip]

This is a self-moderated topic, so I assume that Usagi will be deleting all post critical of him. I'll be surprised if he leaves this one up.

This is all very convenient, isn't it?

Your post isn't critical of me, it's just asinine. You quoted the entire OP so that I could not delete it later, thus accusing me of having a shady ulterior motive for making the post in the first place. Please, I don't have time for such foolishness.

The intent of self-moderation is not to prevent what you said from being seen, it's to retain a good signal to noise ratio. The last thing that will happen here is people making baseless accusations out of a B-movie like "this is all very convenient, isn't it". If that's all you have, then you have nothing. Tell you what -- you have 100 days to explain your reasoning and provide evidence or I surely will delete your post (but not this record of your post).

[theMiracle]

you didn't have any backup wallet backed up locally???

No, I had paid for remote backups on my KVM.
...

Don't these "remote backups" have a long and storied history here in Bitcoinland?
They're more dangerous than those homework-eating dogs...

[usagi]

you didn't have any backup wallet backed up locally???

No, I had paid for remote backups on my KVM.
...

Don't these "remote backups" have a long and storied history here in Bitcoinland?
They're more dangerous than those homework-eating dogs...

I have an update from BitVPS and it looks like a server can be put up today, I may have panicked. The truth is BitVPS is probably not to blame because they've had a change in management recently. I actually really like BitVPS.

[DrGregMulhauser]

Hello. This is the letter I never expected I would have to write, primarily because I never saw this coming.

...

I may have slightly panicked; they have assured me a new server can be up in about a day. Fingers crossed!

Regardless of what the specific technical failure might have been, and regardless of whether someone else is going to be able to step in and fix that technical failure for you, it is fairly startling that there doesn't seem to be any hint of an apology in your post or even any acknowledgement of culpability.

Would you agree that ultimately, this has almost nothing to do with the server hardware and everything to do with the decision to store private keys on a web-accessible server that is operated and maintained by somebody else, relying on a backup strategy over which you don't appear to have any direct control or direct involvement? I am not suggesting that it is never a good idea to store private keys on a web-accessible server -- obviously many services could not be operated otherwise. But I would suggest that any decision to do so ought to be undertaken with full awareness of exactly what the h*** is going on with that server, and with an absolute willingness to take full responsibility for it and everything that happens to it. From what I can tell, you have expressed neither familiarity with what is going on nor a willingness to take full responsibility for it.

Am I wrong about that?

[twentyseventy]

Hello. This is the letter I never expected I would have to write, primarily because I never saw this coming.

...

I may have slightly panicked; they have assured me a new server can be up in about a day. Fingers crossed!

Regardless of what the specific technical failure might have been, and regardless of whether someone else is going to be able to step in and fix that technical failure for you, it is fairly startling that there doesn't seem to be any hint of an apology in your post or even any acknowledgement of culpability.

Would you agree that ultimately, this has almost nothing to do with the server hardware and everything to do with the decision to store private keys on a web-accessible server that is operated and maintained by somebody else, relying on a backup strategy over which you don't appear to have any direct control or direct involvement? I am not suggesting that it is never a good idea to store private keys on a web-accessible server -- obviously many services could not be operated otherwise. But I would suggest that any decision to do so ought to be undertaken with full awareness of exactly what the h*** is going on with that server, and with an absolute willingness to take full responsibility for it and everything that happens to it. From what I can tell, you have expressed neither familiarity with what is going on nor a willingness to take full responsibility for it.

Am I wrong about that?

I can't agree more - did you learn nothing from the 17K BTC that Bitomat lost?

Your post isn't critical of me, it's just asinine. Anyways no, the intent is not to prevent what you said from being seen, it's to retain a good signal to noise ratio. The last thing that will happen here is people making baseless accusations out of a B-movie like "this is all very convenient, isn't it". If that's all you have, then you have nothing. Tell you what -- you have 100 days to explain your reasoning and provide evidence or I surely will delete your post (but not this record of your post).

It was meant to be critical of you; I've always been wary of your dealings in the past. One of the first things to happen are accusations, baseless or otherwise. I don't see the point in deleting a post if you're going to leave the 'record' of it up - what's the point.

In any case, it's extremely irresponsible to not be checking on the backups with that amount of money at stake.

[usagi]

Hello. This is the letter I never expected I would have to write, primarily because I never saw this coming.

...

I may have slightly panicked; they have assured me a new server can be up in about a day. Fingers crossed!

Regardless of what the specific technical failure might have been, and regardless of whether someone else is going to be able to step in and fix that technical failure for you, it is fairly startling that there doesn't seem to be any hint of an apology in your post or even any acknowledgement of culpability.

Would you agree that ultimately, this has almost nothing to do with the server hardware and everything to do with the decision to store private keys on a web-accessible server that is operated and maintained by somebody else, relying on a backup strategy over which you don't appear to have any direct control or direct involvement? I am not suggesting that it is never a good idea to store private keys on a web-accessible server -- obviously many services could not be operated otherwise. But I would suggest that any decision to do so ought to be undertaken with full awareness of exactly what the h*** is going on with that server, and with an absolute willingness to take full responsibility for it and everything that happens to it. From what I can tell, you have expressed neither familiarity with what is going on nor a willingness to take full responsibility for it.

Am I wrong about that?

Yes you are wrong. I took reasonable precautions with weekly backups and the server failure was not due to any kind of mechanical error, hack attempt, shoddy code etc. it was due to something dumb which happened in the server room, like someone spilling coffee on a RAID disk. BitVPS already apologized to me about it. I can see how my saying "sorry" can make people feel better but this "sorry" is more akin to "sorry it rained, we can't play baseball today". So yeah, I'm sorry about what happened. You better believe it. But it's not logical for me to assume culpability, no.

Actually I have to admit I am a little suprised at the negativity I'm getting over this, it's as if people automatically assume every failure is intentional.

We're still incorporating in June.

We're still getting an office and a telephone number.

We're still moving forward with "secret project X" (send a msg if you are interested in this project).

The issue here is we lost a couple thousand dollars in the wallet and the ability to prove we're GLBSE claimants.

[usagi]

Your post isn't critical of me, it's just asinine. [snip]

It was meant to be critical of you; I've always been wary of your dealings in the past. One of the first things to happen are accusations, baseless or otherwise. I don't see the point in deleting a post if you're going to leave the 'record' of it up - what's the point.

In any case, it's extremely irresponsible to not be checking on the backups with that amount of money at stake.

*shrug* Thanks for your opinion, but you don't know what you are talking about. Try designing a web wallet sometime. You need to keep what happened here in perspective. You are flying off the handle at me because someone spilled coffee on a raid disk 5,000 miles away? Huh? Go back and read the OP then read what you just wrote. If you aren't embarrassed enough to delete your own post I would be very surprised. Beyond that, unless you have something positive to say no you are not welcome to post in this thread anymore.

It's you who is creating the situation of "One of the first things to happen are accusations..." -- if you don't like it, then don't be part of the problem.

[jonsi]

We can all learn something from here:
   The act of spilling cofee absolve everybody from any guilt. Nobody is to blaim. Look no further! (but if you do you will be deleted)

[usagi]

Yes you are wrong. I took reasonable precautions with weekly backups and the server failure was not due to any kind of mechanical error, hack attempt, shoddy code etc. it was due to something dumb which happened in the server room, like someone spilling coffee on a RAID disk. BitVPS already apologized to me about it.

You absolutely should be culpable, I am shocked that you think you are not. Your attitude of not assuming you have done anything wrong has prompted me to speak out.
[...]
Not only did you fail to follow the most basic of back principles...
[...]
This is beyond - BEYOND - an oversight, it is gross negligence to the point of being criminal.
[...]
Why should anyone invested with you, or anyone that has any future dealings with you, not view it similarly?

7/10

The update I have for you all today is that we have a server up and running, the backups have not been restored yet but I am assured someone is working on it. Given that we have a server up and running now I am confident things will work out. To the people who really set into me, you are more than welcome to use some other web wallet service, if it makes you feel better that they aren't honest with their customers. I would have thought you learned something from all the people who cut and run in this community over the years but I guess not.

Further, we have purchased a Synology DS-4 with WD Red 4tbs (hosted locally) and have set up rotating backups to it via cron.daily.

[usagi]

We can all learn something from here:
   The act of spilling cofee absolve everybody from any guilt. Nobody is to blaim. Look no further! (but if you do you will be deleted)

Nahh I don't delete people unless they're trolling. As I said I am sorry it happened but these things do end up occurring one in a million once in a while. It's like having a top executive commit suicide; If you want I guess you can blame the boss for giving him too much pressure at work but it's a bit of a stretch, isn't it?

[usagi]

This reminds me of when Lando Calrissian was trying to help Leia and the others escape the Cloud City (of Bespin), after handing over Han to Jabba the Hutt (using the Empire as a facilitator).

This belongs in the troll thread.

[damiano]

As mentioned before, this is totally irresponsible.

For someone that is sorta long standing in this community I would of thought a bit better.

Just my .02

[usagi]

As mentioned before, this is totally irresponsible.

For someone that is sorta long standing in this community I would of thought a bit better.

Just my .02

Point well taken -- "This" makes me seem totally irresponsible. But the problem is there is a lot you don't understand. You were given a news release and an overtrolled community response. Here's the deal; we had regular, weekly, off-site backups. Even if someone got a hold of the backups they would be unable to take money out of the wallet without the password. And despite people claiming I said "RAID is not a backup" I did not say that. The server is kept on a RAID and so is the backup for access and reduncancy. That does not mean RAID is a backup. It means the server and the backup were on a RAID array. The only reason I mentioned RAID at all was to point out how the server went down. It is otherwise inconsequential. What kind of troll would twist what I said into a claim I believed "RAID was a backup?" Preposterous.

I also kept secondary backups at home. I just finished running through backups of 8 or 9 different wallets. Unfortunately the server's wallet was kept on a local hard drive which had recently failed. Which just goes to show you that nothing is immune; I've had USB drives fail on me before as well. Especially tails partitions, which is a perfect place to keep wallet backups.

But let me tell you what this is really about.  People seem to have seized upon the number $10,000 and the word "wallet" as if I had said I had $10,000 in the wallet. I did not say that. In fact someone on IRC told me I had said that (it was fluffypony, more on him below) -- when I pointed out to him that I did not say that, he told me "his point still stands" -- no, it does not stand. It never did. It was a troll. You KNOW it was a troll because of the ridiculous, stupid things people said about my post, such as I was lazy and retarded because had "edited dates on the fly". Please. Don't you realize how dumb that sounds? The only reason I didn't just delete the line is because I have integrity and wanted to show I had made a mistake in my post. Lord knows you don't want trolls stating "OMG he changed the date!! He's a liar!" because oh wow, 10 seconds between edits makes a lie. Nevermind the fact that actually reading the post you would realize the words "May 29th" were a mistake because I mention that the server went down two months ago and half the story is how I have been trying to recover it since then. People, think.

The actual wallet had few bitcoins -- certainly less than three -- if anything. Nothing was in production. I was completely justified in the backup solution I chose. When I said realized losses it was a specific financial term. Now that I have pointed that out I expect you to look it up on investopedia and to understand what I said. What I said was (in short form), we marked down the remaining value of our server and IP, plus the lost data in man hours. Easily $10,000. I had lost at least two weeks' worth of coding. It does not mean the company lost $10,000. In fact, what you should have read from this is that I was exceedingly competent in judging the amount of money we spent on our backup system based on how much was lost.

Now that I have recovered everything except the wallet file the actual loss should be pegged at less than $2,000, twelve hundred of which is the cost point of our new backup system.

Remember the OP?

I may have slightly panicked; they have assured me a new server can be up in about a day. Fingers crossed!

Now let's take things from the troll's point of view, to show just how stupid they have been. How would people have felt if I had posted "We had a server crash, but our $10,000 backup system luckily saved the day!!" You should feel horrified, but instead you probably would feel elated that someone had the good idea to properly back things up. In the industry we call this "ass-backwards". It's difficult to understand, I know, but we lost less money in this crash than we would have if we had initially purchased a $10,000 backup system. And, as mentioned several times now, I just spent $1200 on a new DS-4 and drives. I assure you that this is a risk vs. reward scenario and that there is no one better qualified to explain this to you than ME.

The other thing is the security of the "web wallet". Which had not been under development or production for more than one year. It's last activity was being sold to someone on the OTC network. They couldn't get it to work so I offered them a full refund. What a noble thing for me to do, step out like that for members of the community.

This is what I find so humorous about people like floppypony and team -- they just don't get it. I did the all the right things -- including not overspending on financially unjustifiable redundancy. But it gets worse. Did you know fluffypony is actually a well-known scammer who has ripped off dozens of people with his openrigs scam? I know -- what's he even doing here?

"Extensive research has uncovered many users complaining about not receiving their orders. Out of all 20 users I found, only two users ever received their frame. One of the users who received his frame stated that its build quality was very poor, and included pictures to substantiate their claims."

http://www.reddit.com/r/vertcoin/comments/2590id/hello_members_of_the_cryptocurrency_community/
https://www.bitcoinregime.com/2014/05/10/hello-members-of-the-cryptocurrency-community-important-info-about-openrigs-scam-please-read/

[stuvwxyz99]

Couple of points.

1. I don't think you need to spend $10k on backups, nor have I seen anyone advocate spending this amount. 

2. You say you have regular offsite backups (one presumes these don't cost $10k) but if that is the case, why has the wallet not been recovered?

[usagi]

Couple of points.

1. I don't think you need to spend $10k on backups, nor have I seen anyone advocate spending this amount.  

2. You say you have regular offsite backups (one presumes these don't cost $10k) but if that is the case, why has the wallet not been recovered?

1. The problem is, "a couple of USB sticks" are no safer, and may in fact be less safe, than remote off-site backups. It may not look that way to you, but I deal with a lot of data and I have seen a lot of corrupted USB sticks. I have never seen a remote array for backups go down. So how much should be spent? A "home" solution like a DS-4 is what I decided to go with, and I know for a fact this system is going to be less reliable than the kind of remote backups I had previously. Keep in mind. This is not a production system nor is it any kind of "main wallet". It was the claim address I used for GLBSE several years ago. Do you understand now?

2. I don't know. The last update I have is that there is someone looking for the disk right now.

But frankly I don't even care. I was much more worried about the time I had spent writing all those wordpress plugins and the lost content of my obligatory gaijin/chinglish blog. Don't get the wrong idea -- I'm not being uncaring or flippant -- it's others that are reading far, far too much into what happened.

The main point of the announcement was so that people didn't send any money to the old claim address. I don't see any serious problems remaining to discuss -- do you?

[fluffypony]

This is what I find so humorous about people like floppypony and team -- they just don't get it. I did the all the right things -- including not overspending on financially unjustifiable redundancy. But it gets worse. Did you know fluffypony is actually a well-known scammer who has ripped off dozens of people with his openrigs scam? I know -- what's he even doing here?

"Extensive research has uncovered many users complaining about not receiving their orders. Out of all 20 users I found, only two users ever received their frame. One of the users who received his frame stated that its build quality was very poor, and included pictures to substantiate their claims."

http://www.reddit.com/r/vertcoin/comments/2590id/hello_members_of_the_cryptocurrency_community/
https://www.bitcoinregime.com/2014/05/10/hello-members-of-the-cryptocurrency-community-important-info-about-openrigs-scam-please-read/

You've caught me out! Oh dear. Guess I join the hallowed halls of scammers. Let me know where our monthly meeting is, usagi - I look forward to being among such revered company!

[Sydboy]

Apologies again as I mentioned in your other thread.
I was totally out of line.

good luck friend..