Is it possible to run Bitcoin on Google App Engine? / alternatives?

[stevenroose]

Can anyone please move this topic to the Development section?
This question is cross-posted on bitcoin.stackexchange, stackoverflow and bitcointalks.


I'm planning to build an application on Google App Engine that will heavily make use of Bitcoin trading. I've been Googling along a little but I couldn't find whether it is possible to run Bitcoin itself on App Engine (with Java). I have some experience with App Engine, but limited to a pure web-app centered usage.
I've read about a few people that have made applications using Bitcoin with App Engine as well that are hosting Bitcoin separately on an Amazon EC2 instance.

So, does anyone here either has experience with running Bitcoin in App Engine for Java or would anyone have an idea how this could possibly be done?
I know there are a lot of Bitcoin applications out there, I'd like to know how these manage their Bitcoin traffic.

I'm trying to avoid needing a separate Amazon service running all the time next to App Engine.

In fact, receiving Bitcoin can easily be done by using passive APIs like blockexplorer or blockchain, so I'm considering to find a reliable API to handle my outgoing payments. But this approach causes extreme dependency on this API service, which I actually want to avoid as much as possible.

[Stephen Gornick]

\So, does anyone here either has experience with running Bitcoin in App Engine for Java or would anyone have an idea how this could possibly be done?
I know there are a lot of Bitcoin applications out there, I'd like to know how these manage their Bitcoin traffic.

I'm trying to avoid needing a separate Amazon service running all the time next to App Engine.

If what you are after is the blockchain there might be a way to replicate the data into App Engine's Google Cloud SQL (from libbitcoin's MySQL data, perhaps).  

But if you are wanting to put a wallet on App Engine, here's an idea: Why don't you give me half the bitcoins you were thinking of storing, then we'll go out back, I'll kick you in the nuts and we'll call it a day.



 - http://www.youtube.com/watch?v=CEeqCbEFIJw (length: 10s)
 - http://www.youtube.com/watch?v=lvDu3IysKiM (length: 60s)

i.e.,  If there is any lesson to be learned from the Bitcoinica mess is that shared systems and cloud infrastructure cannot be secured.

[stevenroose]

But if you are wanting to put a wallet on App Engine, why don't you give me half the bitcoins you were thinking of storing, then we'll go out back, I'll kick you in the nuts and we'll call it a day.

i.e.,  If there is any lesson to be learned from the Bitcoinica mess is that shared systems cannot be secured.

What's so less secure on running Bitcoin in App Engine compared to running it on an Amazon instance? In the latter option you have to make a secure connection between App Engine and this instance, which in my eyes leads to the same, if not even more, security issues?

[Stephen Gornick]

What's so less secure on running Bitcoin in App Engine compared to running it on an Amazon instance? In the latter option you have to make a secure connection between App Engine and this instance, which in my eyes leads to the same, if not even more, security issues?

Neither can be secured.

This is some useful reading:
 - http://bitcointalk.org/index.php?topic=81045.msg894870#msg894870

[stevenroose]

Of course we are already working with strong caution and we know that hosting all your coins on the web is dumb.
We were thinking about a daily routine checking what the average amount of bitcoins is required per day. Have this amount + some buffer available for the service and just have all the rest sitting somewhere on an offline address.
This would inevitably mean we have to manually refill the online reserve when it got lower than the calculated required amount, but we are aware of the safety issues of not doing this.

That's also the big reason why I wanted to host on App Engine and not on a self-managed Amazon instance. I trust Google when it comes to server security. I trust them way more than myself, since I'm not a security expert, and also trust them more than I trust Amazon. So I thought App Engine would be the safest solution.

But as ThePiachu pointed out on the Bitcoin StackExchange, hosting a bitcoin client on App Engine is not possible.

[gweedo]

Why don't you get a small VPS, put bitcoind on it, lock it down so only request from and to GAE are possible using the linux firewall, you know lockdown SSH. And you should be good to go.

[Bitcoin Oz]

Dedicated server in a locked cage is the best choice for financial sites.

[stevenroose]

Ok,  thanks!

[MoneyIsDebt]

I don't know about your VPS, but mine can't even fit the blockchain anymore.

[stevenroose]

I don't have anything running yet, so can't tell.

Hmm, yeah the blockchain seems to be growing exponentially, quite frightening...
How much memory does it take to host it? Is it equal to the size listen here https://blockchain.info/charts/blocks-size or is it more?

[John (John K.)]

You're whitelisted, so feel free to post at the Development subforum.

[stevenroose]

Thanks!

I have another question about VPS services.

If a VPS is setup only for your Bitcoin transactions, the Blockchain doesn't have to be loaded into RAM, right?
Most simple VPS services offer at least 10GB in their most basic setup, so that can't be a problem?

I'm not sure and this quote makes me even less sure

I don't know about your VPS, but mine can't even fit the blockchain anymore.

Can anyone confirm / correct?

[gweedo]

Thanks!

I have another question about VPS services.

If a VPS is setup only for your Bitcoin transactions, the Blockchain doesn't have to be loaded into RAM, right?
Most simple VPS services offer at least 10GB in their most basic setup, so that can't be a problem?

I'm not sure and this quote makes me even less sure

I don't know about your VPS, but mine can't even fit the blockchain anymore.

Can anyone confirm / correct?

no 10gb should be good for now, but keep an eye on it and make sure your not going to run out of space in the future.
and if you get 512mb for the ram and only run bitcoind on it you should be good.

[Mike Hearn]

I work for Google. AppEngine servers aren't going to get hacked. However, your admin account might well do. If you want to do that, you should at the very least activate 2-step verification. Even then it's not unhackable - people have managed to access such accounts by taking over the backing phone numbers (eg, by tricking the phone company or starting a number port). So it's not a panacea.

AppEngine is designed to load and unload servers on demand, it's not really designed for a long-running P2P process. But it can still make sense for your app - as mentioned earlier, you can run bitcoind on a locked down machine that only accepts connections from your service, have some Bitcoin related software running there, and run the rest of your webapp logic on AppEngine. The services it provides can still be very useful.

You may find bitcoinj a useful Java library. I wrote it Just be aware, that you will need to keep up to date if you use that, as it still is changing quite fast and will continue to do so for the forseeable future. It might make it easier for you, if you prefer Java - you can just use the class library and point it at a normal bitcoind you run on a hosted machine (or even your own). When events happen, like your balance changing, you can then upload the relevant data to your app and do whatever you want with it there.

[stevenroose]

Oh, nice, thanks!

I can find bitcoinj by googling it I suppose? On Google Code or GitHub somewhere

Yeah we don't have anything ready, but I think we will start working on it in about a month or 2, so thank you!
(I suppose bitcoinj is completely compatible with AppEngine )

Oh btw, You say it can connect to any bitcoind instance running on a vps or so, do I need specific software on that machine? I'm not familiar with running software on a VPS, I usually work in systems like AppEngine that are managed by others...

[Mike Hearn]

You can find it at www.bitcoinj.org

bitcoinj is a set of classes that help you use the Bitcoin protocol. It isn't designed for AppEngine specifically. Some classes, like the P2P networking classes and the block stores, won't work with AppEngine. Others just parse data and let you manipulate it.

Knowing more details about what you want to do would help.

If we assume you want to have a site that both sends and receives money automatically (the most dangerous/risky type!), the way I'd do it is to have a regular Linux box - ideally, one you physically own and is in a nearby datacenter so you can drive over to it - that runs normal bitcoind as downloaded from bitcoin.org, and a Java app you wrote using bitcoinj. The Java app connects to your local node and manages a wallet. You'd program it to generate batches of keys and POST the addreses to your appengine app, which would then stuff them into a data store.

Now when you want to receive a payment from a user, you take an unused address from the data store and give it to the user. They send the money, which is noticed by the app on your physically owned machine. It then POSTs a notification to your AE app which can then do whatever it needs to do, having received the payment. For the reverse, your AE app would POST to the dedicated server (your java app can both talk to bitcoin and be a web app).

If you aren't familiar with running your own machines, I'd think twice about hosting wallets. The moment you have significant funds on a machine, you get a massive set of cross-hairs painted on your back.  If you aren't really sharp when it comes to computer security, you'll get robbed. Outsourcing it to other companies is a technique that's failed several times now ...... most hosting operations are not intended for high security applications like Bitcoin financial services. Not even AppEngine.

[stevenroose]

That's why I was considering AppEngine, since they handle security for me.

Aren't there VPS services that just let me run those 2 programs and fix the security themselves? (Preferably not too expensive ofc...)

@Mike, What I want to do is indeed what you suggested. But, I think the traffic of Bitcoin in and out our service would not be soo high (spoken as fraction of the total amount we'd store). So we were thinking just about hosting just the estimated amount needed for one or two days on the server and have the rest send to an offline address. I think most input will cover the output so when we just always have like twice the average output volume hosted it would be ok.
Also, unlike services like wallets we can f.e. say that withdrawing a greater number of coins could take 24 hrs. I think people won't bother if they know their coins are stored more securely that way.

Although I still have to find out how to produce safe offline transactions, I think that can't be that hard. (I suppose it's possible to put the blockchain on an USB stick, put into an offline machine, produce transactions and broadcast them to the network.

About the outsourcing. Yes I think it's really weird that nobody offers a good reliable Bitcoin API for these purposes. Ofc it's quite dangerous to trust them with your money, but some should be reliable enough by now... I considered Blockchain.info's API, they host your private keys encrypted. But I doubted they could handle large amounts of requests.

[Mike Hearn]

It looks like AE has got better since I last looked at it. Now you can have "backends" that are basically long-running Java programs that only occasionally get shut down for load balancing, etc.

bitcoinj still isn't totally suitable for this environment because it assumes it can store the wallet into a file. Nothing stops you from writing code to store it in a database instead, of course, it's just not done for you out of the box.

I think the integration between all these parts will get better over time. You're right that having an API that makes your use case easy would be a good thing, though maybe it shouldn't be too easy to throw other peoples money around :-)

[stevenroose]

It would of course be nice if you as a Googler could provide AE support for bitcoinj

Albeit, I think hosting the complete Blockchain on EA including a separate CPU-consuming instance that has to be awake 100% of the time could cost a dime


BTW, receiving coins is no problem anyway, public APIs like blockexplorer and blockchain are really great and reliable when used together (the one as backup for the other), so the only functionality still absent is a sending API :p I would pay for one, wouldn't that be business? It's a tough business though.

Anyway I go to sleep, I have a few more months to find out. In the worst case we can just let app engine generate a transaction list each day which we create offline and broadcast every day But that'd be too much of a hassle

[MoneyIsDebt]

Is it possible for a third party to host the bitcoind and blockchain? Your app would not share any keys with it of course, but send signed transactions to it, and use it for getting notifications on balance updates etc.