Bitcoin Forum
December 04, 2016, 04:39:37 PM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: Running on the Google App Engine  (Read 6200 times)
Gavin Andresen
Legendary
*
qt
Offline Offline

Activity: 1652


Chief Scientist


View Profile WWW
June 12, 2010, 04:09:43 AM
 #1

Joozero asked if I'd make the source to freebitcoins.appspot.com available, and I promised to start a thread explaining how I created it-- and here it is.

freebitcoins is running on the Google App Engine.  Well, the part you see is running on App Engine.

There's also a back-end bitcoind server running on a debian VPS that I've had for years.  The App Engine code makes JSON-RPC calls via the App Engine url fetching API.  Making that connection secure was a bit tricky; here's how I did it:

First, I connect to the VPS using https so the traffic is encrypted.

I need a secure connection because I add a secret pre-shared value to the JSON call.  I'm not going to make the full source for freebitcoins fully open source because I don't want to bother with constantly working to keeping that secret value secret.    I should hash the entire request with a secret key and an ever-increasing nonce, but I was lazy and I'm just not very worried about a man-in-the-middle attack between Google and my VPS.

I could have hacked my copy of the bitcoind C++ code to check for the secret value in requests and also modified it so it accepted connections from the Internet... but instead I wrote this little proxy server in Python that runs on the same box as bitcoind:
Code:
#
# WSGI Proxy into the bitcoind daemon (which only listens on 127.0.0.1 for security).
# Rejects non-HTTPs connections, or non-POST requests
# Rejects queries that don't include a pre-shared secret value
#
# Apache calls this because of a directive in /etc/apache2/sites-available/xyz.com
#

import hashlib
import urllib2

def application(environ, start_response):
  serverURL = "http://127.0.0.1:8332/"

  def error_response(message):
    response_headers = [('Content-type', 'text/plain'),
            ('Content-Length', str(len(message)))]
    start_response('500 Internal Server error', response_headers)
    return message

  if environ.get("HTTPS") != "1":
    return error_response("Insecure connections not allowed.")

  request = environ.get("wsgi.input").read()
  secret = request[0:32]
  json_request = request[32:]

  if hashlib.md5(" pre shared secret goes here "+json_request).hexdigest() != secret:
    return error_response("Authentication failed.")

  req = urllib2.Request(serverURL, json_request)
  response = urllib2.urlopen(req)
  json_response = response.read()

  status = '200 OK'

  response_headers = [('Content-type', 'text/plain'),
            ('Content-Length', str(len(json_response)))]
  start_response(status, response_headers)

  return [json_response]
The other end of the connection is also django+Python code running on App Engine:
Code:
import hashlib
import jsonrpc
from google.appengine.api import urlfetch

def make_bitcoin_request(server_url, server_secret, method, params):
  json_request = {
    "jsonrpc": "2.0",
    "id": str(time.time()),
    "method": method, "params": params
    }
 
  json_string = jsonrpc.dumps(json_request)
  secret = hashlib.md5(server_secret+json_string).hexdigest()
 
  try:
    fetch_result = urlfetch.fetch(payment_server,
                                  method="POST", payload=secret+json_string,
                                  headers={ 'Content-Type' : 'text/plain' })
  except urlfetch.Error, e:
    logging.error('make_bitcoin_request failed: '+str(e))
    logging.error('Request:'+json_string)
    return None
 
  if fetch_result.status_code != 200:
    logging.error('make_bitcoin_request failed; urlfetch status code %d'%(fetch_result.status_code))
    logging.error('Request:'+json_string)
    return None

  r = jsonrpc.loads(fetch_result.content)
  if r['error'] is not None:
    logging.error('make_bitcoin_request failed; JSON error returned')
    logging.error('Request:'+json_string)
    logging.error('Result:'+fetch_result.content)
    return None

  return r['result']

I'm happy with how it is working.  My biggest worry is that the bitcoind process might unexpectedly exit, or fill up the disk with debug.log messages, or just generally be flaky.  But so far so good...

How often do you get the chance to work on a potentially world-changing project?
1480869577
Hero Member
*
Offline Offline

Posts: 1480869577

View Profile Personal Message (Offline)

Ignore
1480869577
Reply with quote  #2

1480869577
Report to moderator
1480869577
Hero Member
*
Offline Offline

Posts: 1480869577

View Profile Personal Message (Offline)

Ignore
1480869577
Reply with quote  #2

1480869577
Report to moderator
1480869577
Hero Member
*
Offline Offline

Posts: 1480869577

View Profile Personal Message (Offline)

Ignore
1480869577
Reply with quote  #2

1480869577
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1480869577
Hero Member
*
Offline Offline

Posts: 1480869577

View Profile Personal Message (Offline)

Ignore
1480869577
Reply with quote  #2

1480869577
Report to moderator
1480869577
Hero Member
*
Offline Offline

Posts: 1480869577

View Profile Personal Message (Offline)

Ignore
1480869577
Reply with quote  #2

1480869577
Report to moderator
1480869577
Hero Member
*
Offline Offline

Posts: 1480869577

View Profile Personal Message (Offline)

Ignore
1480869577
Reply with quote  #2

1480869577
Report to moderator
lachesis
Full Member
***
Offline Offline

Activity: 210


View Profile
June 12, 2010, 05:46:40 AM
 #2

So basically, you're running Bitcoin on a VPS and using an HTTPS connection from App Engine to signal it what to do? Why not just host the site on your server?

Bitcoin Calculator | Scallion | GPG Key | WoT Rating | 1QGacAtYA7E8V3BAiM7sgvLg7PZHk5WnYc
Gavin Andresen
Legendary
*
qt
Offline Offline

Activity: 1652


Chief Scientist


View Profile WWW
June 12, 2010, 11:51:33 AM
 #3

So basically, you're running Bitcoin on a VPS and using an HTTPS connection from App Engine to signal it what to do? Why not just host the site on your server?
App Engine gives me:
 Free bandwidth and data storage (with the option to pay for what I use if I go over a gigabyte)
 Built-in, already-configured-and-tweaked tools like memcache and BigTable
 An architecture where scaling up is built-in (App Engine will automagically replicate my app and distribute it across Google's worldwide content distribution network if it gets a lot of traffic)

And all of that is administered by Google's sysadmins (who are WAY better at that than I am).

But mostly I did it as an experiment and because it was a technical challenge and because lately I'm an App Engine fanboy.  I would've been done a day earlier if I'd run everything on the same server and wrote the front end in PHP+MySQL.

How often do you get the chance to work on a potentially world-changing project?
jimbobway
Legendary
*
Offline Offline

Activity: 1380



View Profile
July 13, 2010, 10:29:14 PM
 #4

Is it possible to rewrite portions of bitcoin so it runs entirely in GAE?  Particularly the feature that handles transactions, so it is fully scalable?

I am thinking one can get the sourcecode from SourceForge and write the C code into GAE language (Python or Java).

░░░░░░░░░██████░░░░░░░░░░░░▄▄▄
░░███░░██████░░░░░▄▄▄▄▄░░██
░░███░░█████████████
░░░░░░░░██████▀▀██████████
░░░░░░░░██████░░░░░██████████
░░░░░░▄▄▄▄▄▄░░░▄▄▄░░░░███████
░░░░░██████░░░███░░░░███████
░░░░░██████░░░███
░░░░░███████▄▄▄▄▄████████
░░░░░████████████████████
░░▄▄▄▄▄░░█████░░░░█████████
█████░░█████░░░░█████████
█████░░░░░░░░░░░░█████████
█████░░░░░░░░░░░░░█████████

START GETTING PAID FOR YOUR ATTENTION!
███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
JUSTICE IN THE WORLD OF ONLINE ADVERTISING!

BUY MASS COIN 】【 ICO PROSPECTUS
VISIT OUR WEBSITE
TWITTER 】【 FACEBOOK 】【 TELEGRAM

mizerydearia
Hero Member
*****
Offline Offline

Activity: 574



View Profile
September 22, 2010, 12:52:10 AM
 #5

Apparently it is considering at http://pastebin.com/10k74rSd (which is a summary of Bitcoin client hosts in #bitcoin on LFNet as of about 15 minutes ago) lines 432-529 are all GAE.

How did u generate that list?
http://bitcointalk.org/index.php?topic=1068.msg13665#msg13665
jimbobway
Legendary
*
Offline Offline

Activity: 1380



View Profile
September 22, 2010, 01:33:19 AM
 #6

Apparently it is considering at http://pastebin.com/10k74rSd (which is a summary of Bitcoin client hosts in #bitcoin on LFNet as of about 15 minutes ago) lines 432-529 are all GAE.


How did u generate that list?

░░░░░░░░░██████░░░░░░░░░░░░▄▄▄
░░███░░██████░░░░░▄▄▄▄▄░░██
░░███░░█████████████
░░░░░░░░██████▀▀██████████
░░░░░░░░██████░░░░░██████████
░░░░░░▄▄▄▄▄▄░░░▄▄▄░░░░███████
░░░░░██████░░░███░░░░███████
░░░░░██████░░░███
░░░░░███████▄▄▄▄▄████████
░░░░░████████████████████
░░▄▄▄▄▄░░█████░░░░█████████
█████░░█████░░░░█████████
█████░░░░░░░░░░░░█████████
█████░░░░░░░░░░░░░█████████

START GETTING PAID FOR YOUR ATTENTION!
███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
JUSTICE IN THE WORLD OF ONLINE ADVERTISING!

BUY MASS COIN 】【 ICO PROSPECTUS
VISIT OUR WEBSITE
TWITTER 】【 FACEBOOK 】【 TELEGRAM

Cdecker
Hero Member
*****
Offline Offline

Activity: 487



View Profile WWW
September 22, 2010, 12:48:43 PM
 #7

Is it possible to rewrite portions of bitcoin so it runs entirely in GAE?  Particularly the feature that handles transactions, so it is fully scalable?

I am thinking one can get the sourcecode from SourceForge and write the C code into GAE language (Python or Java).
We are in fact trying to build a small network module to interface the BitCoin Protocol in Python, if you'd like to help feel free to contribute: http://bitcointalk.org/index.php?topic=231.0

Want to see what developers are chatting about? http://bitcoinstats.com/irc/bitcoin-dev/logs/
Bitcoin-OTC Rating
TiagoTiago
Hero Member
*****
Offline Offline

Activity: 616


Firstbits.com/1fg4i                :Ƀ


View Profile
March 11, 2011, 09:06:01 AM
 #8

Why not have your thing read the secret from a file, so you can release all the code, but leave the secret key to your machines still hidden?

(I dont always get new reply notifications, pls send a pm when you think it has happened)

Wanna gimme some BTC for any or no reason? 1FmvtS66LFh6ycrXDwKRQTexGJw4UWiqDX Smiley

The more you believe in Bitcoin, and the more you show you do to other people, the faster the real value will soar!

Do you like mmmBananas?!
stevenroose
Member
**
Offline Offline

Activity: 71



View Profile WWW
April 22, 2013, 08:15:17 PM
 #9

You probably already found it out and this topic is extremely old, I know. But:

Why not have your thing read the secret from a file, so you can release all the code, but leave the secret key to your machines still hidden?

App Engine doesn't support file I/O. You could write it to the Blob store but that's probably too much overhead.

There is one thing I will never understand about Bitcoin.
Why didn't Satoshi Nakamoto start with a 100 BTC bounty per block instead of 50. 100 is so much more logical.
But especially, 42 million is a tremendously fancier number for the maximum amount of bitcoins in circulation than the current 21 million.
maraoz
Jr. Member
*
Offline Offline

Activity: 55



View Profile
September 24, 2013, 10:06:05 PM
 #10

Hi Gavin, I was reading this old topic and I wanted to ask: why did you add the secret parameter to the request? Isn't the connection secure already if you use SSL? Thanks!

btw: go google app engine!
I've managed to get https://github.com/jgarzik/python-bitcoinrpc to work on GAE Cheesy

Manuel

Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!