Bitcoin Forum
November 05, 2024, 01:48:07 AM *
News: Latest Bitcoin Core release: 28.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Running on the Google App Engine  (Read 14649 times)
Gavin Andresen (OP)
Legendary
*
qt
Offline Offline

Activity: 1652
Merit: 2301


Chief Scientist


View Profile WWW
June 12, 2010, 04:09:43 AM
Last edit: June 14, 2010, 05:53:01 PM by gavinandresen
 #1

Joozero asked if I'd make the source to freebitcoins.appspot.com available, and I promised to start a thread explaining how I created it-- and here it is.

freebitcoins is running on the Google App Engine.  Well, the part you see is running on App Engine.

There's also a back-end bitcoind server running on a debian VPS that I've had for years.  The App Engine code makes JSON-RPC calls via the App Engine url fetching API.  Making that connection secure was a bit tricky; here's how I did it:

First, I connect to the VPS using https so the traffic is encrypted.

I need a secure connection because I add a secret pre-shared value to the JSON call.  I'm not going to make the full source for freebitcoins fully open source because I don't want to bother with constantly working to keeping that secret value secret.    I should hash the entire request with a secret key and an ever-increasing nonce, but I was lazy and I'm just not very worried about a man-in-the-middle attack between Google and my VPS.

I could have hacked my copy of the bitcoind C++ code to check for the secret value in requests and also modified it so it accepted connections from the Internet... but instead I wrote this little proxy server in Python that runs on the same box as bitcoind:
Code:
#
# WSGI Proxy into the bitcoind daemon (which only listens on 127.0.0.1 for security).
# Rejects non-HTTPs connections, or non-POST requests
# Rejects queries that don't include a pre-shared secret value
#
# Apache calls this because of a directive in /etc/apache2/sites-available/xyz.com
#

import hashlib
import urllib2

def application(environ, start_response):
  serverURL = "http://127.0.0.1:8332/"

  def error_response(message):
    response_headers = [('Content-type', 'text/plain'),
            ('Content-Length', str(len(message)))]
    start_response('500 Internal Server error', response_headers)
    return message

  if environ.get("HTTPS") != "1":
    return error_response("Insecure connections not allowed.")

  request = environ.get("wsgi.input").read()
  secret = request[0:32]
  json_request = request[32:]

  if hashlib.md5(" pre shared secret goes here "+json_request).hexdigest() != secret:
    return error_response("Authentication failed.")

  req = urllib2.Request(serverURL, json_request)
  response = urllib2.urlopen(req)
  json_response = response.read()

  status = '200 OK'

  response_headers = [('Content-type', 'text/plain'),
            ('Content-Length', str(len(json_response)))]
  start_response(status, response_headers)

  return [json_response]
The other end of the connection is also django+Python code running on App Engine:
Code:
import hashlib
import jsonrpc
from google.appengine.api import urlfetch

def make_bitcoin_request(server_url, server_secret, method, params):
  json_request = {
    "jsonrpc": "2.0",
    "id": str(time.time()),
    "method": method, "params": params
    }
 
  json_string = jsonrpc.dumps(json_request)
  secret = hashlib.md5(server_secret+json_string).hexdigest()
 
  try:
    fetch_result = urlfetch.fetch(payment_server,
                                  method="POST", payload=secret+json_string,
                                  headers={ 'Content-Type' : 'text/plain' })
  except urlfetch.Error, e:
    logging.error('make_bitcoin_request failed: '+str(e))
    logging.error('Request:'+json_string)
    return None
 
  if fetch_result.status_code != 200:
    logging.error('make_bitcoin_request failed; urlfetch status code %d'%(fetch_result.status_code))
    logging.error('Request:'+json_string)
    return None

  r = jsonrpc.loads(fetch_result.content)
  if r['error'] is not None:
    logging.error('make_bitcoin_request failed; JSON error returned')
    logging.error('Request:'+json_string)
    logging.error('Result:'+fetch_result.content)
    return None

  return r['result']

I'm happy with how it is working.  My biggest worry is that the bitcoind process might unexpectedly exit, or fill up the disk with debug.log messages, or just generally be flaky.  But so far so good...

How often do you get the chance to work on a potentially world-changing project?
lachesis
Full Member
***
Offline Offline

Activity: 210
Merit: 105


View Profile
June 12, 2010, 05:46:40 AM
 #2

So basically, you're running Bitcoin on a VPS and using an HTTPS connection from App Engine to signal it what to do? Why not just host the site on your server?

Bitcoin Calculator | Scallion | GPG Key | WoT Rating | 1QGacAtYA7E8V3BAiM7sgvLg7PZHk5WnYc
Gavin Andresen (OP)
Legendary
*
qt
Offline Offline

Activity: 1652
Merit: 2301


Chief Scientist


View Profile WWW
June 12, 2010, 11:51:33 AM
 #3

So basically, you're running Bitcoin on a VPS and using an HTTPS connection from App Engine to signal it what to do? Why not just host the site on your server?
App Engine gives me:
 Free bandwidth and data storage (with the option to pay for what I use if I go over a gigabyte)
 Built-in, already-configured-and-tweaked tools like memcache and BigTable
 An architecture where scaling up is built-in (App Engine will automagically replicate my app and distribute it across Google's worldwide content distribution network if it gets a lot of traffic)

And all of that is administered by Google's sysadmins (who are WAY better at that than I am).

But mostly I did it as an experiment and because it was a technical challenge and because lately I'm an App Engine fanboy.  I would've been done a day earlier if I'd run everything on the same server and wrote the front end in PHP+MySQL.

How often do you get the chance to work on a potentially world-changing project?
jimbobway
Legendary
*
Offline Offline

Activity: 1304
Merit: 1015



View Profile
July 13, 2010, 10:29:14 PM
 #4

Is it possible to rewrite portions of bitcoin so it runs entirely in GAE?  Particularly the feature that handles transactions, so it is fully scalable?

I am thinking one can get the sourcecode from SourceForge and write the C code into GAE language (Python or Java).
mizerydearia
Hero Member
*****
Offline Offline

Activity: 574
Merit: 513



View Profile
September 22, 2010, 12:52:10 AM
Last edit: September 22, 2010, 01:38:17 PM by mizerydearia
 #5

Apparently it is considering at http://pastebin.com/10k74rSd (which is a summary of Bitcoin client hosts in #bitcoin on LFNet as of about 15 minutes ago) lines 432-529 are all GAE.

How did u generate that list?
http://bitcointalk.org/index.php?topic=1068.msg13665#msg13665
jimbobway
Legendary
*
Offline Offline

Activity: 1304
Merit: 1015



View Profile
September 22, 2010, 01:33:19 AM
 #6

Apparently it is considering at http://pastebin.com/10k74rSd (which is a summary of Bitcoin client hosts in #bitcoin on LFNet as of about 15 minutes ago) lines 432-529 are all GAE.


How did u generate that list?
Cdecker
Hero Member
*****
Offline Offline

Activity: 489
Merit: 505



View Profile WWW
September 22, 2010, 12:48:43 PM
 #7

Is it possible to rewrite portions of bitcoin so it runs entirely in GAE?  Particularly the feature that handles transactions, so it is fully scalable?

I am thinking one can get the sourcecode from SourceForge and write the C code into GAE language (Python or Java).
We are in fact trying to build a small network module to interface the BitCoin Protocol in Python, if you'd like to help feel free to contribute: http://bitcointalk.org/index.php?topic=231.0

Want to see what developers are chatting about? http://bitcoinstats.com/irc/bitcoin-dev/logs/
Bitcoin-OTC Rating
TiagoTiago
Hero Member
*****
Offline Offline

Activity: 616
Merit: 500


Firstbits.com/1fg4i :)


View Profile
March 11, 2011, 09:06:01 AM
 #8

Why not have your thing read the secret from a file, so you can release all the code, but leave the secret key to your machines still hidden?

(I dont always get new reply notifications, pls send a pm when you think it has happened)

Wanna gimme some BTC/BCH for any or no reason? 1FmvtS66LFh6ycrXDwKRQTexGJw4UWiqDX Smiley

The more you believe in Bitcoin, and the more you show you do to other people, the faster the real value will soar!

Do you like mmmBananas?!
stevenroose
Full Member
***
Offline Offline

Activity: 530
Merit: 100


https://www.pax-coin.io/


View Profile
April 22, 2013, 08:15:17 PM
 #9

You probably already found it out and this topic is extremely old, I know. But:

Why not have your thing read the secret from a file, so you can release all the code, but leave the secret key to your machines still hidden?

App Engine doesn't support file I/O. You could write it to the Blob store but that's probably too much overhead.

PAXCOIN     Cryptocurrency 2.0 for Real Economy  Join our ICO: 
  ──  Whitepaper ⬝  Twitter Telegram ⬝   Facebook ⬝  ANN Thread   ──
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!