Tutorial - Electrum Server on Amazon EC2 - Step by Step.

[BkkCoins]

Creating an Electrum (pruning) Server Amazon EC2 instance and AMI.

I chose to run an Electrum server on EC2 because I already use this service for other work. It's not the best bang-for-buck performance wise but it does have it's advantages. It's proven to be quite robust in my past use and I like that I only pay according to my use as I start/stop instances. It's easy to make snapshots and experiment with.

This step-by-step tutorial shows you how to create an Electrum Server AMI (Amazon Machine Image). An AMI is a ready-to-run virtual machine that you can start when desired, or leave running always. Using spot instances you accept some variability in uptime for a much lower running cost. At this time the monthly cost to run an AMI like this is roughtly $6.84 ($5.04 cpu + $1.80 storage). It's possible to have a small data transfer fee as well (if very busy).

I use timkay's handy cmd line tool here. EC2 operations can also be done easily from the AWS (Web) Console, point and click style, but I won't explain here which buttons and clicks are equivalent. Each command has some equivalent on the web interface.

1. If you want the timkay ec2 cmd line tools.

Code:

cd ~
wget http://github.com/timkay/aws/raw/master/aws
nano .awssecret  (put your EC2 access key and secret key here)
chmod 600 .awssecret
sudo perl aws --install  (creates links for short names)
rm aws
ec2din --simple (test it works)

You can put --simple in a ~/.awsrc file if you always want simple listings.

2. Start an Ubuntu 12.04 server spot instance.

I suggest using a high CPU instance (c1.medium) for compiling and initial blockchain download (saves much time, costs a few cents more), and then switching down to a regular small instance (m1.small) for economy during normal operation (t1.micro is just too small). The blockchain is on a separate data volume so can be detached and attached as needed. This is the process I follow below.

Before requesting a spot instance you'll want to be sure you have a security group with the right port ranges open. If you have one already you can specify the group. If not then this will create a group for electrum with suitable ports, and then use that group when startign the instance.

Code:

ec2addgrp electrum -d "allow electrum ports"
ec2auth electrum -P icmp -s 127.0.0.0/8
ec2auth electrum -P tcp -p 0-65535 -s 127.0.0.0/8
ec2auth electrum -P udp -p 0-65535 -s 127.0.0.0/8
ec2auth electrum -P tcp -p 22 -s 0.0.0.0/0
ec2auth electrum -P tcp -p 8333 -s 0.0.0.0/0
ec2auth electrum -P tcp -p 8081-8082 -s 0.0.0.0/0
ec2auth electrum -P tcp -p 50001-50002 -s 0.0.0.0/0
ec2dgrp |less -S  (check your results)

To request a spot instance we specify AMI, max price, type, security group and login key name. This AMI code is for the Ubuntu 12.04 64bit virtual server image.

Code:

ec2rsi ami-9c78c0f5 -p 0.02 -t c1.medium -g electrum -k MyKey
ec2dsir --simple   (check for status of spot request)
ec2din --simple    (check if instance is running, takes a few minutes sometimes, and copy the public url output)

3. Login to the new instance.

You can make login cmd simpler by editing your .ssh/config file to, eg.

Code:

Host *.compute-1.amazonaws.com
User ubuntu
IdentityFile ~/.ssh/MyKey.pem 

Then login with only,

Code:

ssh <public-url-from-above>

4. Prepare your tools - build.

For Ubuntu 12.04 then you need to add this line to your /etc/apt/sources.list (for python-leveldb).

Code:

deb http://archive.ubuntu.com/ubuntu precise-backports main universe

Code:

sudo apt-get update
sudo apt-get install git htop build-essential libssl-dev libboost-all-dev libdb5.1++-dev python-leveldb python-setuptools
cd /usr/src
sudo git clone http://github.com/bitcoin/bitcoin
sudo git clone http://github.com/spesmilo/electrum-server
sudo easy_install jsonrpclib
sudo chown -R ubuntu: *
cd bitcoin
patch -p1 <../electrum-server/patch/patch
cd src
make -f makefile.unix  USE_UPNP=-
sudo cp bitcoind /usr/bin/
sudo ln -s /usr/src/electrum-server/server.py /usr/bin/electrum

5. Create a data volume and attach to server. (to hold blockchain data)

At this writing the bitcoin/electrum data is about 6GB so I'm creating an EBS data volume of 10G. This will be big enough for a while anyway.

Again these cmds use the timkay tools; you can use the AWS Console, your choice.

Code:

ec2din |less -S  (copy the zone id and instance id for your server)
ec2cvol --size 10 --zone <your-zone-id>  (create  volume, choose zone to match your instance)
ec2dvol |less -S   (check for status, repeat until the new volume is available, copy the volume id given)
ec2attvol <volume-id-above> -i <instance-id-from-above> -d /dev/sdf  (attach to your instance as device /dev/sdf)

sudo mkfs -t ext4 /dev/xvdf  (for some odd reason /dev/sdf maps to /dev/xvdf)
mkdir /home/ubuntu/.bitcoin
sudo mount /dev/xvdf /home/ubuntu/.bitcoin
mkdir /home/ubuntu/.bitcoin/electrum_db

6. Configure and start bitcoin.

Let it run until the blockchain is up to date (a long time!).

Code:

cd ~
sudo chown ubuntu: .bitcoin
cat << END > /home/ubuntu/.bitcoin/bitcoin.conf
rpcuser=electrum
rpcpassword=<long-rpc-pwd-here>
END

bitcoind -daemon

You should see a messge that bitcoind is starting and you can monitor it with the getinfo cmd. The blockchain should be far along before you can start Electrum, but you can continue with SSL certs and config files while waiting.

Code:

bitcoind getinfo

7. Create an SSL certificate and key.

There are 4 connection modes in Electrum client TCP,TCP/SSL,HTTP,HTTPS. A certificate is required for the server to start. You can use a self-signed one for TCP/SSL. For HTTPS to work you will need one that is recognized by the client. So either install your own root on the client or get one from a recognized authority. StartSSL.com provides free, but well recognized server certificates. I describe briefly how to create your own self-signed for TCP/SSL mode below.

Code:

openssl genrsa -des3 -out server.key 2048   (use any pwd as we will remove it after)
openssl req -new -key server.key -out server.csr  (answer as suitable, common name is usually your domain name)
sudo openssl rsa -in server.key -out /etc/ssl/electrum.key  (this step removes pwd)
sudo chown ubuntu: /etc/ssl/electrum.key  (since electrum runs as ubuntu we need the key readable by ubuntu)
sudo chmod 600 /etc/ssl/electrum.key  (make sure only owner can read)

(now create the certificate, good for 1 year)
sudo openssl x509 -req -days 365 -in server.csr -signkey server.key -out /etc/ssl/electrum.crt 
rm server.key server.csr

8. Configure and start Electrum.

Start out as private (no irc) for testing.

Code:

sudo su 
cat << END > /etc/electrum.conf
[server]
host = <your-hostname-here> (make sure the hostname resolves or will not bind)
native_port = 50000
stratum_tcp_port:50001
stratum_http_port:8081
ssl_certfile = /etc/ssl/electrum.crt
ssl_keyfile = /etc/ssl/electrum.key
password = <your-long-electrum-pwd-here>
banner = Welcome to Electrum!
irc = no
cache = yes
backend = leveldb

[bitcoind]
host = localhost
port = 8332
user = electrum
password = <long-rpc-pwd-here>

[leveldb]
path = /run/shm/electrum_db
END

cat << END >/usr/bin/start-electrum
#!/bin/bash
nohup /usr/bin/python -u /usr/bin/electrum &>> /var/log/electrum.log &
END

chmod +x /usr/bin/start-electrum
touch /var/log/electrum.log
chown ubuntu: /var/log/electrum.log

cat << END >/etc/logrotate.d/electrum
/var/log/electrum.log
/home/ubuntu/.bitcoin/debug.log
{
        rotate 5
        copytruncate
        daily
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
}

END
exit

Edit /etc/electrum.conf to have correct host name and passwords. You can place any "wall" or banner text in the file /etc/electrum.banner.

You can start electrum, but if it catches up with bitcoind block count it will shut down. I started electrum when bitcoind had reached about 170,000 blocks, and this worked for me. Electrum won't start listening for clients until it is fully up to date.

Code:

start-electrum

Check the log file for results. It should show initializing the database, and catching up blocks.

Code:

less /var/log/electrum.log

You can stop Electrum (but it doesn't work until it's caught up and starts listening).

Code:

electrum stop

9. Wait for Electrum to catch up (takes a quite while) and test with a client.

Bitcoin took around 7 hours for me to fully download the blockchain to 211,860. Electrum had reached around block 194,000  by that time. It took roughly another 2 hours for Electrum to fully catch up.

10. Configure both servers to start at boot time.

We initialized electrum with it's data in shared memory (/run/shm/electrum_db) for speed, but we need to move this to the data volume for persistence. So first thing is to move it over, and edit /etc/electrum.conf for the new location.

Code:

electrum stop
mv /run/shm/electrum_db /home/ubuntu/.bitcoin/
sudo sed -i 's/\/run\/shm/\/home\/ubuntu\/.bitcoin/' /etc/electrum.conf

Create an init config so both start on boot. (be sure to change your hostname here)

Code:

sudo su
cat << END >/etc/init/electrum.conf
description     "Start Electrum server"

start on runlevel [2345]
stop on runlevel [016]

exec su -c /usr/bin/init-electrum ubuntu
END

cat << END >/usr/bin/init-electrum
#!/bin/bash
hostname <your-hostname-here>
hostname >/etc/hostname
bitcoind -daemon
until [ "`nc -z localhost 8332;echo $?`" == "0" ]; do sleep 5; done
start-electrum
END

chmod +x /usr/bin/init-electrum

cat << END >> /etc/fstab
/dev/xvdf  /home/ubuntu/.bitcoin  ext4  defaults  0  2
END
exit

Now stop electrum and bitcoind and try out the init manually to be sure it works.

Code:

electrum stop
bitcoind stop
sudo start electrum
htop

In htop you should see bitcoind running. The init script makes sure Electrum only starts after bitcoind is listening on rpc port 8332, so it may take a while (you can see the 5 second polling).

11. Create an AMI from this running instance.

First, now is the time to make Electrum public if you want that for the AMI. Any changes to persist across all instances need to be made before creating the AMI.

Code:

sed -i 's/irc = no/irc = yes/' /etc/electrum.conf

It's possible to create an AMI from the command line but I prefer using the AWS console as I find it is less mistake prone. Open the AWS Console and go to Instance view. Select the correct instance. Choose Actions (above) and Create AMI Image. A panel of options pops up. Enter an AMI name (like Electrum) and description. Click "Create Image".

A snapshot will be created for each the root and data volumes we have. This process takes time and you can view progress on the "Snapshots" and "AMIs" views of the console by clicking the refresh button. During this process the running instance will reboot. After a while the snapshots will complete, and then the new AMI will change status to availble.

12. Start the new AMI as a small instance.

Now you can try starting a working Electrum server from this AMI. This time we'll start a small instance as it costs less to run. Similar to before:

Code:

ec2rsi <new-ami-id-from-console> -p 0.01 -t m1.small -g electrum -k MyKey
ec2dsir --simple   (check for status of spot request)
ec2din --simple    (check if instance is running, takes a few minutes sometimes, and copy the public url output)

Login and check if electrum is up. Try a client to test availability.

Final steps and tweaks.

The AMI created above is not ideal, but it's a starting point for tweaking to perfection. For one, it starts with a snapshot of the data volume and when stopped this new volume will be left existing. What we really need is for the AMI to start alone, attaching the data volume after boot. That way all updates and changes made to the data will persist on the original volume. Another issue is with DNS and IP addresses. This can be solved with an Elastic IP and associating it during the boot process. Finally, it would be nice if the hostname and IP could be given to the instance, when requesting the start up, allowing multiple instances and more flexibility. I have worked out these details but will save that for part two - Seeking Perfection, or upon request.

Please submit feedback or corrections below and I'll try to update as needed.

[flatfly]

Excellent! Thanks for sharing. Perhaps this can be included somewhere in the wiki too.

[Tachikoma]

Great stuff, perhaps you could add this as how-to to the repository. This also explains all the join/quit spam on the irc channel yesterday ^^

[BkkCoins]

Great stuff, perhaps you could add this as how-to to the repository. This also explains all the join/quit spam on the irc channel yesterday ^^

Thanks.

Yes, that was likely me as I started/stopped Electrum quite a bit. Today too, as I'm testing some final details. I'm planning to add a part 2 with details that allow passing [hostname, IP, data volume] as startup values. This makes the image more generic and allows starting multiple servers very easily, or scaling up and down since these aren't hard coded. I'd publish the AMI after but I think it's better for people to make their own for security/trust reasons.

If we want this as a how-to, then once I sync my fork, and update/fix my code, I can add the file and send a pull request for it to be included.

[BkkCoins]

Note I updated the logrotate config file to include the bitcoin debug.log file (as this is getting very big already), and added the option "copytruncate". It seems today after a log rotation occurred that electrum is still writing to the rotated (.1) log apparently indicating it keeps the log file open. So copy truncate should force it to rotate properly. I'm hoping this will work better.

edit: tested a forced rotation and it worked fine.

[Raize]

Does the fee go up if you provision yourself around 250 gig or so? I was contemplating running a p2pool, bitcoind, and electrum all in one.

[BkkCoins]

Does the fee go up if you provision yourself around 250 gig or so? I was contemplating running a p2pool, bitcoind, and electrum all in one.

The fee is around 0.10 per GB per month. So for 250 GB you would pay $25/month. Not really an economical way to get that storage but not bad compared to other similar sites.

Each instance has a local hard disk that is usually 160GB or 350GB depending on the instance type. But that disk is transient, meaning when you terminate the instance it's contents are lost. For persistent storage you would either use S3 or EBS (what I use for the blockchain) and they both cost about the same.

Why would you need so much storage? Not for bitcoin or Electrum, and I'm not aware of p2pool needing much. EC2 is a bit different than running on a VPS or regular server. For a large amount of storage you'd be better with a VPS. Or if you don't need it to be "fast" then have it on some other service that is free/cheap and access it from an instance.

[Raize]

I actually probably wouldn't use it right away. I'm wondering about drastic adoption of Bitcoin, however and scalability issues associated with it. Thanks for the tutorial, I think it pretty much answers my questions, namely that the costs are going to be under $10/month, which is pretty cheap!

[BkkCoins]

Part Two - Tweaking the System

One thing that is important about spot instances is that they can and will get terminated when some user requests many instances at a higher price. Judging by past price history this doesn't happen often but perhaps once every few months the price shoots up for some hours. I'd like to make the Electrum server configuration deal with this more robustly. The way to do that is to use "persistent" spot requests when starting the server, and to make sure that the server can fully self configure during boot up.

After my inital deployment I made a series of config changes that achieve this along with also being able to set unique server details at startup so that multiple independent nodes can started from a single AMI. So what I do is pass a user data string with "<hostname> <ip> <data-volume>" during the spot request. I create a script that runs during network init and does config file substitutions so that each node boots uniquely defined.

Follow steps below to alter/copy the config files as "original" versions containing our variables, and create a network init script. The hosts file is updated so that it ensures the hostname gets resolved correctly to the local IP.

To make this all work smoothly you should allocate an elastic IP and use provided IP value when starting the instance. Our init process will auto-associate the IP and attach and mount the correct blockchain data volume. This doesn't cost extra as long as you don't leave the elastic IP unused, eg. when the server is down.

As before replace <your-host-name> as used before so they get converted to our new #HOSTNAME variable instead. We also replace our init conf and script with new versions below.

Code:

sudo su
sed "s/<your-host-name>/#HOSTNAME/" /etc/electrum.conf >/etc/electrum.conf.orig
sed "s/<your-host-name>/#HOSTNAME/" /etc/electrum.banner >/etc/electrum.banner.orig

cat << END >> /etc/hosts
#ELECTRUM_IP #HOSTNAME
END

cat << END > /etc/network.if-up.d/electrum
#!/bin/bash

# update hostname to resolve to local ip
IP=$(ec2metadata --local-ipv4)
ID=$(ec2metadata --instance-id)

# substitite hostname wherever required
read HOST EIP VOL <<< $(ec2metadata --user-data)
if [ "$HOST" == "unavailable" ]; then
  HOST="<some-default-hostname>"
fi

sed -e "s/#HOSTNAME/${HOST}/" -e "s/#VOLUME/${VOL}/" /etc/init/electrum.conf.orig >/etc/init/electrum.conf
sed "s/#ELECTRUM_IP/${IP} ${HOST}/" /etc/hosts.orig >/etc/hosts
sed "s/#HOSTNAME/${HOST}/" /etc/electrum.conf.orig >/etc/electrum.conf
sed "s/#HOSTNAME/${HOST}/" /etc/electrum.banner.orig >/etc/electrum.banner

# associate elastic ip to this instance
su -c "ec2aad $EIP -i $ID" ubuntu >/dev/null
END

cat << END > /etc/init/electrum.conf.orig
description     "Start Electrum server"

start on runlevel [2345]
stop on runlevel [016]

pre-start script
hostname #HOSTNAME
hostname >/etc/hostname

if [ ! -f /home/ubuntu/.bitcoin/bitcoin.conf ]; then
  su -c "ec2attvol #VOLUME -i $(ec2metadata --instance-id) -d /dev/sdf" ubuntu
  sleep 15
  mount /dev/xvdf /home/ubuntu/.bitcoin
fi
end script

exec su -c /usr/bin/init-electrum ubuntu

END

cat << END > /usr/bin/init-electrum
#!/bin/bash

bitcoind -daemon -logtimestamps
until [ "`nc -z localhost 8332;echo $?`" == "0" ]; do sleep 5; done
start-electrum
END
exit

The script above relies on the timkay tools so you'll either need to use them or modify to use another cmd line tool. I also turn on bitcoind debug timestamps as I found this useful when trying to see what's happening.

Once these changes are made you'll want to use the AWS console to create a new AMI locking in the server changes as a new image. You can start up a test instance from this new AMI by requesting a spot instance but this time you'll need to fill in a user-data value, separated with spaces,

Code:

<my-domain.com> <my-elastic-ip> <block-data-volume>
eg.
super-electrum.com 123.123.123.123 vol-a123b456

If all goes well the instance will start and be fully working at the given domain url. You should check that electrum is up and can be reached by a client. If you selected the "persistent" option during spot request then if/when a forced instance termination occurs a new instance will be created again once the spot price moves below the price you requested.

I'm looking at one further tweak which hopefully I can post step-by-step soon. The AMI above is 8GB by default and I'd like to reduce this to 3GB to reduce costs. I've done this before but I'd like to detail the specifics for this build.

Once again please give feedback or mistakes and I'll udpate as reported.

[BkkCoins]

Just thought I'd update here after receiving my first Amazon bill since starting Electrum on EC2. It turned out to be quite a bit more than expected.

Seems a bitcoind/electrum server uses a fairly high amount of data transfer. I was billed for 56 GB of data out and some million number of I/O ops. Data xfer in is not billed. Result was about $7 more than I thought. The EC2 instance was around $5 for a partial month, and was as expected, but combined with the data xfer, and I had a few snapshots and data volumes kicking about, so ended up paying about $15 for the month. I don't think this is a very economical way to run an Electrum server.

So reverting, I've upgraded one of my regular VPS accounts and moved the server back to that. At Cloud3K I'm paying $7.95/mo. for 1GB RAM and 4 CPU cores (whatever that ends up meaning). Up to 2 TB of data xfer and 50 GB SSD HD space. At least here I don't have to worry about more xfer costs and HD space is already plenty.

Oh well, nice experiment but too costly in the end. Most of the config and setup is the same except the EC2 related items. I could post a revised step by step if that would be useful for anyone, or just follow above, and adjust.

[Raize]

That's too bad it cost more than you were anticipating. Thanks for the tutorial anyway, quite a bit of it it is applicable for more than just Amazon EC2, obviously!

[sounds]

I particularly appreciate the update re: the bill. Perhaps this idea can be revisited in a year or so?

[Red Emerald]

Thanks for the guide.  I did something similar except I installed everything in a virualenv.

Amazon just dropped their prices for a bunch of things.  It should be a little cheaper to run an electrum server now.