Running bitcoind on Amazon AWS (EC2) help

[newguy05]

Hi guys, total newbie to amazon's AWS EC2 instances, but am very familiar with linux. Have some questions if anyone can help, thanks!

I have setup a micro EC2 instance using amazon's free trial offer (installed amazon's ami linux), downloaded and have bitcoind running on the server. I can see in the log it's syncing up.  Here are my questions:

1) From my understanding, EC2 instance is not perm, as soon as it shuts down everything is wiped. Which means as soon as it shuts down all the data from bitcoind is wiped and next time i have to resync from block 1.  I must not be using the right tool (EC2), can someone point me in the right direction on what exactly i should be using to fix this? I just want a regular/vps linux server that doesn't wipe the data every time it goes down.

2) Bitcoind is doing massive amount of sync on the network, i can see the network spike up in amazon's monitoring tool, is there an easy way to tell how much in/out network data i have used vs the threshold before i get charged. I clicked on all their links/monitoring but can't figure out where they keep this info.  I plan to leave this running in the background but don't want to get hit with a surprise bill a week later.

Thanks!

[cr1776]

You may want to look at the ec2 ebs which is persistent. I use it for two of my servers (not bitcoin) an for 6 years now have been happy with its reliability.

As far as usage goes, I am not sure off the top of my head it has been so long.

[newguy05]

You may want to look at the ec2 ebs which is persistent. I use it for two of my servers (not bitcoin) an for 6 years now have been happy with its reliability.

As far as usage goes, I am not sure off the top of my head it has been so long.

thanks! looks like by default it is already running on EBS, it auto mounted 8 GB of space from EBS when the EC2 instance was created. Now just waiting for the block chain to catch up.

It is taking forever and already used up 3GB of space.   

I wonder if this will be the major flaw of bitcoins, say 20 years from now, how the heck on earth will any new client able to catch up, it will take weeks and tons of space.

[kostagr33k]

I actually did this last week. It took days, and I kept blowing away my bitcoind while tweaking things.

I eventually gave in and downloaded the torrent for the recent blocks, but even that took a long time to import, as Disk IO on the micro instances is very limited as well as CPU instances ... Note that the Micro instance is not guaranteed performance as it is their "free" tier"

I eventually went to a m1.large for the two day period to sync all blocks, and now seems to be working fine on the Micro instance once it caught up and I reverted.

kosta

[cr1776]

You'll want your own EBS to store the blockchain and it will be persistent.  Amazon has lots of information detailing it.

[thenoblebot]

Whoa .. please make sure you do not exceed the "free tier" limit for bandwidth. Cause then you are charged. Apparently its not all that "free". Check the limits or you might be surprised later.

[newguy05]

I actually did this last week. It took days, and I kept blowing away my bitcoind while tweaking things.

I eventually gave in and downloaded the torrent for the recent blocks, but even that took a long time to import, as Disk IO on the micro instances is very limited as well as CPU instances ... Note that the Micro instance is not guaranteed performance as it is their "free" tier"

I eventually went to a m1.large for the two day period to sync all blocks, and now seems to be working fine on the Micro instance once it caught up and I reverted.

kosta

Thanks guys, Kosta do you know the approx disk space requirement after bitcoind is synced up?

I get 30 gb ebs for free, so unmounted the default 8gb and created a new 21 gb mount to use for bitcoind, hope that is enough space.

This to me is one of the biggest shortfalls of bitcoin as time goes on, imagine 30 years from now and when bitcoin actually becomes popular, you almost need to buy hardware with all the blocks preloaded otherwise no retail users can ever catch up.

[newguy05]

ah crap i am already getting billed by amazon for I/O charges and i am only at the 9M difficulty level, this may not be such a bright idea after all, probably just going to stick with regular dedicated vps for 20 bucks a month


EBS
$0.00 for the first 2 million I/O requests under monthly free tier   2,000,000 IOs(used)   $0.00(charge)
$0.10 per 1 million I/O requests   2,393,049 IOs(used)   $0.24(charge)
Total:   $0.24

[thenoblebot]

ah crap i am already getting billed by amazon for I/O charges and i am only at the 9M difficulty level, this may not be such a bright idea after all, probably just going to stick with regular dedicated vps for 20 bucks a month


EBS
$0.00 for the first 2 million I/O requests under monthly free tier   2,000,000 IOs(used)   $0.00(charge)
$0.10 per 1 million I/O requests   2,393,049 IOs(used)   $0.24(charge)
Total:   $0.24

told ya not everything is "free" in the "Free tier".

[e4xit]

--snip--
This to me is one of the biggest shortfalls of bitcoin as time goes on, imagine 30 years from now and when bitcoin actually becomes popular, you almost need to buy hardware with all the blocks preloaded otherwise no retail users can ever catch up.
--snip--

Really? Do you not think connection speeds will increase? That seems to me like it might be some serious short-sightedness on you part if I'm honest...

Luckily the architect did not envision that **every** user requires a full copy of the blockchain, only enough to keep it decentralised. Prefessionals, hobbyists and those with the spare resources may keep full purchase ledgers, while "average Joe's" will use light/SPV clients.

/topic derailment.

Also, nice work hosting a bitcoind instance on EC2! 

[moni3z]

You can run this Go implementation of bitcoind with google app engine but who knows what the costs will be. It purposely doesn't include wallet functionaity though https://github.com/conformal/btcd the wallet is here https://github.com/conformal/btcwallet

[newguy05]

You can run this Go implementation of bitcoind with google app engine but who knows what the costs will be. It purposely doesn't include wallet functionaity though https://github.com/conformal/btcd the wallet is here https://github.com/conformal/btcwallet

interesting i will take a look, yeah i looked at google app engine first but it's not a true vps like aws and has limitations in place that prevent bitcoind to run easily without some major tinkering.

In the end i think a regular vps is still my best(economical) option to keep bicoind running 24/7. Something like linode that cost $20 a month and not have to worry about i/o limits etc...

[thenoblebot]

You can run this Go implementation of bitcoind with google app engine but who knows what the costs will be. It purposely doesn't include wallet functionaity though https://github.com/conformal/btcd the wallet is here https://github.com/conformal/btcwallet

interesting i will take a look, yeah i looked at google app engine first but it's not a true vps like aws and has limitations in place that prevent bitcoind to run easily without some major tinkering.

In the end i think a regular vps is still my best(economical) option to keep bicoind running 24/7. Something like linode that cost $20 a month and not have to worry about i/o limits etc...

Which vps do you use ? cause other server options like digital ocean, rackspace, etc have bandwidth limitations.

[kostagr33k]

Sorry newguy05.

Its about 16gb for the blocks folder alone:

root@i .bitcoin]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvde1       30G   19G   11G  65% /
none            296M     0  296M   0% /dev/shm

[root@i .bitcoin]# du -chs
17G   .
17G   total


Kosta

I actually did this last week. It took days, and I kept blowing away my bitcoind while tweaking things.

I eventually gave in and downloaded the torrent for the recent blocks, but even that took a long time to import, as Disk IO on the micro instances is very limited as well as CPU instances ... Note that the Micro instance is not guaranteed performance as it is their "free" tier"

I eventually went to a m1.large for the two day period to sync all blocks, and now seems to be working fine on the Micro instance once it caught up and I reverted.

kosta

Thanks guys, Kosta do you know the approx disk space requirement after bitcoind is synced up?

I get 30 gb ebs for free, so unmounted the default 8gb and created a new 21 gb mount to use for bitcoind, hope that is enough space.

This to me is one of the biggest shortfalls of bitcoin as time goes on, imagine 30 years from now and when bitcoin actually becomes popular, you almost need to buy hardware with all the blocks preloaded otherwise no retail users can ever catch up.

[coinrevo]

at digitalocean you can run a node for 5$/month (with added swap space). probably cheaper.

[thenoblebot]

at digitalocean you can run a node for 5$/month (with added swap space). probably cheaper.

would that be enough for a fully functional node with all the normal traffic of a website ? I believe they offer 1 TB of traffic with $5/month package.

[lauren2014]

at digitalocean you can run a node for 5$/month (with added swap space). probably cheaper.

i agree, aws is expensive sometimes, and not that much more reliable.
ive had many websites down on aws

[moni3z]

Whatever you use be sure to scrub data before you abandon or cancel your VPS or the next user may be able to recover it
https://github.com/fog/fog/issues/2525

[Zeal0t]

I had no idea Amazon has cloud computing services, wow. Thanks for the info!

[U1TRA_L0RD]

I had no idea Amazon has cloud computing services, wow. Thanks for the info!

It sucks... haha, Even google has cloud services which are way better if you ask me.