Bitcoin Forum
November 11, 2024, 05:27:35 PM *
News: Latest Bitcoin Core release: 28.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1] 2 »  All
  Print  
Author Topic: Running bitcoind on Amazon AWS (EC2) help  (Read 3874 times)
newguy05 (OP)
Hero Member
*****
Offline Offline

Activity: 1316
Merit: 503


Someone is sitting in the shade today...


View Profile WWW
January 19, 2014, 11:22:35 PM
 #1

Hi guys, total newbie to amazon's AWS EC2 instances, but am very familiar with linux. Have some questions if anyone can help, thanks!

I have setup a micro EC2 instance using amazon's free trial offer (installed amazon's ami linux), downloaded and have bitcoind running on the server. I can see in the log it's syncing up.  Here are my questions:

1) From my understanding, EC2 instance is not perm, as soon as it shuts down everything is wiped. Which means as soon as it shuts down all the data from bitcoind is wiped and next time i have to resync from block 1.  I must not be using the right tool (EC2), can someone point me in the right direction on what exactly i should be using to fix this? I just want a regular/vps linux server that doesn't wipe the data every time it goes down.

2) Bitcoind is doing massive amount of sync on the network, i can see the network spike up in amazon's monitoring tool, is there an easy way to tell how much in/out network data i have used vs the threshold before i get charged. I clicked on all their links/monitoring but can't figure out where they keep this info.  I plan to leave this running in the background but don't want to get hit with a surprise bill a week later.

Thanks!

         ▄██████
       ▄████████
     ▄██████████
   ▄█▀     █████
 ▄███      █████
█████      ███████████████████████████
█████      ███████████████████████████
█████      ███████████████████████████
█████                           ████▀
█████                           ██▀
█████                          ▄▀   
██████████████████████████████▀
████████████████████████████▀
L I N K
by BLOCKMASON





..CREATE WEB APIS........
..FROM ANY SMART........
..CONTRACT.........................






█▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀█
█  ██████    ▄▄▄▄▄▄▄▄ █
█  ██████    ▄▄▄▄▄▄▄▄ █
█  ██████    ▄▄▄▄▄▄▄▄ █
█            ▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█                     █
█ ▄▄▄▄▄▄              █
█▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄█

READ THE
WHITEPAPER

>>>






▄          █▄                         ▄
██▄        ███▄                     ▄██
████▄      █████▄                 ▄████
██████▄    ███████▄             ▄██████
████████▄   ▀███████▄         ▄███████▀
██████████▄   ▀███████▄     ▄███████▀
████████████▄   ▀███████▄ ▄███████▀  
██████████████▄   ▀█████████████▀   ▄██
████████████████▄   ▀█████████▀   ▄████
██████████████████▄   ▀█████▀   ▄██████
████████████████████▄   ▀█▀   ▄████████
BLOCKMASON
BUILDING THE FUTURE









FACEBOOK
REDDIT
TWITTER

LINKEDIN
GITHUB
MEDIUM
cr1776
Legendary
*
Offline Offline

Activity: 4214
Merit: 1313


View Profile
January 19, 2014, 11:52:06 PM
 #2

You may want to look at the ec2 ebs which is persistent. I use it for two of my servers (not bitcoin) an for 6 years now have been happy with its reliability.

As far as usage goes, I am not sure off the top of my head it has been so long.
newguy05 (OP)
Hero Member
*****
Offline Offline

Activity: 1316
Merit: 503


Someone is sitting in the shade today...


View Profile WWW
January 20, 2014, 04:20:29 AM
 #3

You may want to look at the ec2 ebs which is persistent. I use it for two of my servers (not bitcoin) an for 6 years now have been happy with its reliability.

As far as usage goes, I am not sure off the top of my head it has been so long.

thanks! looks like by default it is already running on EBS, it auto mounted 8 GB of space from EBS when the EC2 instance was created. Now just waiting for the block chain to catch up.

It is taking forever and already used up 3GB of space.   

I wonder if this will be the major flaw of bitcoins, say 20 years from now, how the heck on earth will any new client able to catch up, it will take weeks and tons of space.

         ▄██████
       ▄████████
     ▄██████████
   ▄█▀     █████
 ▄███      █████
█████      ███████████████████████████
█████      ███████████████████████████
█████      ███████████████████████████
█████                           ████▀
█████                           ██▀
█████                          ▄▀   
██████████████████████████████▀
████████████████████████████▀
L I N K
by BLOCKMASON





..CREATE WEB APIS........
..FROM ANY SMART........
..CONTRACT.........................






█▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀█
█  ██████    ▄▄▄▄▄▄▄▄ █
█  ██████    ▄▄▄▄▄▄▄▄ █
█  ██████    ▄▄▄▄▄▄▄▄ █
█            ▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█                     █
█ ▄▄▄▄▄▄              █
█▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄█

READ THE
WHITEPAPER

>>>






▄          █▄                         ▄
██▄        ███▄                     ▄██
████▄      █████▄                 ▄████
██████▄    ███████▄             ▄██████
████████▄   ▀███████▄         ▄███████▀
██████████▄   ▀███████▄     ▄███████▀
████████████▄   ▀███████▄ ▄███████▀  
██████████████▄   ▀█████████████▀   ▄██
████████████████▄   ▀█████████▀   ▄████
██████████████████▄   ▀█████▀   ▄██████
████████████████████▄   ▀█▀   ▄████████
BLOCKMASON
BUILDING THE FUTURE









FACEBOOK
REDDIT
TWITTER

LINKEDIN
GITHUB
MEDIUM
kostagr33k
Full Member
***
Offline Offline

Activity: 309
Merit: 100


View Profile
January 20, 2014, 05:23:02 AM
 #4

I actually did this last week. It took days, and I kept blowing away my bitcoind while tweaking things.

I eventually gave in and downloaded the torrent for the recent blocks, but even that took a long time to import, as Disk IO on the micro instances is very limited as well as CPU instances ... Note that the Micro instance is not guaranteed performance as it is their "free" tier"

I eventually went to a m1.large for the two day period to sync all blocks, and now seems to be working fine on the Micro instance once it caught up and I reverted.

kosta
cr1776
Legendary
*
Offline Offline

Activity: 4214
Merit: 1313


View Profile
January 20, 2014, 08:53:49 AM
 #5

You'll want your own EBS to store the blockchain and it will be persistent.  Amazon has lots of information detailing it.
thenoblebot
Sr. Member
****
Offline Offline

Activity: 364
Merit: 252


View Profile
January 20, 2014, 11:56:09 AM
 #6

Whoa .. please make sure you do not exceed the "free tier" limit for bandwidth. Cause then you are charged. Apparently its not all that "free". Check the limits or you might be surprised later.
newguy05 (OP)
Hero Member
*****
Offline Offline

Activity: 1316
Merit: 503


Someone is sitting in the shade today...


View Profile WWW
January 20, 2014, 05:04:39 PM
 #7

I actually did this last week. It took days, and I kept blowing away my bitcoind while tweaking things.

I eventually gave in and downloaded the torrent for the recent blocks, but even that took a long time to import, as Disk IO on the micro instances is very limited as well as CPU instances ... Note that the Micro instance is not guaranteed performance as it is their "free" tier"

I eventually went to a m1.large for the two day period to sync all blocks, and now seems to be working fine on the Micro instance once it caught up and I reverted.

kosta

Thanks guys, Kosta do you know the approx disk space requirement after bitcoind is synced up?

I get 30 gb ebs for free, so unmounted the default 8gb and created a new 21 gb mount to use for bitcoind, hope that is enough space.

This to me is one of the biggest shortfalls of bitcoin as time goes on, imagine 30 years from now and when bitcoin actually becomes popular, you almost need to buy hardware with all the blocks preloaded otherwise no retail users can ever catch up.

         ▄██████
       ▄████████
     ▄██████████
   ▄█▀     █████
 ▄███      █████
█████      ███████████████████████████
█████      ███████████████████████████
█████      ███████████████████████████
█████                           ████▀
█████                           ██▀
█████                          ▄▀   
██████████████████████████████▀
████████████████████████████▀
L I N K
by BLOCKMASON





..CREATE WEB APIS........
..FROM ANY SMART........
..CONTRACT.........................






█▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀█
█  ██████    ▄▄▄▄▄▄▄▄ █
█  ██████    ▄▄▄▄▄▄▄▄ █
█  ██████    ▄▄▄▄▄▄▄▄ █
█            ▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█                     █
█ ▄▄▄▄▄▄              █
█▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄█

READ THE
WHITEPAPER

>>>






▄          █▄                         ▄
██▄        ███▄                     ▄██
████▄      █████▄                 ▄████
██████▄    ███████▄             ▄██████
████████▄   ▀███████▄         ▄███████▀
██████████▄   ▀███████▄     ▄███████▀
████████████▄   ▀███████▄ ▄███████▀  
██████████████▄   ▀█████████████▀   ▄██
████████████████▄   ▀█████████▀   ▄████
██████████████████▄   ▀█████▀   ▄██████
████████████████████▄   ▀█▀   ▄████████
BLOCKMASON
BUILDING THE FUTURE









FACEBOOK
REDDIT
TWITTER

LINKEDIN
GITHUB
MEDIUM
newguy05 (OP)
Hero Member
*****
Offline Offline

Activity: 1316
Merit: 503


Someone is sitting in the shade today...


View Profile WWW
January 20, 2014, 06:24:22 PM
 #8

ah crap i am already getting billed by amazon for I/O charges and i am only at the 9M difficulty level, this may not be such a bright idea after all, probably just going to stick with regular dedicated vps for 20 bucks a month


EBS
$0.00 for the first 2 million I/O requests under monthly free tier   2,000,000 IOs(used)   $0.00(charge)
$0.10 per 1 million I/O requests   2,393,049 IOs(used)   $0.24(charge)
Total:   $0.24

         ▄██████
       ▄████████
     ▄██████████
   ▄█▀     █████
 ▄███      █████
█████      ███████████████████████████
█████      ███████████████████████████
█████      ███████████████████████████
█████                           ████▀
█████                           ██▀
█████                          ▄▀   
██████████████████████████████▀
████████████████████████████▀
L I N K
by BLOCKMASON





..CREATE WEB APIS........
..FROM ANY SMART........
..CONTRACT.........................






█▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀█
█  ██████    ▄▄▄▄▄▄▄▄ █
█  ██████    ▄▄▄▄▄▄▄▄ █
█  ██████    ▄▄▄▄▄▄▄▄ █
█            ▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█                     █
█ ▄▄▄▄▄▄              █
█▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄█

READ THE
WHITEPAPER

>>>






▄          █▄                         ▄
██▄        ███▄                     ▄██
████▄      █████▄                 ▄████
██████▄    ███████▄             ▄██████
████████▄   ▀███████▄         ▄███████▀
██████████▄   ▀███████▄     ▄███████▀
████████████▄   ▀███████▄ ▄███████▀  
██████████████▄   ▀█████████████▀   ▄██
████████████████▄   ▀█████████▀   ▄████
██████████████████▄   ▀█████▀   ▄██████
████████████████████▄   ▀█▀   ▄████████
BLOCKMASON
BUILDING THE FUTURE









FACEBOOK
REDDIT
TWITTER

LINKEDIN
GITHUB
MEDIUM
thenoblebot
Sr. Member
****
Offline Offline

Activity: 364
Merit: 252


View Profile
January 22, 2014, 10:35:51 AM
 #9

ah crap i am already getting billed by amazon for I/O charges and i am only at the 9M difficulty level, this may not be such a bright idea after all, probably just going to stick with regular dedicated vps for 20 bucks a month


EBS
$0.00 for the first 2 million I/O requests under monthly free tier   2,000,000 IOs(used)   $0.00(charge)
$0.10 per 1 million I/O requests   2,393,049 IOs(used)   $0.24(charge)
Total:   $0.24

told ya not everything is "free" in the "Free tier".
e4xit
Sr. Member
****
Offline Offline

Activity: 302
Merit: 250



View Profile
January 22, 2014, 04:22:02 PM
 #10

--snip--
This to me is one of the biggest shortfalls of bitcoin as time goes on, imagine 30 years from now and when bitcoin actually becomes popular, you almost need to buy hardware with all the blocks preloaded otherwise no retail users can ever catch up.
--snip--

Really? Do you not think connection speeds will increase? That seems to me like it might be some serious short-sightedness on you part if I'm honest...

Luckily the architect did not envision that **every** user requires a full copy of the blockchain, only enough to keep it decentralised. Prefessionals, hobbyists and those with the spare resources may keep full purchase ledgers, while "average Joe's" will use light/SPV clients.

/topic derailment.

Also, nice work hosting a bitcoind instance on EC2!  Wink

Not your keys, not your coins.
CoinJoin, always.
moni3z
Hero Member
*****
Offline Offline

Activity: 899
Merit: 1002



View Profile
January 23, 2014, 04:32:59 AM
 #11

You can run this Go implementation of bitcoind with google app engine but who knows what the costs will be. It purposely doesn't include wallet functionaity though https://github.com/conformal/btcd the wallet is here https://github.com/conformal/btcwallet
newguy05 (OP)
Hero Member
*****
Offline Offline

Activity: 1316
Merit: 503


Someone is sitting in the shade today...


View Profile WWW
January 23, 2014, 04:41:19 PM
 #12

You can run this Go implementation of bitcoind with google app engine but who knows what the costs will be. It purposely doesn't include wallet functionaity though https://github.com/conformal/btcd the wallet is here https://github.com/conformal/btcwallet

interesting i will take a look, yeah i looked at google app engine first but it's not a true vps like aws and has limitations in place that prevent bitcoind to run easily without some major tinkering.

In the end i think a regular vps is still my best(economical) option to keep bicoind running 24/7. Something like linode that cost $20 a month and not have to worry about i/o limits etc...

         ▄██████
       ▄████████
     ▄██████████
   ▄█▀     █████
 ▄███      █████
█████      ███████████████████████████
█████      ███████████████████████████
█████      ███████████████████████████
█████                           ████▀
█████                           ██▀
█████                          ▄▀   
██████████████████████████████▀
████████████████████████████▀
L I N K
by BLOCKMASON





..CREATE WEB APIS........
..FROM ANY SMART........
..CONTRACT.........................






█▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀█
█  ██████    ▄▄▄▄▄▄▄▄ █
█  ██████    ▄▄▄▄▄▄▄▄ █
█  ██████    ▄▄▄▄▄▄▄▄ █
█            ▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ █
█                     █
█ ▄▄▄▄▄▄              █
█▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄█

READ THE
WHITEPAPER

>>>






▄          █▄                         ▄
██▄        ███▄                     ▄██
████▄      █████▄                 ▄████
██████▄    ███████▄             ▄██████
████████▄   ▀███████▄         ▄███████▀
██████████▄   ▀███████▄     ▄███████▀
████████████▄   ▀███████▄ ▄███████▀  
██████████████▄   ▀█████████████▀   ▄██
████████████████▄   ▀█████████▀   ▄████
██████████████████▄   ▀█████▀   ▄██████
████████████████████▄   ▀█▀   ▄████████
BLOCKMASON
BUILDING THE FUTURE









FACEBOOK
REDDIT
TWITTER

LINKEDIN
GITHUB
MEDIUM
thenoblebot
Sr. Member
****
Offline Offline

Activity: 364
Merit: 252


View Profile
January 25, 2014, 08:01:39 AM
 #13

You can run this Go implementation of bitcoind with google app engine but who knows what the costs will be. It purposely doesn't include wallet functionaity though https://github.com/conformal/btcd the wallet is here https://github.com/conformal/btcwallet

interesting i will take a look, yeah i looked at google app engine first but it's not a true vps like aws and has limitations in place that prevent bitcoind to run easily without some major tinkering.

In the end i think a regular vps is still my best(economical) option to keep bicoind running 24/7. Something like linode that cost $20 a month and not have to worry about i/o limits etc...

Which vps do you use ? cause other server options like digital ocean, rackspace, etc have bandwidth limitations.
kostagr33k
Full Member
***
Offline Offline

Activity: 309
Merit: 100


View Profile
January 25, 2014, 06:50:58 PM
 #14

Sorry newguy05.

Its about 16gb for the blocks folder alone:

root@i .bitcoin]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvde1       30G   19G   11G  65% /
none            296M     0  296M   0% /dev/shm

[root@i .bitcoin]# du -chs
17G   .
17G   total


Kosta

I actually did this last week. It took days, and I kept blowing away my bitcoind while tweaking things.

I eventually gave in and downloaded the torrent for the recent blocks, but even that took a long time to import, as Disk IO on the micro instances is very limited as well as CPU instances ... Note that the Micro instance is not guaranteed performance as it is their "free" tier"

I eventually went to a m1.large for the two day period to sync all blocks, and now seems to be working fine on the Micro instance once it caught up and I reverted.

kosta

Thanks guys, Kosta do you know the approx disk space requirement after bitcoind is synced up?

I get 30 gb ebs for free, so unmounted the default 8gb and created a new 21 gb mount to use for bitcoind, hope that is enough space.

This to me is one of the biggest shortfalls of bitcoin as time goes on, imagine 30 years from now and when bitcoin actually becomes popular, you almost need to buy hardware with all the blocks preloaded otherwise no retail users can ever catch up.
coinrevo
Member
**
Offline Offline

Activity: 70
Merit: 10


View Profile
January 25, 2014, 07:12:23 PM
 #15

at digitalocean you can run a node for 5$/month (with added swap space). probably cheaper.
thenoblebot
Sr. Member
****
Offline Offline

Activity: 364
Merit: 252


View Profile
January 25, 2014, 08:31:05 PM
 #16

at digitalocean you can run a node for 5$/month (with added swap space). probably cheaper.

would that be enough for a fully functional node with all the normal traffic of a website ? I believe they offer 1 TB of traffic with $5/month package.
lauren2014
Newbie
*
Offline Offline

Activity: 39
Merit: 0


View Profile
January 26, 2014, 12:40:45 AM
 #17

at digitalocean you can run a node for 5$/month (with added swap space). probably cheaper.

i agree, aws is expensive sometimes, and not that much more reliable.
ive had many websites down on aws
moni3z
Hero Member
*****
Offline Offline

Activity: 899
Merit: 1002



View Profile
January 26, 2014, 12:59:15 AM
 #18

Whatever you use be sure to scrub data before you abandon or cancel your VPS or the next user may be able to recover it
https://github.com/fog/fog/issues/2525
Zeal0t
Member
**
Offline Offline

Activity: 78
Merit: 10


To the moon?


View Profile WWW
January 26, 2014, 02:35:16 AM
 #19

I had no idea Amazon has cloud computing services, wow. Thanks for the info!

I vow never to use this space for sleazy referrals, gambling, spam, or to beg for handouts.
U1TRA_L0RD
Full Member
***
Offline Offline

Activity: 126
Merit: 100

CAUTION: Angry Man with Attitude.


View Profile
January 26, 2014, 10:48:47 AM
 #20

I had no idea Amazon has cloud computing services, wow. Thanks for the info!
It sucks... haha, Even google has cloud services which are way better if you ask me.
Pages: [1] 2 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!