Bitcoin Forum
December 05, 2016, 02:31:49 AM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: [1] 2 »  All
  Print  
Author Topic: bitcoind is too heavy  (Read 6388 times)
rupy
Hero Member
*****
Offline Offline

Activity: 724



View Profile
July 18, 2012, 10:07:26 PM
 #1

I tried running bitcoind on AWS EC2 micro instance, it doesen't work. CPU is throttling and memory is swapping.

So is there any lightweight implementation of the bitcoin server?

Otherwise I'll just have to hack bitcoinj to act as a server, simply by adding the JSON-RPC parts, that would work no?

BANKBOOK GWT Wallet & no-FIAT Billing API
BTC 14xr5Q1j61A1eA6Mrs5MRhUmYZKboY8iq2 | Vanillacoin FPGA Miner
1480905109
Hero Member
*
Offline Offline

Posts: 1480905109

View Profile Personal Message (Offline)

Ignore
1480905109
Reply with quote  #2

1480905109
Report to moderator
1480905109
Hero Member
*
Offline Offline

Posts: 1480905109

View Profile Personal Message (Offline)

Ignore
1480905109
Reply with quote  #2

1480905109
Report to moderator
1480905109
Hero Member
*
Offline Offline

Posts: 1480905109

View Profile Personal Message (Offline)

Ignore
1480905109
Reply with quote  #2

1480905109
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1480905109
Hero Member
*
Offline Offline

Posts: 1480905109

View Profile Personal Message (Offline)

Ignore
1480905109
Reply with quote  #2

1480905109
Report to moderator
1480905109
Hero Member
*
Offline Offline

Posts: 1480905109

View Profile Personal Message (Offline)

Ignore
1480905109
Reply with quote  #2

1480905109
Report to moderator
1480905109
Hero Member
*
Offline Offline

Posts: 1480905109

View Profile Personal Message (Offline)

Ignore
1480905109
Reply with quote  #2

1480905109
Report to moderator
gweedo
Legendary
*
Offline Offline

Activity: 1246


Java, PHP, HTML/CSS Programmer for Hire!


View Profile WWW
July 18, 2012, 10:39:12 PM
 #2

see the problem isn't that you need a lightweight server, it is that problem that AWS gives you short burst of CPU time and downloading the blockchain and verifying is CPU intensive so anyway you approach it, it probably wouldn't work without upgrading your plan, or getting a VPS.

Want to earn 2500 SATOSHIS per hour? Come Chat and Chill in https://goseemybits.com/lobby
rupy
Hero Member
*****
Offline Offline

Activity: 724



View Profile
July 18, 2012, 10:47:29 PM
 #3

Well, on my Atom running an older version of bitcoin it takes 4% CPU! So either the latest bitcoin is bloated or aws is completely underpowered. And since I run my own java app server on it, and it's really fast and only uses like 3% CPU under load I'm leaning towards the first option.

What in the verification is so CPU intense?

BANKBOOK GWT Wallet & no-FIAT Billing API
BTC 14xr5Q1j61A1eA6Mrs5MRhUmYZKboY8iq2 | Vanillacoin FPGA Miner
BasementMiner!
Member
**
Offline Offline

Activity: 109



View Profile
July 18, 2012, 11:18:33 PM
 #4

The EC2 micro instance is crap for running anything remotely intensive.

You could just use it for a tunnel for your internet browsing. Smiley
Peter Todd
Legendary
*
expert
Offline Offline

Activity: 1064


View Profile
July 18, 2012, 11:26:51 PM
 #5

One trick with EC2 is to take your EBS store for your micro instance, deassociate it, reassociate it as the root partition of a more powerful instance, and use that to get the blockchain verification done in a reasonable amount of time. Then you can move the partition back to the micro-instance, which in my experience is still adequate for the current transaction load.

Equally, do the blockchain verification on your own computer then copy it (the .bitcoin directory) to the EC2 instance.

rupy
Hero Member
*****
Offline Offline

Activity: 724



View Profile
July 18, 2012, 11:44:17 PM
 #6

Well, see that's the problem, while downloading the block chain, load was fine (and if it wasnt, who cares it's only a few hours). But it's only since the whole blockchain was downloaded that the bitcoind started performing badly.

And it performs badly in a weird way, peaking other processes CPU % too... so I'm just guessing there's some kind of throttling going on...

BANKBOOK GWT Wallet & no-FIAT Billing API
BTC 14xr5Q1j61A1eA6Mrs5MRhUmYZKboY8iq2 | Vanillacoin FPGA Miner
gweedo
Legendary
*
Offline Offline

Activity: 1246


Java, PHP, HTML/CSS Programmer for Hire!


View Profile WWW
July 19, 2012, 12:10:06 AM
 #7

Well, see that's the problem, while downloading the block chain, load was fine (and if it wasnt, who cares it's only a few hours). But it's only since the whole blockchain was downloaded that the bitcoind started performing badly.

And it performs badly in a weird way, peaking other processes CPU % too... so I'm just guessing there's some kind of throttling going on...

as i explained before you have a short time on the cpu, so that is why your getting weird readings, and it looks like it is bloated. tldr AWS micro is the problem not bitcoind

Want to earn 2500 SATOSHIS per hour? Come Chat and Chill in https://goseemybits.com/lobby
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
July 19, 2012, 12:11:58 AM
 #8

Running a lightweight client on a server would certainly help in these situations, providing you trust the lightweight validation method.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
gweedo
Legendary
*
Offline Offline

Activity: 1246


Java, PHP, HTML/CSS Programmer for Hire!


View Profile WWW
July 19, 2012, 12:15:47 AM
 #9

Running a lightweight client on a server would certainly help in these situations, providing you trust the lightweight validation method.

no really, cause you do get 612mb of ram, it is the amount of time you get on the CPU is not ever much

Want to earn 2500 SATOSHIS per hour? Come Chat and Chill in https://goseemybits.com/lobby
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
July 19, 2012, 01:08:59 AM
 #10

Well the lightweight clients only look for specific transactions and done use the CPU for full validation.

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
gweedo
Legendary
*
Offline Offline

Activity: 1246


Java, PHP, HTML/CSS Programmer for Hire!


View Profile WWW
July 19, 2012, 01:58:34 AM
 #11

Well the lightweight clients only look for specific transactions and done use the CPU for full validation.

he can try but i doubt it helps since the CPU time you get is so little, so if you get a lot of RPC calls it will still spike and make the webpage or the client calling could either timeout or just not work correctly

Want to earn 2500 SATOSHIS per hour? Come Chat and Chill in https://goseemybits.com/lobby
rupy
Hero Member
*****
Offline Offline

Activity: 724



View Profile
July 19, 2012, 07:43:00 AM
 #12

What is so expensive, adding a block to the chain and making the "checksum" so you know it's valid? Is the bitcoind actually computing each transaction as they come in and what is it doing, rehashing the whole block chain or what?

BANKBOOK GWT Wallet & no-FIAT Billing API
BTC 14xr5Q1j61A1eA6Mrs5MRhUmYZKboY8iq2 | Vanillacoin FPGA Miner
NRF
Member
**
Offline Offline

Activity: 105



View Profile
July 19, 2012, 10:18:22 AM
 #13

I tried running bitcoind on AWS EC2 micro instance, it doesen't work. CPU is throttling and memory is swapping.

Hi, I have a little experience with EC2 so I will try to help you out (I run ~30 instances for various clients).

An EC2 instance will actually complete the block chain, but it takes about 6-7 days and takes up over 2GB's of your 8GB EBS storage.  Once it has done that it will actually perform quite well as an on-line wallet for you.  You could even use it for a few small websites, MySQL db's etc, just don't get too ambitious.

If you are on the "free tear" it will even be free for a year, it hard to beat!

On the other hand, if you don't mind spending a bit of money and getting a bit of value for money look into getting yourself a small "spot instance".  They use unspent capacity on the EC2 network so the costs can vary, but a typical small instance will average out to costing about $0.01 an hour or $7.44 a month (assuming a 744 hour 31 day month), that less than half the on demand micro instance costs ($0.025 an hour).

You do risk that the spot price will go up and your instance will spin down a bit (and then restart when the spot price goes down again), so I put a maximum spot price in for $0.05 or so and I have never had one shut down and I have never had one cost more than $9 in a month.
2112
Legendary
*
Offline Offline

Activity: 1708



View Profile
July 19, 2012, 10:31:07 AM
 #14

but a typical small instance will average out to costing about $0.01 an hour or $7.44 a month (assuming a 744 hour 31 day month),
Very interesting. Do your sums above include the EBS charges for the I/O operations? 'bitcoind' seems to me fairly I/O-intensive in relation to the fairly small amount of CPU work that it does.

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
NRF
Member
**
Offline Offline

Activity: 105



View Profile
July 19, 2012, 10:49:53 AM
 #15

but a typical small instance will average out to costing about $0.01 an hour or $7.44 a month (assuming a 744 hour 31 day month),
Very interesting. Do your sums above include the EBS charges for the I/O operations? 'bitcoind' seems to me fairly I/O-intensive in relation to the fairly small amount of CPU work that it does.

Nope, well spotted Wink As luck would have it though (since there is no real way of estimating IO's) I have real work data for you.  Last month on a "EC2 small" running bitcoind, namecoind, mysqld, httpd and a few scripts I use to collect data from various json sources all the extra costs break down to;-

$0.110 per GB-month of provisioned storage                         8.000 GB-Mo       $0.88
$0.110 per 1 million I/O requests                                 1,119,146 IOs       $0.12
$0.000 per GB - data transfer in per month                         1.580 GB          $0.00
$0.000 per GB - first 1 GB of data transferred out per month                0.690 GB          $0.00
$0.010 per GB - regional data transfer - in/out/between EC2 AZs or using IPs or ELB   0.024 GB          $0.01

So add $1.01, that particular instance came to a total of $8.86 for last month without spinning down.

All up still cheaper than a micro, and FAR more performance.  Your mileage may vary though, with spot instances we are never talking about a fixed cost.
2112
Legendary
*
Offline Offline

Activity: 1708



View Profile
July 19, 2012, 11:18:10 AM
 #16

$0.110 per GB-month of provisioned storage
8.000 GB-Mo
$0.88
$0.110 per 1 million I/O requests
1,119,146 IOs
$0.12

All up still cheaper than a micro, and FAR more performance.  Your mileage may vary though, with spot instances we are never talking about a fixed cost.
Thanks for the info. So I was wrong about the I/O-intensity of bitcoind (and friends). I tended to avoid the EBS-based instances because in my case the storage charges were higher than processor charges. I only used ephemeral storage because there's no per-I/O-request charges. But then I couldn't use spot instances because the ephemeral storage is, well, ephemeral.

But my workload must have been significantly different than yours.

Thanks again for providing your data point.

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
NRF
Member
**
Offline Offline

Activity: 105



View Profile
July 19, 2012, 11:46:18 AM
 #17

Thanks again for providing your data point.

No worries, sorry I could not go back further (I have not been in the "bitcoin" game that long).  I also had similar concerns that bitcoind would thrash the EBS drive but when I actually looked at it the blockchain is only ever downloaded once, and when you download it is scanned for the keys in your wallet so one it is written to storage.

There will be read IO's for previous blocks when you do things that don't involve your key's but that should be very minimal, nothing like running a real read/write db.

But that is just my understanding from briefly browsing the code, I have only been playing with bitcoind for a few months but my testing at amazon seems to back it up.  I am mainly into servers and such.
rupy
Hero Member
*****
Offline Offline

Activity: 724



View Profile
July 19, 2012, 12:00:28 PM
 #18

I tried running bitcoind on AWS EC2 micro instance, it doesen't work. CPU is throttling and memory is swapping.

Hi, I have a little experience with EC2 so I will try to help you out (I run ~30 instances for various clients).

An EC2 instance will actually complete the block chain, but it takes about 6-7 days and takes up over 2GB's of your 8GB EBS storage.  Once it has done that it will actually perform quite well as an on-line wallet for you.  You could even use it for a few small websites, MySQL db's etc, just don't get too ambitious.

If you are on the "free tear" it will even be free for a year, it hard to beat!

On the other hand, if you don't mind spending a bit of money and getting a bit of value for money look into getting yourself a small "spot instance".  They use unspent capacity on the EC2 network so the costs can vary, but a typical small instance will average out to costing about $0.01 an hour or $7.44 a month (assuming a 744 hour 31 day month), that less than half the on demand micro instance costs ($0.025 an hour).

You do risk that the spot price will go up and your instance will spin down a bit (and then restart when the spot price goes down again), so I put a maximum spot price in for $0.05 or so and I have never had one shut down and I have never had one cost more than $9 in a month.

Thanks, so maybe my bitciond is not done downloading the chain... but i thought the data was 2gb... ill check it out!

BANKBOOK GWT Wallet & no-FIAT Billing API
BTC 14xr5Q1j61A1eA6Mrs5MRhUmYZKboY8iq2 | Vanillacoin FPGA Miner
NRF
Member
**
Offline Offline

Activity: 105



View Profile
July 19, 2012, 12:06:46 PM
 #19

Login to whatever user is running bitcoind (not root I hope) and type;-

bitcoind getinfo

You should get a response that says something like;-

{
    "version" : 60300,
    "protocolversion" : 60001,
    "walletversion" : 60000,
    "balance" : 64009467.64009467,
    "blocks" : 189791,
    "connections" : 8,
    "proxy" : "",
    "difficulty" : 1866391.30500321,
    "testnet" : false,
    "keypoololdest" : 1341275641,
    "keypoolsize" : 101,
    "paytxfee" : 0.00000000,
    "errors" : ""
}

The line you are after is the one that says ["blocks" : 189791,]  if it is not 189791 or more you have a way to go sorry.  The blockchain download is painful on a micro, that is why I recommended the spot instance.
Xenland
Legendary
*
Offline Offline

Activity: 980


I'm not just any shaman, I'm a Sha256man


View Profile
July 19, 2012, 12:11:14 PM
 #20

What if you uploaded the blockchain to the server? that might work... but then again when you get some btc sent to the server it will crash because you sound like you have low amount of ram anyways
Pages: [1] 2 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!