bitcoind is too heavy

[rupy]

I tried running bitcoind on AWS EC2 micro instance, it doesen't work. CPU is throttling and memory is swapping.

So is there any lightweight implementation of the bitcoin server?

Otherwise I'll just have to hack bitcoinj to act as a server, simply by adding the JSON-RPC parts, that would work no?

[1714005807]

1714005807

[1714005807]

1714005807

[gweedo]

see the problem isn't that you need a lightweight server, it is that problem that AWS gives you short burst of CPU time and downloading the blockchain and verifying is CPU intensive so anyway you approach it, it probably wouldn't work without upgrading your plan, or getting a VPS.

[rupy]

Well, on my Atom running an older version of bitcoin it takes 4% CPU! So either the latest bitcoin is bloated or aws is completely underpowered. And since I run my own java app server on it, and it's really fast and only uses like 3% CPU under load I'm leaning towards the first option.

What in the verification is so CPU intense?

[BasementMiner!]

The EC2 micro instance is crap for running anything remotely intensive.

You could just use it for a tunnel for your internet browsing.

[Peter Todd]

One trick with EC2 is to take your EBS store for your micro instance, deassociate it, reassociate it as the root partition of a more powerful instance, and use that to get the blockchain verification done in a reasonable amount of time. Then you can move the partition back to the micro-instance, which in my experience is still adequate for the current transaction load.

Equally, do the blockchain verification on your own computer then copy it (the .bitcoin directory) to the EC2 instance.

[rupy]

Well, see that's the problem, while downloading the block chain, load was fine (and if it wasnt, who cares it's only a few hours). But it's only since the whole blockchain was downloaded that the bitcoind started performing badly.

And it performs badly in a weird way, peaking other processes CPU % too... so I'm just guessing there's some kind of throttling going on...

[gweedo]

Well, see that's the problem, while downloading the block chain, load was fine (and if it wasnt, who cares it's only a few hours). But it's only since the whole blockchain was downloaded that the bitcoind started performing badly.

And it performs badly in a weird way, peaking other processes CPU % too... so I'm just guessing there's some kind of throttling going on...

as i explained before you have a short time on the cpu, so that is why your getting weird readings, and it looks like it is bloated. tldr AWS micro is the problem not bitcoind

[MatthewLM]

Running a lightweight client on a server would certainly help in these situations, providing you trust the lightweight validation method.

[gweedo]

Running a lightweight client on a server would certainly help in these situations, providing you trust the lightweight validation method.

no really, cause you do get 612mb of ram, it is the amount of time you get on the CPU is not ever much

[MatthewLM]

Well the lightweight clients only look for specific transactions and done use the CPU for full validation.

[gweedo]

Well the lightweight clients only look for specific transactions and done use the CPU for full validation.

he can try but i doubt it helps since the CPU time you get is so little, so if you get a lot of RPC calls it will still spike and make the webpage or the client calling could either timeout or just not work correctly

[rupy]

What is so expensive, adding a block to the chain and making the "checksum" so you know it's valid? Is the bitcoind actually computing each transaction as they come in and what is it doing, rehashing the whole block chain or what?

[NRF]

I tried running bitcoind on AWS EC2 micro instance, it doesen't work. CPU is throttling and memory is swapping.

Hi, I have a little experience with EC2 so I will try to help you out (I run ~30 instances for various clients).

An EC2 instance will actually complete the block chain, but it takes about 6-7 days and takes up over 2GB's of your 8GB EBS storage.  Once it has done that it will actually perform quite well as an on-line wallet for you.  You could even use it for a few small websites, MySQL db's etc, just don't get too ambitious.

If you are on the "free tear" it will even be free for a year, it hard to beat!

On the other hand, if you don't mind spending a bit of money and getting a bit of value for money look into getting yourself a small "spot instance".  They use unspent capacity on the EC2 network so the costs can vary, but a typical small instance will average out to costing about $0.01 an hour or $7.44 a month (assuming a 744 hour 31 day month), that less than half the on demand micro instance costs ($0.025 an hour).

You do risk that the spot price will go up and your instance will spin down a bit (and then restart when the spot price goes down again), so I put a maximum spot price in for $0.05 or so and I have never had one shut down and I have never had one cost more than $9 in a month.

[2112]

but a typical small instance will average out to costing about $0.01 an hour or $7.44 a month (assuming a 744 hour 31 day month),

Very interesting. Do your sums above include the EBS charges for the I/O operations? 'bitcoind' seems to me fairly I/O-intensive in relation to the fairly small amount of CPU work that it does.

[NRF]

but a typical small instance will average out to costing about $0.01 an hour or $7.44 a month (assuming a 744 hour 31 day month),

Very interesting. Do your sums above include the EBS charges for the I/O operations? 'bitcoind' seems to me fairly I/O-intensive in relation to the fairly small amount of CPU work that it does.

Nope, well spotted As luck would have it though (since there is no real way of estimating IO's) I have real work data for you.  Last month on a "EC2 small" running bitcoind, namecoind, mysqld, httpd and a few scripts I use to collect data from various json sources all the extra costs break down to;-

$0.110 per GB-month of provisioned storage                         8.000 GB-Mo       $0.88
$0.110 per 1 million I/O requests                                 1,119,146 IOs       $0.12
$0.000 per GB - data transfer in per month                         1.580 GB          $0.00
$0.000 per GB - first 1 GB of data transferred out per month                0.690 GB          $0.00
$0.010 per GB - regional data transfer - in/out/between EC2 AZs or using IPs or ELB   0.024 GB          $0.01

So add $1.01, that particular instance came to a total of $8.86 for last month without spinning down.

All up still cheaper than a micro, and FAR more performance.  Your mileage may vary though, with spot instances we are never talking about a fixed cost.

[2112]

$0.110 per GB-month of provisioned storage
8.000 GB-Mo
$0.88
$0.110 per 1 million I/O requests
1,119,146 IOs
$0.12

All up still cheaper than a micro, and FAR more performance.  Your mileage may vary though, with spot instances we are never talking about a fixed cost.

Thanks for the info. So I was wrong about the I/O-intensity of bitcoind (and friends). I tended to avoid the EBS-based instances because in my case the storage charges were higher than processor charges. I only used ephemeral storage because there's no per-I/O-request charges. But then I couldn't use spot instances because the ephemeral storage is, well, ephemeral.

But my workload must have been significantly different than yours.

Thanks again for providing your data point.

[NRF]

Thanks again for providing your data point.

No worries, sorry I could not go back further (I have not been in the "bitcoin" game that long).  I also had similar concerns that bitcoind would thrash the EBS drive but when I actually looked at it the blockchain is only ever downloaded once, and when you download it is scanned for the keys in your wallet so one it is written to storage.

There will be read IO's for previous blocks when you do things that don't involve your key's but that should be very minimal, nothing like running a real read/write db.

But that is just my understanding from briefly browsing the code, I have only been playing with bitcoind for a few months but my testing at amazon seems to back it up.  I am mainly into servers and such.

[rupy]

I tried running bitcoind on AWS EC2 micro instance, it doesen't work. CPU is throttling and memory is swapping.

Hi, I have a little experience with EC2 so I will try to help you out (I run ~30 instances for various clients).

An EC2 instance will actually complete the block chain, but it takes about 6-7 days and takes up over 2GB's of your 8GB EBS storage.  Once it has done that it will actually perform quite well as an on-line wallet for you.  You could even use it for a few small websites, MySQL db's etc, just don't get too ambitious.

If you are on the "free tear" it will even be free for a year, it hard to beat!

On the other hand, if you don't mind spending a bit of money and getting a bit of value for money look into getting yourself a small "spot instance".  They use unspent capacity on the EC2 network so the costs can vary, but a typical small instance will average out to costing about $0.01 an hour or $7.44 a month (assuming a 744 hour 31 day month), that less than half the on demand micro instance costs ($0.025 an hour).

You do risk that the spot price will go up and your instance will spin down a bit (and then restart when the spot price goes down again), so I put a maximum spot price in for $0.05 or so and I have never had one shut down and I have never had one cost more than $9 in a month.

Thanks, so maybe my bitciond is not done downloading the chain... but i thought the data was 2gb... ill check it out!

[NRF]

Login to whatever user is running bitcoind (not root I hope) and type;-

bitcoind getinfo

You should get a response that says something like;-

{
    "version" : 60300,
    "protocolversion" : 60001,
    "walletversion" : 60000,
    "balance" : 64009467.64009467,
    "blocks" : 189791,
    "connections" : 8,
    "proxy" : "",
    "difficulty" : 1866391.30500321,
    "testnet" : false,
    "keypoololdest" : 1341275641,
    "keypoolsize" : 101,
    "paytxfee" : 0.00000000,
    "errors" : ""
}

The line you are after is the one that says ["blocks" : 189791,]  if it is not 189791 or more you have a way to go sorry.  The blockchain download is painful on a micro, that is why I recommended the spot instance.

[Xenland]

What if you uploaded the blockchain to the server? that might work... but then again when you get some btc sent to the server it will crash because you sound like you have low amount of ram anyways