bitcoind is too heavy

[rupy]

I tried running bitcoind on AWS EC2 micro instance, it doesen't work. CPU is throttling and memory is swapping.

So is there any lightweight implementation of the bitcoin server?

Otherwise I'll just have to hack bitcoinj to act as a server, simply by adding the JSON-RPC parts, that would work no?

[1715649294]

1715649294

[gweedo]

see the problem isn't that you need a lightweight server, it is that problem that AWS gives you short burst of CPU time and downloading the blockchain and verifying is CPU intensive so anyway you approach it, it probably wouldn't work without upgrading your plan, or getting a VPS.

[rupy]

Well, on my Atom running an older version of bitcoin it takes 4% CPU! So either the latest bitcoin is bloated or aws is completely underpowered. And since I run my own java app server on it, and it's really fast and only uses like 3% CPU under load I'm leaning towards the first option.

What in the verification is so CPU intense?

[BasementMiner!]

The EC2 micro instance is crap for running anything remotely intensive.

You could just use it for a tunnel for your internet browsing.

[Peter Todd]

One trick with EC2 is to take your EBS store for your micro instance, deassociate it, reassociate it as the root partition of a more powerful instance, and use that to get the blockchain verification done in a reasonable amount of time. Then you can move the partition back to the micro-instance, which in my experience is still adequate for the current transaction load.

Equally, do the blockchain verification on your own computer then copy it (the .bitcoin directory) to the EC2 instance.

[rupy]

Well, see that's the problem, while downloading the block chain, load was fine (and if it wasnt, who cares it's only a few hours). But it's only since the whole blockchain was downloaded that the bitcoind started performing badly.

And it performs badly in a weird way, peaking other processes CPU % too... so I'm just guessing there's some kind of throttling going on...

[gweedo]

Well, see that's the problem, while downloading the block chain, load was fine (and if it wasnt, who cares it's only a few hours). But it's only since the whole blockchain was downloaded that the bitcoind started performing badly.

And it performs badly in a weird way, peaking other processes CPU % too... so I'm just guessing there's some kind of throttling going on...

as i explained before you have a short time on the cpu, so that is why your getting weird readings, and it looks like it is bloated. tldr AWS micro is the problem not bitcoind

[MatthewLM]

Running a lightweight client on a server would certainly help in these situations, providing you trust the lightweight validation method.

[gweedo]

Running a lightweight client on a server would certainly help in these situations, providing you trust the lightweight validation method.

no really, cause you do get 612mb of ram, it is the amount of time you get on the CPU is not ever much

[MatthewLM]

Well the lightweight clients only look for specific transactions and done use the CPU for full validation.

[gweedo]

Well the lightweight clients only look for specific transactions and done use the CPU for full validation.

he can try but i doubt it helps since the CPU time you get is so little, so if you get a lot of RPC calls it will still spike and make the webpage or the client calling could either timeout or just not work correctly

[rupy]

What is so expensive, adding a block to the chain and making the "checksum" so you know it's valid? Is the bitcoind actually computing each transaction as they come in and what is it doing, rehashing the whole block chain or what?

[NRF]

I tried running bitcoind on AWS EC2 micro instance, it doesen't work. CPU is throttling and memory is swapping.

Hi, I have a little experience with EC2 so I will try to help you out (I run ~30 instances for various clients).

An EC2 instance will actually complete the block chain, but it takes about 6-7 days and takes up over 2GB's of your 8GB EBS storage.  Once it has done that it will actually perform quite well as an on-line wallet for you.  You could even use it for a few small websites, MySQL db's etc, just don't get too ambitious.

If you are on the "free tear" it will even be free for a year, it hard to beat!

On the other hand, if you don't mind spending a bit of money and getting a bit of value for money look into getting yourself a small "spot instance".  They use unspent capacity on the EC2 network so the costs can vary, but a typical small instance will average out to costing about $0.01 an hour or $7.44 a month (assuming a 744 hour 31 day month), that less than half the on demand micro instance costs ($0.025 an hour).

You do risk that the spot price will go up and your instance will spin down a bit (and then restart when the spot price goes down again), so I put a maximum spot price in for $0.05 or so and I have never had one shut down and I have never had one cost more than $9 in a month.

[2112]

but a typical small instance will average out to costing about $0.01 an hour or $7.44 a month (assuming a 744 hour 31 day month),

Very interesting. Do your sums above include the EBS charges for the I/O operations? 'bitcoind' seems to me fairly I/O-intensive in relation to the fairly small amount of CPU work that it does.

[NRF]

but a typical small instance will average out to costing about $0.01 an hour or $7.44 a month (assuming a 744 hour 31 day month),

Very interesting. Do your sums above include the EBS charges for the I/O operations? 'bitcoind' seems to me fairly I/O-intensive in relation to the fairly small amount of CPU work that it does.

Nope, well spotted As luck would have it though (since there is no real way of estimating IO's) I have real work data for you.  Last month on a "EC2 small" running bitcoind, namecoind, mysqld, httpd and a few scripts I use to collect data from various json sources all the extra costs break down to;-

$0.110 per GB-month of provisioned storage                         8.000 GB-Mo       $0.88
$0.110 per 1 million I/O requests                                 1,119,146 IOs       $0.12
$0.000 per GB - data transfer in per month                         1.580 GB          $0.00
$0.000 per GB - first 1 GB of data transferred out per month                0.690 GB          $0.00
$0.010 per GB - regional data transfer - in/out/between EC2 AZs or using IPs or ELB   0.024 GB          $0.01

So add $1.01, that particular instance came to a total of $8.86 for last month without spinning down.

All up still cheaper than a micro, and FAR more performance.  Your mileage may vary though, with spot instances we are never talking about a fixed cost.

[2112]

$0.110 per GB-month of provisioned storage
8.000 GB-Mo
$0.88
$0.110 per 1 million I/O requests
1,119,146 IOs
$0.12

All up still cheaper than a micro, and FAR more performance.  Your mileage may vary though, with spot instances we are never talking about a fixed cost.

Thanks for the info. So I was wrong about the I/O-intensity of bitcoind (and friends). I tended to avoid the EBS-based instances because in my case the storage charges were higher than processor charges. I only used ephemeral storage because there's no per-I/O-request charges. But then I couldn't use spot instances because the ephemeral storage is, well, ephemeral.

But my workload must have been significantly different than yours.

Thanks again for providing your data point.

[NRF]

Thanks again for providing your data point.

No worries, sorry I could not go back further (I have not been in the "bitcoin" game that long).  I also had similar concerns that bitcoind would thrash the EBS drive but when I actually looked at it the blockchain is only ever downloaded once, and when you download it is scanned for the keys in your wallet so one it is written to storage.

There will be read IO's for previous blocks when you do things that don't involve your key's but that should be very minimal, nothing like running a real read/write db.

But that is just my understanding from briefly browsing the code, I have only been playing with bitcoind for a few months but my testing at amazon seems to back it up.  I am mainly into servers and such.

[rupy]

I tried running bitcoind on AWS EC2 micro instance, it doesen't work. CPU is throttling and memory is swapping.

Hi, I have a little experience with EC2 so I will try to help you out (I run ~30 instances for various clients).

An EC2 instance will actually complete the block chain, but it takes about 6-7 days and takes up over 2GB's of your 8GB EBS storage.  Once it has done that it will actually perform quite well as an on-line wallet for you.  You could even use it for a few small websites, MySQL db's etc, just don't get too ambitious.

If you are on the "free tear" it will even be free for a year, it hard to beat!

On the other hand, if you don't mind spending a bit of money and getting a bit of value for money look into getting yourself a small "spot instance".  They use unspent capacity on the EC2 network so the costs can vary, but a typical small instance will average out to costing about $0.01 an hour or $7.44 a month (assuming a 744 hour 31 day month), that less than half the on demand micro instance costs ($0.025 an hour).

You do risk that the spot price will go up and your instance will spin down a bit (and then restart when the spot price goes down again), so I put a maximum spot price in for $0.05 or so and I have never had one shut down and I have never had one cost more than $9 in a month.

Thanks, so maybe my bitciond is not done downloading the chain... but i thought the data was 2gb... ill check it out!

[NRF]

Login to whatever user is running bitcoind (not root I hope) and type;-

bitcoind getinfo

You should get a response that says something like;-

{
    "version" : 60300,
    "protocolversion" : 60001,
    "walletversion" : 60000,
    "balance" : 64009467.64009467,
    "blocks" : 189791,
    "connections" : 8,
    "proxy" : "",
    "difficulty" : 1866391.30500321,
    "testnet" : false,
    "keypoololdest" : 1341275641,
    "keypoolsize" : 101,
    "paytxfee" : 0.00000000,
    "errors" : ""
}

The line you are after is the one that says ["blocks" : 189791,]  if it is not 189791 or more you have a way to go sorry.  The blockchain download is painful on a micro, that is why I recommended the spot instance.

[Xenland]

What if you uploaded the blockchain to the server? that might work... but then again when you get some btc sent to the server it will crash because you sound like you have low amount of ram anyways

[NRF]

You could upload the blockchain, but it would still have to do allot of the real CPU intensive stuff (scanning the blockchain for the keys in your wallet) but it would be a time saver.

The 612MB for ram that comes in a micro while small is more than enough to process transactions so I would not worry too much about that.

[rupy]

Ok, phew, it works fine when the chain is downloaded... uses alot of ram though, but it's ok.

[P_Shep]

I wish it didn't depend so much on this boost crap.
I've been trying to compile it for the mipsel architecture, but I'm stopped at every turn due it using very specific (that is recent) functions , and not  being written in a terribly portable way. For example, I managed to get boost 1.35 to compile, but even though it says it uses 1.37, 1.35 is not up to date enough :/

[MatthewLM]

For example, I managed to get boost 1.35 to compile, but even though it says it uses 1.37, 1.35 is not up to date enough :/

Very silly since the minor revisions (The right most number) should be compatible with each other. Usually the right-most number reffers to performance improvements and bug fixes and not new features and compatibility breakages.

[P_Shep]

Much mangling to get the dependencies compiled, but I've got:
Boost (1.50) - after a flight
miniupnpc (1.6) - had to manually copy some h files. Neither 1.5 nor 1.7 worked at all
OpenSSL 1.0.1c - Manged to find mipsel table entries for ./Configure and edited them to my needs. Had to disable all ASM, even though mips asm exists
db-4.8.30.NC - which was actually easy!

ifaddrs.h is not in my library, trying what was mentioned here: https://bitcointalk.org/?topic=573.0
I got a va_args not defined in util.h - fixed by adding include stdarg.h (which I'm sure is not a good thing to do)
Now I have:
bitcoinrpc.cpp:116: error: reference to ‘int32_t’ is ambiguous
. . .

No idea how to fix that.

Hope you get your version up and Running Mathew. Maybe as some point I could help. Don't have the time or knowledge now.
This is really not portable at all

[tevirk]

One thing I noticed was that when I ran bitcoind with the data dir on an NFS volume, it performed horrendously - basically maxing out my CPU, even after the blockchain was caught up (which took well over a day).  Moving to local disk made an enormous difference, it now only takes a few percent of CPU.

I don't think the problem is that it does so much disk I/O that the network can't keep up; I suspect it's to do with the disk caching behaviour changing.  Data that is left in the buffer-cache when running against a local filesystem is being flushed in tiny pieces over NFS, or something like that.

From what I know of EC2, I wouldn't expect it to have that sort of problem, but you never know.

The CPU requirements are so light, my Grand Plan is to try and run bitcoind on a Raspberry Pi (when mine arrives).  There are two potential problems; handling the 64-bit integers on a 32-bit core (I assume that a 32-bit Atom is able to use SSE registers, the Pi might suffer more), and secondly, the I/O of an SD card.  But it will be interesting to find out.

[rupy]

Now it runs really fine on EC2 micro, CPU is around 0.3%... It uses 250MB RAM like it was hardcoded to do so. Even after sysctl -w vm.drop_caches=3... I don't know enough about bitcoin to explain the need for 250MB of data, but I was hoping someone here could shed some light on this? What is it keeping cached and why?

[racerguy]

tried restarting bitcoind?  My bitcoin-qt is only using 149m right now.

[NRF]

Just a follow up on my suggestion of running small spot instance's, I don't think I stressed it enough above.  Make sure that your Drives and Instance are set to :-

Delete on termination:   No

I had an unfortunate incident a while back with a medium spot instance that a client setup (did not want to let me into his AWS control panel).  He set the instance to persist (good) but when he set up the drives he left the tick box that says "Delete on Terminate" ticked.

The instance went down for 15 minutes (spot price spiked to $1 or something) and it started up correctly..... with a fresh OS install.

You have been warned!

If you are unsure, fell free to drop me a line and I will try to help.