Bitcoin Forum

I suspect someone started downloading the full blockchain off my full node.  Unfortunately, the effect was that my computer came to a screeching halt: moving the mouse and the pressing keys on my keyboard had absolutely no effect for several minutes.  When I regained use of my computer, I looked in the Task Manager and bitcoin-qt was using the most RAM, and a fair amount of processing power was being used as well.  I turned off bitcoin-qt and the problem went away; anecdotally, I'd say that's reason to believe that running the bitcoin-qt client was causing the problem.

So a little bit about my specs, both internet and computer: I'm using an internet connection of 25 Mbps down / 1.5 Mbps up - reasonable download speed (unfortunately I don't live in an area where gigabit internet is available), but not much on the upload front.  My computer has 5 GB of RAM, 250 GB of hard drive space, and a 3 Ghz processor speed.  The only part of my computer itself that I feel is underpowered is the RAM, I plan on eventually upgrading to 20 GB RAM, but haven't gotten around to it yet.

So here's my question: is there something about the way the bitcoin client is built that causes my computer to freeze up when people start to download the blockchain from me?  Or is this just a case of me not having enough upload bandwidth to handle someone downloading a full node from me?  I'd love to be able to support the network by running the bitcoin-qt client regularly, but this is starting to happen semi-regularly as bitcoin becomes more popular and I don't feel like my computer is THAT underpowered.

As you noticed yourself about your specs, I would say the problem is the 5 GB of RAM. You don't mention which OS you are using: a 32 bit version of the OS would typically limit user-space RAM to about 3 GB - that's a bit short for modern apps, especially while running a full node.

What you described (in bold in my quote) is probably caused by what I call "swapping to death". It's the symptom of the OS running out of memory, and because of this, it is forced to swap memory contents back and forth from/to the hard drive, which is about 3 orders of magnitude slower than RAM. This may cause your machine to become completely unresponsive for several minutes.

I always wondered why OS designers wouldn't implement a system warning message whenever the system needs to use the swap file to such an extent that the user experience becomes heavily degraded. Most users don't understand what's happening in the background, and what they can do about it (i.e. add more RAM)

You can do a few things to mitigate coin applications that cause problems:

1. Lower the CPU affinity and process priority. You can do this with task manager right-clicks in Windows if you are going to leave Bitcoin running, or you can make a .cmd file (batch file) to run with the start priority options. CPU is required to validate P2P transaction signatures, and someone could be targeting you specifically with attack transaction broadcasts.

2. If your rotating hard disk is fragmented, that means the disk heads will have to hunt all over the platter to find data, limiting the possible input-output operations per second. You can use defrag programs that completely move and defrag every file, but the new blocks every 10 minutes will again be fragmented. It would be best to give the Bitcoin datadir its own ~40gb disk partition to limit hunting. This problem is solved with a SSD - instead of a limit of 150 IOPS from moving heads, it's more like 40000 IOPS.

3. Monitor RAM usage. I'm not so sure this is an issue. I have Bitcoin Core and Namecoin both running. They use under 400MB each on a 8GB system, and don't impact anything else noticeably.

Ah, good point, I didn't mention the OS.  I'm using Windows 7 64 bit, so it should totally support the 20 GB of RAM that I'm planning on upgrading to eventually.  Thanks for the technical explanation of what's going on, so it sounds like the RAM upgrade will probably fix the issue then.  Unfortunately, the motherboard on my computer only has two RAM slots, so I'm going to have to get a 16 GB chip to upgrade to the 20 GB of RAM that I'm aiming for.  (The one on my Amazon wishlist is going for about $171 right now.  It was actually significantly cheaper (like $20 - 30) on bitcoinstore.com last I checked though, so I'll probably buy it there.)

You're right, it would be smart for OS designers to warn users about this kind of swap file need.  I just hope that this reliance on swap files that's gotten a bit more common lately isn't going to degrade the lifespan of my hard drive.

Hrm.  I was noticing high CPU usage, so it's possible that was an issue.  I ran disk defrag a month or two ago and it only reported 1% of fragmentation, which doesn't seem like that much.  That said, it didn't tell me which 1% of the hard drive was fragmented.  If that 2 GB or so was right smack in the middle of the blockchain, that'd cause some significant problems I'd imagine.

Granted, I do have an old fashioned hard drive, not an SSD.  I'm seriously thinking of upgrading and am wondering whether bumping up my RAM or adding in an SSD would be better off performance wise.  Any thoughts there?

A solid state drive as an OS drive is the single biggest improvement you will ever experience. I boot Ubuntu 14 in under 30 seconds on an otherwise seven-year-old computer. You do have to plan your data allocation, such as C:SSD 240GB and then D:HDD for multimedia. If your current hard drive is not just 250GB of free space, but is actually a 250GB drive, it is also 10-year-old tech (but with the benefit that you can just duplicate it to a new SSD).

I would only recommend Tier-1 SSD, such as Intel 530 or Samsung 840 EVO 250GB. You can get the latter at Best Buy or Newegg for $130.

If you seldom run several RAM-demanding apps at the same time, I would say you'll be better off with an SSD. Otherwise, considering your current 5 GB of RAM, I would go for the RAM extension. Of course, having both would be the best ;-)

If you go for the SSD option, make sure to have your system files (OS) and swap file on it, as it's what makes most of the difference. In other words, keep your hard drive for large data files. Where you will store the block chain depends on your own priorities and the capacity of your SSD; of course, it would be much faster on the SSD (regarding what deepceleron said about fragmentation, timings etc), but the blockchain is getting huge nowadays.

I wonder why there isn't more effort put in the direction of implementing QoS mechanisms (like throttling blockchain downloads) in the protocol. After all, the network as a whole does not become more efficient when single nodes hog down other nodes due to misconfigured blockchain downloading.

There is such a thing as a hybrid SSD HDD. I don't really know anything about it (so you should look into it yourself), but it seems to allow you to boot quickly from SSD while performing most of your everyday tasks on the cheaper hard drive.

Downloading the blockchain is easy. A much bigger resource hog is verifying the blockchain.

For a while now I've happily ran a full node using an old pentium dual core with 4GB ram... around 3.4Ghz with debian linux.  In fact it does much more than just bitcoin and the processor is hardly touched.

As for bitcoin I allow up to 50 connections inbound and I limited the outbound bandwidth to half the upstream speed delivered to my house.

Its worked flawlessly thus far.

1) set 10 or 5 Mb on cache setting (instead of 100Mb)
2) set 12 connexion (not less than 8 instead of 20).

by the way, download the entery blockchain take insane CPU job at the end (with no download).

example : 22h of download, 4h of 100% CPU job at the end (just after that, bitcoin core is OK and have finish).

Something must be wrong with your set up.

I run bitcoind on 1 GB RAM and it's a bit strained at times but works. RAM should be your limiting factor.

5GB is a weird memory amount.  I assume it is 1GB DIMM and 4GB DIMM.  I would either add a 4GB DIMM (8GB total) or remove both and add a pair of 8GB DIMMS (16GB total).  Save the rest of your funds towards an SSD. 

You're right, it's a combination of a 1 GB DIMM and a 4 GB DIMM. The computer originally had 2 GB of RAM but I had the store I bought it from swap out one of the 1 GB for a 4 GB since 2 GB of RAM just doesn't cut it in Windows.  So do you think the 20 GB of RAM I was aiming to upgrade to is overkill?

Sounds like you have a broken SATA driver/controller. Try starting up some disk benchmarks and see if your computer becomes unusable when there is the slightest activity.

Huh, you mean I could fix this with a driver update?  That would sure be convenient.  I'll look into it and report back, thanks for the tip.

Anyone know the suggested specs to run a full node? I want to run one of my own, without having the PC freeze up.

Well, I have a computer with 16GBs of RAM and upload/download speed of 300mbps.

I'm running Bitcoin-qt, Namecoin-qt, Feathercoin-qt, Dogecoin-qt, and Digibyte-qt, all of which are full nodes.

I don't get lag.

Upgrade to 8GB. It should be more than enough. But first check your hard disk. Run chkdsk/scan disk or whatever you have on windows. Do the long test - the one that takes hours :)

edit: it's chkdsk /r

How big is the Blockchain at the moment?

For bitcoin? Something over 20GBs. I have 4TBs, so I never check.

Yes, 20+ GB. Devs are trying to reduce the size to around 20 GB or below. It will be better, IMO.
Kindly,
       MZ

To follow this, how are you going to make 20GB? Historically (not sure if it's the case so much now) memory liked running in equal pairs (or quads), for example 2x2GB for 4 GB total, 2x8GB for 16GB or 4x8GB for 32GB. If you are planning on trying 2x8GB + 1x4GB then I'm not sure this is optimal for the system, although sure someone will chip I'm to correct me if this is no longer true these days....

I would try and get a cheap 16GB kit (2x8GB), use just that and then definitely acquire an SSD which will give you the biggest performance boost.

Well, it definitely wouldn't be a matched pair to get to 20 GB.  Basically, I have two RAM chips now, a 4 GB one and a 1 GB chip, making 5 GB.  I'd replace the 1 GB chip with a 16 GB chip, making it a total of 20 GB of RAM. 

To follow up on  Abdussamad's recommendation, I ran chkdsk /r.  It didn't seem to report any errors.  So at this point my guess is that it's either a not enough RAM issue, not a fast enough hard drive issue, or a not enough upload bandwidth issue.  The first two of those are fixable, the last, well...all I could do to fix that would be to ratelimit bitcoin-qt.   

I actually don't mind the 20GBs. The only problem I have is that the blockchain takes up a huge amount of space (relatively) on my Macbook Air (2012). I solved that problem by deleting apps. I actually discovered I had an application (iMovie) taking up 42 freaking gigabytes of space.

I don't know what's wrong with your computer. Mine runs fine with bitcoin, litecoin, tagcoin, guncoin, and a bunch of other coins. Specs are 2 GB Ram DDR2 666 (slow compared to today's standard 1333/1666), and an old intel dual core T4400.

Of course, I'm still on Win XP 32 bit, but right now, most of my coins have 30+ connections each.

I can do office applications just fine, but drawing and paint stuff do take a noticeable hit. (They've always had that problem though, even without bitcoin.)

4 year old laptop.

Try doing the same software configuration with the same specs... Something is wrong with your computer.

+1

I recently had to sync a new wallet from scratch, so I used a forced connection to a single up-to-date node on the same LAN. At first it was flying - several tens of megs transferred per second - but gradually the sync got slower and slower (more transactions per block?). Towards the end it was spending a second or two to verify each block.

You should run your memory dual-channel (https://en.wikipedia.org/wiki/Multi-channel_memory_architecture). Basically, if you get two of the same type of chips, they can run twice as fast. (consult your motherboard manual for details on if it supports it, where to put the chips, etc.)
E.g. I'd recommend that you get two 8GB chips (16GB total) instead of 16+4GB as you describe.

Yeah...I've been mulling over what Dabs said for a while here and I think he's right.  I just looked up the warning signs of a hard drive failure and my computer seems to be showing three out of the eight signs listed.  Think it's time to buy a new hard drive.

Someone downloading the blockchain shouldn't make your computer unresponsive.  If you don't have enough upstream bandwidth, it could make your internet really slow.

If you really want to run one, just use something like this:

https://vpsdime.com/cart.php?a=add&pid=41

6GB RAM
30GB SSD space
4 vCPU cores
2TB traffic
10Gbps uplink
1x IPv4
OpenVZ/Custom
$7/month

... wouldn't do the annual billing tho, since 30GB won't be enough for a year.

I downloaded the whole thing a couple of days ago from scratch. It's currently 27GB.

You should go to the help menu of Bitcoin Core and pull down the Debug menu.  With this you can look at a graph of Internet traffic. This will show what is going on with network traffic.  You can also use the Windows task manager to monitor computer system resources being eaten up. With my X64 Windows 7 system running with 12 GB of RAM, my problem wasn't the computer system, it was my network bandwidth, specifically upload bandwidth thanks to a DSL connection with pathetic (1 Mbps) upload speed.  I didn't have problems with system performance when running a full node, but I did get my DSL uplink bandwidth completely saturated, making web surfing impossible from any computer on my LAN.

The problem is with the Bitcoin software.  It should come with easily configured user limits on upload and download network bandwidth.  This has been a capability of peer to peer file sharing systems, such as bittorrent clients, for many years.  However, on the Windows 7 platform there is a way around this limitation that should work just fine, albeit with a slight amount of inconvenience.

 My cure was to configure a Group Policy using the Windows Group Policy editor.  I limited the upload bandwidth of bitcoin to about half my available uplink bandwidth.  Problem solved. In the group policy editor under Windows Settings, Policy-based QoS I set two policies for TCP port 8333, bitcoin incoming and bitcoing outgoing. Since doing this, bitcoin no longer steals my computer or network bandwidth.

Is there a way to install the bitcoind package in Ubuntu to be able to run commands but without having to download the blockchain?

For example, I'd like to use getblock, getrawtransaction, etc but not having to get the entire blockchain....

Don't think that would be possible, since RPC commands refer to a local database... which is the blockchain.

If it's time rather than HD space you're trying to save, you can use the existing client to seed the new one:

./bitcoind -connect=x.x.x.x -listen=0

(where x.x.x.x is the IP of your existing client)

Be aware that it can take a bit of time to verify every block, as I mentioned earlier in this thread.

You can also try copying the block database - it should be binary compatible across platforms. Basically, copy the contents of directories blocks/ and chainstate/ from your existing data dir, to the data dir on your linux box, then start bitcoind.

Another option, if you really don't want the duplicate blockchain, may be to use the API of a block explorer?

Unless there's some serious reason that 20 GB of RAM is essential, then it's way overkill for a home user. Whatever gave you the impression that you'd need so much RAM?

Are you still having problems or was it just the one time?

I remember a time, not all that long ago, when the HDD light performed exactly the function very well...

Not that im aware of, there are multiple websites that have all that information available freely without having to download anything.

You could also just switch to paypal and avoid all the complexity of that fussy Bitcoin stuff. Since you're apparently happy to trust oft-anonymous oft-judgement proof parties, paypal would likely be a big security upgrade too.

Man I'd love to see a new specialized computer product which sole purpose is to run a BTC node.

Trezor has hardware for wallets, mining companies have hardware to mine, there are hardware units to make paper wallets...

why not a piece of hardware designed specifically to run a full node?

The 30GB is not going to last long, but the 6GB is overkill. It runs strained on 1GB, but with 2 it never swaps, so you don't need 6.

Because Bitcoin needs a node to be as simple and ubiquitous as possible and be as decentralized as possible without depending on specific hardware. What would be the incentives to run a dedicated hardware node other than being a Bitcoin enthusiast?

it doesn't seem right for a core developer to recommend paypal over bitcoin

It doesn't seem right that you're posting in the technical subforum without understanding what I was saying there.

Another option is getting a decent refurbished desktop box with a reasonably modern processor and acceptable amount of RAM (~4GB should be fine). With that you can toss some linux or BSD on it and run bitcoind with the full transaction index and set yourself up a private electrum server.

The box will cost ~$150 to $200. For a bit more you can get a massive hard drive or an SSD to  stick in it.

What this option gives you is a way to move the heavy lifting of handling the blockchain off of the box you interact with everyday while giving you a similar privacy benefit to running a full bitcoin node on it... As long as you set up electrum on you client machines to exclusively use your private bitcoin node/electrum server. You also won't have to worry about constantly worrying about overpaying for or exhausting your storage space as you would have to on a VPS.

It's already available in the form of a Raspberry Pi or an Odroid.

you're right. I mean no disrespect, I was just more curious as why because I didn't understand. I just assumed Paypal to be competition to bitcoin.

i have a couple, they have 64GB of RAM, and really fast CPUs

My computer kept sounding worse and worse and from a little bit of research, it sounded like it was a dying hard drive.  So I bought a new hard drive and upgraded it today, hopefully that'll fix the problem.

Two reasons for wanting that much RAM: one is my computer started slowing down lately (I have a LOT of bookmarks) and with that much RAM I'd never have to worry about not having enough RAM for anything.  The other reason is probably obsolete now that I have a terabyte hard drive, which is basically wanting to be able to possibly run the old version of Armory that runs off RAM instead of hard drive space.  You need a LOT of RAM to do that and that requirement is only going to get bigger over time.  I was a little worried about my hard drive filling up with my old 200 GB hard drive being a little over half full, but now I'm only using about 10% of the space of my new drive so that's not really a concern any more.

It's a splurge, I admit it, but the other advantage is that 16 GB is likely to be a lot of RAM for quite a while, so even when this computer ends up dying, I could probably swap that chip into the next one to give it an instant boost in RAM.  Although I think several people in this thread have pretty much convinced me that two 8 GB chips running in parallel might well be faster than the 16 GB / 4 GB configuration that I had planned.

Is the RAM DDR2 or DDR3? I ask because I recently upgraded from Windows XP to 8.1 64 bit, and I had to replace nearly every major part of my computer. The innards are only about 3 years old and worked just fine, but were purchased around the time that DDR2 and socket 775 were almost obsolete. I saved a bit then, but it means that this time around I had to buy a new mainboard, CPU, and RAM... otherwise I'd be running a modern version of Windows with only 4GB of RAM.

tl;dr adding to an obsolete system may be false economy in the long run.

I have nodes for Bitcoin, Infinitecoin, Dogecoin, Latium, and P2Pool going in the background, consuming less than 6 GBs of RAM out of my 16 GBs.

Memory usage seems to (mostly) depend on # of connections.

I have one bitcoind at 464MB with 80 connections, one at 503MB with 79 connections, one at 1665MB with 270 connections, and one at 1860MB with 349 connections. 

Before restarting on the two @ 349 and 270 connections, they had 476 & 423 respectively, w/ 2323MB and 1935MB memory used.

What versions are your nodes based on? For current versions there should be relatively little dependency on the number of connections.

Honestly, I don't really know.  I ran Belarc advisor and it just said DIMM0 and DIMM1, don't really feel like cracking the case to check for sure.  So, whatever was standard in 2011 when I bought this computer would be my assumption.

In any case...I think I'm starting to agree with your tl;dr.  It took a few days for the full effect to kick in, but my comp is running so much better now than it was with the old hard drive.  So I think I really did just have a failing drive.  Which is nice because a) I caught it before I lost any data and b) new hard drives are a lot cheaper than the RAM upgrade I had planned.  Now that I have a 1 TB hard drive and it's working better, the new RAM probably isn't necessary.  I was a little worried with the old drive that the blockchain, especially with O(2N) since I use Armory, would eventually fill up my old 200 GB hard drive.  But my new hard drive is big enough that my computer will likely become fully obsolete and need replacing before that happens.

hmm, I'll try a more current version.  It was based on 0.9.1..

I also noticed a lag when bitcoind runs on my desktop. It's mostly IO related, and it is
noticeable when opening new programs etc. Sometimes the whole computer freezes for a while.
It's an older i5, 4 cores@3.6GHz, with 8GB ddr3 ram. Bitcoind eats about 1GB of memory,
which doesn't seem to be that much. I am on linux, btw.

But of course, this is all just a very subjective impression - no idea how to measure it.
I have the impression bitcoind is quite "bursty" in its disk access, regularly going over
60% in iotop. I've been noticing (and complaining about) this since a while, but maybe
I should sit down and try to quantify this one of these days. No idea though how to
do that, suggestions appreciated.

Do your motherboard support ddr3? It may work but just check it whether it works completely. My graphics card is ddr3 and my ram is ddr2 and there is some problem due to that. So just check it.

  ~~MZ~~

Isn't it possible that devs "compact" more than 1 year old transations into a simple text file.
It would be like
on 1.1.2013
12trbdk.....  has x.xxx BTC
1bvdsh...... has x.xxx BTC
etc

confirming these transactions has no interest so this would make things much quicker and reduce CPU and bandwidth usage.

Are the bursts whenever a block comes along?

I have good success running bitcoind with some niceness.

Start the client with the option "-bind=127.0.0.1". You can connect to another people for download but another people cannot connect and download from you.



FaSan

Which is of course exactly what you want when setting up a node for supporting the bitcoin network...  ;)

-listen=0 will also disable inbound connections. Probably makes a bit more sense. :)

If everyone did that the entire network would shudder to a halt. This is not good advice.

I'd really rather use NetBalancer to rate limit bitcoin-qt if I were going to do that than kill connections altogether.  Which, imho, really ought to be built into the bitcoin-qt client, but that's another matter entirely.  I'll do that if it becomes necessary, but so far, most of the problem - I think the hard drive upgrade / replacement mostly fixed the problem. 

Exactly.

Depends on finally usage. If you up a node for support the network maybe you don't need this, but if you need to give a BTC services and want to hide the client this is a good way.



FaSan

It's complicated. TL;DR the consensus is that first we need to fix the inefficiencies in how Bitcoin Core transmits data, as that could alleviate many of the problems. Here's the 3 year long debate on the subject: https://github.com/bitcoin/bitcoin/issues/273

Yeah, I found that discussion a while ago, am a bit frustrated that nothing seems to be done about it.  Just throttled bitcoin-qt's upload bandwidth to 300k with Netbalancer, about 1/3 of my maximum upload speed and it seems to be helping my computer run a lot faster than it used to be, so it looks like I won't be needing that RAM upgrade after all.  Netbalancer is free - in case any other Windows users are having this problem, though the free version will only let you shape the bandwidth of three processes, that's plenty for me since bitcoin-qt is the only one giving me grief.

As a similar stopgap measure, Bitcoin Core provides a Quality of Service bash script for Linux users that will throttle bandwidth usage. https://github.com/bitcoin/bitcoin/tree/master/contrib/qos

Or use trickle

Lots of things are being done about it, but they're not done yet. Wheres your contribution?

Not much, I admit, so maybe I'll just update the wiki to let people know how to work around it (in Windows, anyway).  I'd happily fix it myself, but after reading the whole discussion on github about it, I'm pretty sure that I don't have the technical know how to do it.  And I'd rather just let it continue to be an issue than submit a patch that just makes things worse.

So thanks for your work on it, I appreciate it.

If you're using the default directory, delete all the stuff in your /users/<name>/appdata/roaming/bitcoin folder make a new bitcoin.conf file. After blockchain is done downloading, change maxconnections, also change listen to 1 (if you want), and maybe you need to change keypool to make a majority of addresses you probably wont need to have your wallet bloating

This doesn't make sense. Why would you delete the local blockchain, then download exactly the same thing?

Agreed, this is terrible advice.  The blockchain is 25.5 GB right now, there's no way I'm going to delete & redownload it just for the hell of it.

In any case, I figured out a solution to the problem a little while ago: I ratelimited bitcoin-qt's upload speed to about 1/3 of my bandwidth several days ago in Netbalancer, and then put Netbalancer in my Startup folder so that it'll stay that way.  Seems to have fixed the problem.

I've updated the wiki (https://en.bitcoin.it/wiki/Bitcoin-Qt) to mention this issue and explain how I successfully worked around it in Windows.  If anyone has any recommendations for other places to put this information on the wiki that people will be likely to see, please let me know and I'll be happy to add it in there too.