Why does catching up with the blockchain hammer the disk so much?

[stakhanov]

I'm using bitcoin-qt on Arch linux, and catching up from a long time ago. I've noticed that the hard disk is working very hard all the time. It seems pretty fishy to me, since I'm downloading from a pretty slow DSL line, the disk should be working far less than that. What is going on?

[Revalin]

Database transactions require writing a journal, syncing, writing the data, syncing again, and then deleting the journal.  That's a much harder disk hit than linear writes.

Try using the just-released 0.6.0.  The settings have been changed for better disk performance.

[stakhanov]

Thanks for your answer. I do have version 0.6x

I wonder if the database commits are simply too close to each other? The blockchain is public information and can always be redownloaded in the even of a crash, there is really no reason to write to the disk several times per second to secure it.

[Pieter Wuille]

Thanks for your answer. I do have version 0.6x

I wonder if the database commits are simply too close to each other? The blockchain is public information and can always be redownloaded in the even of a crash, there is really no reason to write to the disk several times per second to secure it.

The hard part is not downloading or storing the block chain. It's building, maintaining and guaranteeing the on-disk consistency of the block and transaction index that is necessary for validation.

[Revalin]

They used to be much too close; hence the adjustment.

Obviously there's still considerable room for improvement.  

Edit: And why would you ever quit Bitcoin?  You smell like a traitor. 

[stakhanov]

The hard part is not downloading or storing the block chain. It's building, maintaining and guaranteeing the on-disk consistency of the block and transaction index that is necessary for validation.

Well, I suppose you've given it a lot of thought already. But I still have a hard time believing downloading the block chain and creating an index requires all that disk hammering. On my computer, the .bitcoin directory is 1.6GB big. It's big, but not that big. It even fits in the memory of most computers! How can building it require hammering the disk for hours?

[Revalin]

I was just kidding, but everything you say is correct.  The startup and shutdown times are very high.  SOME disk hit is expected, but it's still quite excessive.

[Pieter Wuille]

The hard part is not downloading or storing the block chain. It's building, maintaining and guaranteeing the on-disk consistency of the block and transaction index that is necessary for validation.

Well, I suppose you've given it a lot of thought already. But I still have a hard time believing downloading the block chain and creating an index requires all that disk hammering. On my computer, the .bitcoin directory is 1.6GB big. It's big, but not that big. It even fits in the memory of most computers! How can building it require hammering the disk for hours?

It takes about half an hour for me, when loading from a local file (a patch that will probably be in 0.7) instead of from the network.

If you have a lot of RAM, you can run it in a RAM drive (if your OS supports that) and watch it fly. Of course, you'll miss the one hard part: not losing or corrupting the on-disk index when your computer crashes.

[Revalin]

The fact that it's possible to download the whole thing onto a RAM disk, manually copy it to disk, and sync with orders of magnitude less of an IO hit suggests that the DB is doing something pedagogically.

[Pieter Wuille]

The fact that it's possible to download the whole thing onto a RAM disk, manually copy it to disk, and sync with orders of magnitude less of an IO hit suggests that the DB is doing something pedagogically.

Patches very welcome

[stakhanov]

The fact that it's possible to download the whole thing onto a RAM disk, manually copy it to disk, and sync with orders of magnitude less of an IO hit suggests that the DB is doing something pedagogically.

Patches very welcome

Just to be clear, this is a perfectly acceptable answer to me! I just wanted to make sure I wasn't missing something obvious.

[deepceleron]

The hard part is not downloading or storing the block chain. It's building, maintaining and guaranteeing the on-disk consistency of the block and transaction index that is necessary for validation.

Well, I suppose you've given it a lot of thought already. But I still have a hard time believing downloading the block chain and creating an index requires all that disk hammering. On my computer, the .bitcoin directory is 1.6GB big. It's big, but not that big. It even fits in the memory of most computers! How can building it require hammering the disk for hours?

You'd believe how much it writes to the disk if you make a new ~8gb disk image, defrag & push all the data to front, and then simply install and run Bitcoin. The blockchain will be fragmented all over the disk from all the writes and erases to the database.

[fghj]

Sorry for necroing this thread, posting something that is obvious to most people (was for me in hindsight) and that will be moot when v0.8 will be released but I didn't found it written anywhere. If you don't have enough RAM for storing entire blockchain in RAMdisk you can put blkindex.dat into RAM and do a symlink to it from datadir. This will also speed up syncing and will require only about 1.8GB of ram instead of 6+GB.
_{Also remember to copy it back to hard drive when finished}

[paraipan]

Sorry for necroing this thread, posting something that is obvious to most people (was for me in hindsight) and that will be moot when v0.8 will be released but I didn't found it written anywhere. If you don't have enough RAM for storing entire blockchain in RAMdisk you can put blkindex.dat into RAM and do a symlink to it from datadir. This will also speed up syncing and will require only about 1.8GB of ram instead of 6+GB.
_{Also remember to copy it back to hard drive when finished}

Better, buy a 64gb SSD drive (around 50€) and move bitcoin datadir onto it. You should see the difference right away