Bitcoin Forum
December 04, 2016, 02:06:05 AM *
News: Latest stable version of Bitcoin Core: 0.13.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: Why does catching up with the blockchain hammer the disk so much?  (Read 2611 times)
stakhanov
Full Member
***
Offline Offline

Activity: 175


View Profile
March 31, 2012, 03:05:01 PM
 #1

I'm using bitcoin-qt on Arch linux, and catching up from a long time ago. I've noticed that the hard disk is working very hard all the time. It seems pretty fishy to me, since I'm downloading from a pretty slow DSL line, the disk should be working far less than that. What is going on?
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1480817165
Hero Member
*
Offline Offline

Posts: 1480817165

View Profile Personal Message (Offline)

Ignore
1480817165
Reply with quote  #2

1480817165
Report to moderator
Revalin
Hero Member
*****
Offline Offline

Activity: 728


165YUuQUWhBz3d27iXKxRiazQnjEtJNG9g


View Profile
March 31, 2012, 03:07:57 PM
 #2

Database transactions require writing a journal, syncing, writing the data, syncing again, and then deleting the journal.  That's a much harder disk hit than linear writes.

Try using the just-released 0.6.0.  The settings have been changed for better disk performance.

      War is God's way of teaching Americans geography.  --Ambrose Bierce
Bitcoin is the Devil's way of teaching geeks economics.  --Revalin 165YUuQUWhBz3d27iXKxRiazQnjEtJNG9g
stakhanov
Full Member
***
Offline Offline

Activity: 175


View Profile
March 31, 2012, 03:09:49 PM
 #3

Thanks for your answer. I do have version 0.6x

I wonder if the database commits are simply too close to each other? The blockchain is public information and can always be redownloaded in the even of a crash, there is really no reason to write to the disk several times per second to secure it.
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1036


View Profile WWW
March 31, 2012, 03:14:05 PM
 #4

Thanks for your answer. I do have version 0.6x

I wonder if the database commits are simply too close to each other? The blockchain is public information and can always be redownloaded in the even of a crash, there is really no reason to write to the disk several times per second to secure it.

The hard part is not downloading or storing the block chain. It's building, maintaining and guaranteeing the on-disk consistency of the block and transaction index that is necessary for validation.

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
Revalin
Hero Member
*****
Offline Offline

Activity: 728


165YUuQUWhBz3d27iXKxRiazQnjEtJNG9g


View Profile
March 31, 2012, 03:32:19 PM
 #5

They used to be much too close; hence the adjustment.

Obviously there's still considerable room for improvement.  Smiley

Edit: And why would you ever quit Bitcoin?  You smell like a traitor.  Smiley

      War is God's way of teaching Americans geography.  --Ambrose Bierce
Bitcoin is the Devil's way of teaching geeks economics.  --Revalin 165YUuQUWhBz3d27iXKxRiazQnjEtJNG9g
stakhanov
Full Member
***
Offline Offline

Activity: 175


View Profile
March 31, 2012, 03:40:37 PM
 #6

The hard part is not downloading or storing the block chain. It's building, maintaining and guaranteeing the on-disk consistency of the block and transaction index that is necessary for validation.

Well, I suppose you've given it a lot of thought already. But I still have a hard time believing downloading the block chain and creating an index requires all that disk hammering. On my computer, the .bitcoin directory is 1.6GB big. It's big, but not that big. It even fits in the memory of most computers! How can building it require hammering the disk for hours?
Revalin
Hero Member
*****
Offline Offline

Activity: 728


165YUuQUWhBz3d27iXKxRiazQnjEtJNG9g


View Profile
March 31, 2012, 03:44:30 PM
 #7

I was just kidding, but everything you say is correct.  The startup and shutdown times are very high.  SOME disk hit is expected, but it's still quite excessive.

      War is God's way of teaching Americans geography.  --Ambrose Bierce
Bitcoin is the Devil's way of teaching geeks economics.  --Revalin 165YUuQUWhBz3d27iXKxRiazQnjEtJNG9g
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1036


View Profile WWW
March 31, 2012, 03:45:11 PM
 #8

The hard part is not downloading or storing the block chain. It's building, maintaining and guaranteeing the on-disk consistency of the block and transaction index that is necessary for validation.

Well, I suppose you've given it a lot of thought already. But I still have a hard time believing downloading the block chain and creating an index requires all that disk hammering. On my computer, the .bitcoin directory is 1.6GB big. It's big, but not that big. It even fits in the memory of most computers! How can building it require hammering the disk for hours?

It takes about half an hour for me, when loading from a local file (a patch that will probably be in 0.7) instead of from the network.

If you have a lot of RAM, you can run it in a RAM drive (if your OS supports that) and watch it fly. Of course, you'll miss the one hard part: not losing or corrupting the on-disk index when your computer crashes.

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
Revalin
Hero Member
*****
Offline Offline

Activity: 728


165YUuQUWhBz3d27iXKxRiazQnjEtJNG9g


View Profile
March 31, 2012, 03:51:14 PM
 #9

The fact that it's possible to download the whole thing onto a RAM disk, manually copy it to disk, and sync with orders of magnitude less of an IO hit suggests that the DB is doing something pedagogically.

      War is God's way of teaching Americans geography.  --Ambrose Bierce
Bitcoin is the Devil's way of teaching geeks economics.  --Revalin 165YUuQUWhBz3d27iXKxRiazQnjEtJNG9g
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1036


View Profile WWW
March 31, 2012, 04:02:21 PM
 #10

The fact that it's possible to download the whole thing onto a RAM disk, manually copy it to disk, and sync with orders of magnitude less of an IO hit suggests that the DB is doing something pedagogically.

Patches very welcome Smiley

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
stakhanov
Full Member
***
Offline Offline

Activity: 175


View Profile
March 31, 2012, 05:06:41 PM
 #11

The fact that it's possible to download the whole thing onto a RAM disk, manually copy it to disk, and sync with orders of magnitude less of an IO hit suggests that the DB is doing something pedagogically.

Patches very welcome Smiley

Just to be clear, this is a perfectly acceptable answer to me! I just wanted to make sure I wasn't missing something obvious.
deepceleron
Legendary
*
Offline Offline

Activity: 1470



View Profile WWW
March 31, 2012, 05:20:57 PM
 #12

The hard part is not downloading or storing the block chain. It's building, maintaining and guaranteeing the on-disk consistency of the block and transaction index that is necessary for validation.

Well, I suppose you've given it a lot of thought already. But I still have a hard time believing downloading the block chain and creating an index requires all that disk hammering. On my computer, the .bitcoin directory is 1.6GB big. It's big, but not that big. It even fits in the memory of most computers! How can building it require hammering the disk for hours?
You'd believe how much it writes to the disk if you make a new ~8gb disk image, defrag & push all the data to front, and then simply install and run Bitcoin. The blockchain will be fragmented all over the disk from all the writes and erases to the database.

fghj
Member
**
Offline Offline

Activity: 65


View Profile
December 24, 2012, 11:37:47 AM
 #13


Sorry for necroing this thread, posting something that is obvious to most people (was for me in hindsight) and that will be moot when v0.8 will be released but I didn't found it written anywhere. If you don't have enough RAM for storing entire blockchain in RAMdisk you can put blkindex.dat into RAM and do a symlink to it from datadir. This will also speed up syncing and will require only about 1.8GB of ram instead of 6+GB.
Also remember to copy it back to hard drive when finished
paraipan
Legendary
*
Offline Offline

Activity: 924


Firstbits: 1pirata


View Profile WWW
December 24, 2012, 02:21:02 PM
 #14


Sorry for necroing this thread, posting something that is obvious to most people (was for me in hindsight) and that will be moot when v0.8 will be released but I didn't found it written anywhere. If you don't have enough RAM for storing entire blockchain in RAMdisk you can put blkindex.dat into RAM and do a symlink to it from datadir. This will also speed up syncing and will require only about 1.8GB of ram instead of 6+GB.
Also remember to copy it back to hard drive when finished

Better, buy a 64gb SSD drive (around 50€) and move bitcoin datadir onto it. You should see the difference right away  Wink

BTCitcoin: An Idea Worth Saving - Q&A with bitcoins on rugatu.com - Check my rep
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!