Bitcoin Forum
September 21, 2021, 07:36:55 PM *
News: Latest Bitcoin Core release: 22.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Hard Drive Speed running a Core Node  (Read 218 times)
biggieM
Newbie
*
Offline Offline

Activity: 1
Merit: 0


View Profile
April 25, 2018, 12:03:25 AM
 #1

Hey, I'm getting a Bitcoin Core node running on an old MacBook Air I have lying around. It doesn't have enough SSD space to store the blockchain, so I want to use an old 1TB external hard drive I have. Its reads and writes are around 25MB/s. Is this fast enough to run a core node? On Bitcoin.org it recommends 100 MB/s. Do you think I could get away with 25? Thanks in advance

-BiggieM
1632253015
Hero Member
*
Offline Offline

Posts: 1632253015

View Profile Personal Message (Offline)

Ignore
1632253015
Reply with quote  #2

1632253015
Report to moderator
1632253015
Hero Member
*
Offline Offline

Posts: 1632253015

View Profile Personal Message (Offline)

Ignore
1632253015
Reply with quote  #2

1632253015
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1632253015
Hero Member
*
Offline Offline

Posts: 1632253015

View Profile Personal Message (Offline)

Ignore
1632253015
Reply with quote  #2

1632253015
Report to moderator
achow101
Moderator
Legendary
*
Offline Offline

Activity: 2590
Merit: 4180


Just writing some code


View Profile WWW
April 25, 2018, 12:15:42 AM
 #2

Syncing Bitcoin Core for you is going to take a long time with slow write speeds. Also, because it is an external drive, the blockchain will have a higher chance of becoming corrupted. So you could store the blockchain on your external drive, but I would not recommend it.

ranochigo
Legendary
*
Offline Offline

Activity: 2464
Merit: 2635

@ me if you need my response


View Profile
April 25, 2018, 12:23:43 AM
 #3

The synchronizing is going to take forever since its constantly reading and writing to the disk. Why don't you consider pruning it? If you've got at least 550MB on your MacBook, you can prune it without sacrificing too much of the functionality.

Jet Cash
Legendary
*
Offline Offline

Activity: 2100
Merit: 2145


https://fittotalk.com/english-talk/


View Profile WWW
April 25, 2018, 08:15:04 AM
 #4

I've got 3 copies of the blockchain. One on an internal HDD, one on an external SSD, and the third is an external HDD that I use for backup. I've got two computers one running Ubuntu, and one with Windows 10. The fastest combo for sync'ing the blockchain is the external SSD on the Ubuntu machine. I wouldn't bother with pruning, as it isn't the space that is your problem, but the sync speed. I'd be tempted to buy an external 256Gb SSD to download the blockchain, and then copy it onto your HDD. Put everything ( blockchain and wallet ) into a Bitcoin directory, and you can do the download on a faster computer to get started.

[
Offgrid campers allow you to enjoy life and preserve your health and wealth.
LoyceV
Legendary
*
Offline Offline

Activity: 2338
Merit: 8890


Thick-Skinned Gang Leader


View Profile WWW
April 25, 2018, 10:12:47 AM
 #5

Is this fast enough to run a core node? On Bitcoin.org it recommends 100 MB/s.
This doesn't make much sense, as the disk access time is much more important than sequential read write speed. Do you have a link to this recommendation? Could it be the internet speed they recommend to run a node?

To share my 2 cents out of my own experience: I would just install Bitcoin Core, let it create it's default ~/.bitcoin directory on your SSD, then close it.
Then, inside the .bitcoin-directory, move the blocks-directory to your external HDD, and create a simlink to the new location. While you're doing this anyway, create a backup of your wallet.dat.
Start Bitcoin Core again. Most disk activity happens in the chainstate-directory, which is now on your fast SSD. The blocks directory is on your large HDD.

Note though that you should make absolutely sure your USB disk doesn't disconnect while running Bitcoin Core!

cellard
Legendary
*
Offline Offline

Activity: 1372
Merit: 1225


View Profile
April 25, 2018, 02:25:51 PM
 #6

Is this fast enough to run a core node? On Bitcoin.org it recommends 100 MB/s.
This doesn't make much sense, as the disk access time is much more important than sequential read write speed. Do you have a link to this recommendation? Could it be the internet speed they recommend to run a node?

To share my 2 cents out of my own experience: I would just install Bitcoin Core, let it create it's default ~/.bitcoin directory on your SSD, then close it.
Then, inside the .bitcoin-directory, move the blocks-directory to your external HDD, and create a simlink to the new location. While you're doing this anyway, create a backup of your wallet.dat.
Start Bitcoin Core again. Most disk activity happens in the chainstate-directory, which is now on your fast SSD. The blocks directory is on your large HDD.

Note though that you should make absolutely sure your USB disk doesn't disconnect while running Bitcoin Core!

achow101 doesn't recommend using symlinks, it can lead to blockchain corruption. Everything must be on the same folder, blocks and chainstate files. Even after the latest updates where you can select where to put the wallet.dat, this doesn't change chaintate and blocks folder must be on the same place.

In my opinion one should try to save money monthly until you can afford an SSD of 1TB of storage, which should last you for years to host the blockchain. HDD disks take a massive toll when validating the blockchain due high i/o activity, the headers may break eventually due wear and tear, maybe you end up saving money long term investing in an SSD.
LoyceV
Legendary
*
Offline Offline

Activity: 2338
Merit: 8890


Thick-Skinned Gang Leader


View Profile WWW
April 25, 2018, 07:49:27 PM
 #7

achow101 doesn't recommend using symlinks, it can lead to blockchain corruption. Everything must be on the same folder, blocks and chainstate files.
It's never happened to me, but even if it would happen, I don't really mind losing my blockchain directory. When it comes to that, I get a new disk and download it again.
I'm not running any critical business environment, it's just my home PC.

Jet Cash
Legendary
*
Offline Offline

Activity: 2100
Merit: 2145


https://fittotalk.com/english-talk/


View Profile WWW
April 26, 2018, 12:48:50 PM
 #8

My current experience more or less confirms the comments above. I decided to sync the external HDD using the small Ubuntu machine, which uses the external SSD normally. No problems connecting it, but it reports that it is 7 weeks behind, and at one point, it said that it would take 12 weeks to complete the sync. Smiley It's now dropped to 6 days, but I don't think I'll bother. The SSD is reasonably fast, and that's what I use daily. The 2Tb external HDD is really just a backup. The next plan is to create a Bitcoin1 and a Bitcoin2 directory on the HDD, and to use a file management program to move the Core sub-directories into them alternately.

For me, it is important to keep a backup of the blockchain, as I don't think I could handle downloading it again, especially as it is double the size of my original download.

[
Offgrid campers allow you to enjoy life and preserve your health and wealth.
cellard
Legendary
*
Offline Offline

Activity: 1372
Merit: 1225


View Profile
April 26, 2018, 03:12:54 PM
 #9

My current experience more or less confirms the comments above. I decided to sync the external HDD using the small Ubuntu machine, which uses the external SSD normally. No problems connecting it, but it reports that it is 7 weeks behind, and at one point, it said that it would take 12 weeks to complete the sync. Smiley It's now dropped to 6 days, but I don't think I'll bother. The SSD is reasonably fast, and that's what I use daily. The 2Tb external HDD is really just a backup. The next plan is to create a Bitcoin1 and a Bitcoin2 directory on the HDD, and to use a file management program to move the Core sub-directories into them alternately.

For me, it is important to keep a backup of the blockchain, as I don't think I could handle downloading it again, especially as it is double the size of my original download.

How safe is to re-use blockchain files? could they have been compromised if they were in a windows setup in the past? At the beginning I was using windows to run Bitcoin Core, then I got a Linux setup and moved my block files in there to avoid downloading from scratch which is hell, but a part of me is paranoid about the fact that the block files were hosted in a Windows setup for a long time... I don't see how a hacker would be able to extract value from messing with the blocks file but still a bit paranoid about that. Im considering making a clean download from Linux from scratch but just thinking about it gives me an headache by how long it will take for it to sync.
LoyceV
Legendary
*
Offline Offline

Activity: 2338
Merit: 8890


Thick-Skinned Gang Leader


View Profile WWW
April 26, 2018, 03:46:02 PM
Last edit: April 26, 2018, 04:39:00 PM by LoyceV
 #10

How safe is to re-use blockchain files? could they have been compromised if they were in a windows setup in the past?
What if the node you're downloading from is compromised?
The answer is the same: as long as your version of Bitcoin Core isn't compromised, it can verify all blocks again.

Quote
to avoid downloading from scratch which is hell
I used to have that experience too, but after adding much more RAM and speed improvements within Bitcoin Core, I really don't mind anymore. I'm not going to test it, but I estimate it'll be done within 2 days (on a cheap laptop that's more than 3 years old).

Quote
but a part of me is paranoid about the fact that the block files were hosted in a Windows setup for a long time...
I thought it would be a trivial thing to give a rescan-the-blockchain option at startup, but all it can do is rescan the wallet.
I've read that removing the chainstate forces a rescan, but I don't want to risk it (and I don't have the space to make a copy first). Any thoughts?

Quote
Im considering making a clean download from Linux from scratch but just thinking about it gives me an headache by how long it will take for it to sync.
If you have enough space, you can just rename your current .bitcoin and download a new one. If it takes too long, you can always move it back.

cellard
Legendary
*
Offline Offline

Activity: 1372
Merit: 1225


View Profile
April 26, 2018, 04:10:39 PM
 #11


I used to have that experience too, but after adding much more RAM and speed improvements within Bitcoin Core, I really don't mind anymore. I'm not going to test it, but I estimate it'll be done within 2 days (on a cheap laptop that's more than 3 years old).

How much RAM? safe (older) laptops tipically will only allow 8 GB and it's ddr2 ram.


I thought it would be a trivial thing to give a rescan-the-blockchain option at startup, but all it can do is rescan the wallet.
I've read that removing the chainstate forces a rescan, don't want to risk it (and I don't have the space to make a copy first). Any thoughts?.

Yes removing chainstate files forces a rescan and it's pretty safe as far as I know. You can force a rescan by adding "rescan=1" in bitcoin.conf file or with the terminal adding -rescan.

I will just keep the block files for now, can't be bothered with a full download from scratch. Chainstate files obviously were not imported from windows to linux setup so in theory the blocks that I imported should be valid.
LoyceV
Legendary
*
Offline Offline

Activity: 2338
Merit: 8890


Thick-Skinned Gang Leader


View Profile WWW
April 26, 2018, 04:43:27 PM
 #12

How much RAM? safe (older) laptops tipically will only allow 8 GB and it's ddr2 ram.
4+8 now, but it's not from before 2008 if that's what you mean with "safe".


I thought it would be a trivial thing to give a rescan-the-blockchain option at startup, but all it can do is rescan the wallet.
I've read that removing the chainstate forces a rescan, don't want to risk it (and I don't have the space to make a copy first). Any thoughts?.

Quote
You can force a rescan by adding "rescan=1" in bitcoin.conf file or with the terminal adding -rescan.
Are you sure about this?
Code:
  -rescan
       Rescan the block chain for missing wallet transactions on startup
As far as I know, this only checks your wallet, but doesn't check the blocks. Bitcoin Core assumes the blocks on disk are correct. I would expect rescanning them to use just as much CPU and disk access as when you're downloading them for the first time.

Jet Cash
Legendary
*
Offline Offline

Activity: 2100
Merit: 2145


https://fittotalk.com/english-talk/


View Profile WWW
April 26, 2018, 07:32:27 PM
 #13

I'm pretty certain that the Ubuntu machine has only got 2Gb of ram. and I think it's a Celeron processor. It was a cheap machine that I bought to surf and post whilst I was in the van. I can't see how the blockchain can be corrupted, as Core verifies it at startup. Installing a new hard drive, and a new version of Core will avoid ny chance of infection. I'd need  second opinion on the use of the wallet and other files associated with the blockchin, but as they aren't executables, I don't see how they could introduce a vulnerability.

[
Offgrid campers allow you to enjoy life and preserve your health and wealth.
cellard
Legendary
*
Offline Offline

Activity: 1372
Merit: 1225


View Profile
April 28, 2018, 12:02:54 PM
 #14

I'm pretty certain that the Ubuntu machine has only got 2Gb of ram. and I think it's a Celeron processor. It was a cheap machine that I bought to surf and post whilst I was in the van. I can't see how the blockchain can be corrupted, as Core verifies it at startup. Installing a new hard drive, and a new version of Core will avoid ny chance of infection. I'd need  second opinion on the use of the wallet and other files associated with the blockchin, but as they aren't executables, I don't see how they could introduce a vulnerability.

I wouldn't run a full node with 2 gigs of RAM... that is overkill. Ubuntu is too resource consuming if you want to go down that route. If I were to use a 2GB old Celeron for the task I would for sure use Lubuntu, which is a super minimalist version of Ubuntu, ideal for old machines.

Wallet file's risk resides in if they ever sat in an online machine. You want to ideally create a new wallet in a machine that has never seen, and will never see the internet, and use it as cold storage, and use a separate computer to broadcast transactions.
ranochigo
Legendary
*
Offline Offline

Activity: 2464
Merit: 2635

@ me if you need my response


View Profile
April 29, 2018, 03:30:35 PM
 #15

I wouldn't bother with pruning, as it isn't the space that is your problem, but the sync speed.
Pruning solves the issue with his limited storage on his macbook. I wasn't addressing his external harddrive speed since that's harder to solve.
Yes removing chainstate files forces a rescan and it's pretty safe as far as I know. You can force a rescan by adding "rescan=1" in bitcoin.conf file or with the terminal adding -rescan.

Rescan doesn't make the client validate the blockchain. I believe that deleting the chainstate alone won't force the client to check all the blocks again. If you want to be on the safe side, delete the block index too. Either way, it would take somewhat the same amount of time to validate again as your initial synchronization.

cellard
Legendary
*
Offline Offline

Activity: 1372
Merit: 1225


View Profile
April 30, 2018, 11:11:36 AM
 #16

I wouldn't bother with pruning, as it isn't the space that is your problem, but the sync speed.
Pruning solves the issue with his limited storage on his macbook. I wasn't addressing his external harddrive speed since that's harder to solve.
Yes removing chainstate files forces a rescan and it's pretty safe as far as I know. You can force a rescan by adding "rescan=1" in bitcoin.conf file or with the terminal adding -rescan.

Rescan doesn't make the client validate the blockchain. I believe that deleting the chainstate alone won't force the client to check all the blocks again. If you want to be on the safe side, delete the block index too. Either way, it would take somewhat the same amount of time to validate again as your initial synchronization.

I believe chainstate should force validation of blocks again... otherwise these files would lack sense.

If your bottleneck is your computer and you have a fast internet connection then yes, it will not make that much difference if you can download 165 GB pretty fast, but if it's the opposite (fast computer but slow connection) then it will make an huge difference.
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!