Bitcoin Forum
November 18, 2019, 01:38:00 PM *
News: Help collect the most notable posts made over the last 10 years.
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Bitcoin livenet size in gigabytes  (Read 107 times)
msaikat
Newbie
*
Offline Offline

Activity: 7
Merit: 4


View Profile
September 25, 2019, 02:50:37 PM
Merited by LoyceV (1)
 #1

Hello,

I have a full node running with "bitcored" and it's getting a huge disk space and running out of it several times! Mine is having 495G in total and the node syncing took all of it and now shows "No space left on device" in debug.log. Can anybody mention what is the recommended disk size for a full livenet node. Earlier I heard about 200G which didn't work for me. Or any suggestion that can improve this scenario?

/msaikat
The Bitcoin Forum is turning 10 years old! Join the community in sharing and exploring the notable posts made over the years.
1574084280
Hero Member
*
Offline Offline

Posts: 1574084280

View Profile Personal Message (Offline)

Ignore
1574084280
Reply with quote  #2

1574084280
Report to moderator
BitCryptex
Hero Member
*****
Offline Offline

Activity: 756
Merit: 1077


Write @BitCryptex or quote my post to notify me


View Profile WWW
September 25, 2019, 02:59:49 PM
Last edit: September 25, 2019, 03:13:55 PM by BitCryptex
 #2

Mine is having 495G in total and the node syncing took all of it and now shows "No space left on device" in debug.log. Can anybody mention what is the recommended disk size for a full livenet node. Earlier I heard about 200G which didn't work for me. Or any suggestion that can improve this scenario?

You can use pruning if you don't mind getting rid of old blocks after they have been verified. Only about 288 recent blocks will be kept in case of chain reorganizations by default.

#Edit: I have no experience with bitcored which might be the root of the problem.

OmegaStarScream
Staff
Legendary
*
Offline Offline

Activity: 1834
Merit: 1504


Exchange Bitcoin quicky--https://blockchain.com.do


View Profile
September 25, 2019, 03:11:16 PM
Merited by BitCryptex (2)
 #3

Are you hosting it on VMWare? This looks like the answer you're looking for: https://bitcoin.stackexchange.com/questions/65005/bitcored-vmware-disk-space-increases-incredibly-fast

msaikat
Newbie
*
Offline Offline

Activity: 7
Merit: 4


View Profile
September 25, 2019, 03:37:47 PM
Merited by OmegaStarScream (1)
 #4


You can use pruning if you don't mind getting rid of old blocks after they have been verified. Only about 288 recent blocks will be kept in case of chain reorganizations by default.

#Edit: I have no experience with bitcored which might be the root of the problem.



Hi @BitCryptex Thank you for the reply. I have two things to know here...

(1) The -prune mode is incompatible with -txindex and -rescan. Is that -txindex helps to reduce the disk space?

(2) What options should I use while running bitcoind. Now I am using only "exec": "/usr/lib/node_modules/bitcore/node_modules/bitcore-node/bin/bitcoind" in bitcore-node.json

FYI I installed using "npm install -g --unsafe-perm=true bitcore" and "apt-get install bitcoind"

I really appreciate your help.
bob123
Legendary
*
Offline Offline

Activity: 1078
Merit: 1575



View Profile WWW
September 25, 2019, 04:07:36 PM
Merited by suchmoon (4), BitCryptex (1)
 #5

(1) The -prune mode is incompatible with -txindex and -rescan. Is that -txindex helps to reduce the disk space?

-txindex is for indexing all transactions, not just the ones which affect your wallet.
For regular use this is not needed. If you however are developing something like a blockchain explorer or chain anyalysis software, you definitely need txindex enabled.

If you are simply just running a full node to either support the network or to not rely on any 3rd party, you can freely disable txindex. And if you wish, enable pruning. This will only save X MB of the last blocks (default = 500).

BitCryptex
Hero Member
*****
Offline Offline

Activity: 756
Merit: 1077


Write @BitCryptex or quote my post to notify me


View Profile WWW
September 25, 2019, 04:36:33 PM
 #6

(2) What options should I use while running bitcoind. Now I am using only "exec": "/usr/lib/node_modules/bitcore/node_modules/bitcore-node/bin/bitcoind" in bitcore-node.json

Sorry, I don't have any experience with Bitcore so I will have to rely on the information available on their Github and the Internet since I don't have any spare VPS at the moment to test it. What guide did you follow? Did you create bitcoind config file? Take a look at this guide and see if it helps you. Skip 'testnet=1' in the bitcoind config and change 'testnet' to 'mainnet' in the bitcore-node.json. Do you need bitcore for some kind of service or you just want to run a full node?

msaikat
Newbie
*
Offline Offline

Activity: 7
Merit: 4


View Profile
September 25, 2019, 05:00:57 PM
 #7

Are you hosting it on VMWare? This looks like the answer you're looking for: https://bitcoin.stackexchange.com/questions/65005/bitcored-vmware-disk-space-increases-incredibly-fast


Thnaks for asking. I am using AWS EC2 instance
msaikat
Newbie
*
Offline Offline

Activity: 7
Merit: 4


View Profile
September 25, 2019, 05:20:04 PM
 #8


Sorry, I don't have any experience with Bitcore so I will have to rely on the information available on their Github and the Internet since I don't have any spare VPS at the moment to test it. What guide did you follow? Did you create bitcoind config file? Take a look at this guide and see if it helps you. Skip 'testnet=1' in the bitcoind config and change 'testnet' to 'mainnet' in the bitcore-node.json. Do you need bitcore for some kind of service or you just want to run a full node?

Hi,

I actually followed the exact same steps described in the guide you mentioned and yes, bitcoin.conf file was created. I changed to "network": "livenet". I need bitcore to build my own application to perform transactions with bitcoin address creation, view tx status etc.

I am now facing only the storage issue and wondering if I need to start re-sync the full node!
bob123
Legendary
*
Offline Offline

Activity: 1078
Merit: 1575



View Profile WWW
September 25, 2019, 05:44:15 PM
 #9

I need bitcore to build my own application to perform transactions with bitcoin address creation, view tx status etc.

Does your appllication only need to handle transactions affecting your wallet ? Or all transactions ?
If it is the latter one, you need txindex enabled (which implies no pruning possible), otherwise not.



I am now facing only the storage issue and wondering if I need to start re-sync the full node!

How much storage do you have accessible ?
If your node is pruned and you remove it, then you'll need to resync.
Same applies to if you had txindex disabled, and enable it.

Otherwise, it should not be necessary.

msaikat
Newbie
*
Offline Offline

Activity: 7
Merit: 4


View Profile
September 25, 2019, 06:20:03 PM
 #10


Does your appllication only need to handle transactions affecting your wallet ? Or all transactions ?
If it is the latter one, you need txindex enabled (which implies no pruning possible), otherwise not.


All transaction will be required to be considered



How much storage do you have accessible ?
If your node is pruned and you remove it, then you'll need to resync.
Same applies to if you had txindex disabled, and enable it.

Otherwise, it should not be necessary.

I have 495G of space which is already occupied! I didn't run with -prune mode earlier but there is a "txindex=1" in "bitcoin.conf"
bob123
Legendary
*
Offline Offline

Activity: 1078
Merit: 1575



View Profile WWW
September 25, 2019, 06:25:20 PM
 #11

Well, if you need all transactions accessible then you can't use pruning mode since you need to have txindex enabled.

Those ~500GB do not come from core, right ?
For core fully synced you'll need roughly 250GB.

It seems like your only option is to get more storage. At least if you want to develop something which requires all transactions being indexed.

msaikat
Newbie
*
Offline Offline

Activity: 7
Merit: 4


View Profile
September 25, 2019, 06:57:29 PM
Merited by bob123 (2)
 #12

Those ~500GB do not come from core, right ?

I am not 100% clear what you meant by core here? Here is the folder structure contains everything...

$ ls -l
total 116
Sep 18 14:22 bitcore-node.json
Sep 25 18:50 data
Sep 18 14:22 node_modules
Sep 18 14:22 package-lock.json
Sep 18 14:22 package.json

$ sudo du -sh
481G

$ cd data
$ ls -l
Sep 18 13:00 bitcoin.conf
Sep 24 07:43 blocks
Sep 24 07:54 chainstate
Sep 24 07:54 database
Sep 18 13:00 db.log
Sep 25 18:53 debug.log
Sep 24 07:54 fee_estimates.dat
Sep 24 06:56 peers.dat
Sep 24 07:54 peers.dat.077e
Sep 24 07:37 peers.dat.7e3a
Sep 24 07:52 wallet.dat

$ sudo du -h
16K  ./database
3.6G ./chainstate
261G ./blocks/index
478G ./blocks
481G


It seems like your only option is to get more storage. At least if you want to develop something which requires all transactions being indexed.

Can you give an idea how much disk-space I need to add to the AWS EC2 instance?
BitCryptex
Hero Member
*****
Offline Offline

Activity: 756
Merit: 1077


Write @BitCryptex or quote my post to notify me


View Profile WWW
September 25, 2019, 07:05:24 PM
Merited by bob123 (2)
 #13

261G ./blocks/index
478G ./blocks
481G

It looks like it's because of the txindex set to 1. According to this reddit post, 2 years ago, one had to download 108 GB without txindex turned on and 226 GB once enabled. It looks like you will need about 1 TB of space. I haven't run a node with txindex turned on so someone else should confirm if this is true.

msaikat
Newbie
*
Offline Offline

Activity: 7
Merit: 4


View Profile
September 25, 2019, 07:15:13 PM
 #14

It looks like it's because of the txindex set to 1. According to this reddit post, 2 years ago, one had to download 108 GB without txindex turned on and 226 GB once enabled. It looks like you will need about 1 TB of space. I haven't run a node with txindex turned on so someone else should confirm if this is true.

Thanks for the answer.
BitCryptex
Hero Member
*****
Offline Offline

Activity: 756
Merit: 1077


Write @BitCryptex or quote my post to notify me


View Profile WWW
September 25, 2019, 07:27:56 PM
 #15

Thanks for the answer.

Whoops, I looked at 478 GB in the previous quote instead of subtracting 261 GB from it. Right now, you have downloaded 217 GB out of about 227 GB of block data. You should need about 35 GB of space more to finish the sync. Keep in mind that about 6 blocks are mined every hour so you will need an extra space but definitely less than 1 TB to keep your node operational for now.

Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!