Blockchain is 360GB and growing, can it be consolidated?

[BlackHatCoiner]

this means the more people that prune. the less archives are decentralised. meaning less peers available to get the full blockchain from.

The more people that prune means more nodes, which definitely not means less decentralization or less peers available to get the full chain from.

but just be aware that being a pruned node does not make you part of the full decentralised network, you are more of the edge of the network

What do you mean the edge? That you'll need to redownload the block chain if you ever wanted to add another wallet? That you do not share any block and hence, not help for the bandwidth of the network? If it's the former, that has nothing to do with the network and if it's the latter, you do share those few blocks you have.

Those nodes may be tracking your requests, and they might link your IP with your addresses.

Or even worse:  They can link your addresses. What you're describing as a problem can be tackled by connecting through Tor.

[n0nce]

but just be aware that being a pruned node does not make you part of the full decentralised network, you are more of the edge of the network

What do you mean the edge? That you'll need to redownload the block chain if you ever wanted to add another wallet? That you do not share any block and hence, not help for the bandwidth of the network? If it's the former, that has nothing to do with the network and if it's the latter, you do share those few blocks you have.

You're bringing up an interesting point here. I was always mostly discarding the idea of pruned nodes as helpful for the network, but one might actually prune for example to 150GB or 100GB; thus still serving new nodes almost half of the blockchain, which isn't unsubstantial and still relatively easy on hard disk space. Many games these days take 100GB, so I feel allocating 100 gigs to helping the Bitcoin network like '30%' utility, is better than nothing, if you don't have the space for a full blockchain copy.

Probably my pessmistic view of pruned nodes came from the idea of pruning down to like 5GB, but of course the limit can be set to anything you want.
Added benefit of e.g. keeping the last 100GB: You will probably have a copy of each block mined since you entered and owned BTC if you're still new to Bitcoin.

[pooya87]

You're bringing up an interesting point here. I was always mostly discarding the idea of pruned nodes as helpful for the network, but one might actually prune for example to 150GB or 100GB; thus still serving new nodes almost half of the blockchain, which isn't unsubstantial and still relatively easy on hard disk space. Many games these days take 100GB, so I feel allocating 100 gigs to helping the Bitcoin network like '30%' utility, is better than nothing, if you don't have the space for a full blockchain copy.

Probably my pessmistic view of pruned nodes came from the idea of pruning down to like 5GB, but of course the limit can be set to anything you want.
Added benefit of e.g. keeping the last 100GB: You will probably have a copy of each block mined since you entered and owned BTC if you're still new to Bitcoin.

I may be wrong but the P2P protocol needs to change for this to work. For now the pruned node can only announce that they are pruned and not able to tell the other node how many blocks they have stored so any node that connects to a pruned node assumes they have the minimum number of blocks. This means it may not matter if you store 100 GB or 500 MB they won't ask for old blocks from your node.

[franky1]

this means the more people that prune. the less archives are decentralised. meaning less peers available to get the full blockchain from.

The more people that prune means more nodes, which definitely not means less decentralization or less peers available to get the full chain from.

more that prune.. means more that DONT HAVE THE FULL BLOCKCHAIN.. why.. because they pruned
yep pruned nodes dont have the full blockchain. you can tell because they only have 7gb of space used not 360gb.
more people with only 7gb which is mostly only the last fortnight means more people that dont have all the data since genesis.
this then means more leachers are all then relying on less full nodes that do have 360gb full blockchain. and those less full nodes with full blockchain have to broadcast more data to those leachers meaning more centralised.

but just be aware that being a pruned node does not make you part of the full decentralised network, you are more of the edge of the network

What do you mean the edge? That you'll need to redownload the block chain if you ever wanted to add another wallet? That you do not share any block and hence, not help for the bandwidth of the network? If it's the former, that has nothing to do with the network and if it's the latter, you do share those few blocks you have.

its an old phrase 'supernodes', but its a thing. the big main fullnodes of pools and exchanges select who they want to connect with for anti-DDos security and also to ensure they are linked to other important nodes for transaction relay efficiency.  

imagine pools/exchanges at the centre of a network and they transmit data out. like an exploding star. where each peer(planets) expands the universe of nodes outwards.
the lite/pruned wallets/nodes wont be seen as a priority nodes which the planets near the explosion want to transmit to. instead lite/pruned nodes are far out at the edge of the universe.
[not to be confused with an alt-nets bad naming buzzword for a channel]

You're bringing up an interesting point here. I was always mostly discarding the idea of pruned nodes as helpful for the network, but one might actually prune for example to 150GB or 100GB; thus still serving new nodes almost half of the blockchain, which isn't unsubstantial and still relatively easy on hard disk space. Many games these days take 100GB, so I feel allocating 100 gigs to helping the Bitcoin network like '30%' utility, is better than nothing, if you don't have the space for a full blockchain copy.

a pruned node keeps X amount of most recent block
new users wanting to IBD(initial block download) start from block 0 not from block 560,000
so a pruned node only keeping block 560k to 712k wont have blocks 0-559k. and thus the new peer will disconnect and try finding another peer. meaning the pruned nodes are not getting any sustained downstream connections and there for they become the outer edge of the peer universe.

its the same for old version nodes that dont store all the witness data. new nodes will reject the data because its not full witness and disconnect from the node leaving the old node at the edge of the universe

[BlackHatCoiner]

I may be wrong but the P2P protocol needs to change for this to work.

Yes, it's probably working as you say.

this then means more leachers are all then relying on less full nodes that do have 360gb full blockchain. and those less full nodes with full blockchain have to broadcast more data to those leachers meaning more centralised.

It only means that these nodes will have to transmit a lot more data as much more nodes will want to initially download the chain. The network does not become more centralized or decentralized as the number of full nodes rises; it only helps bandwidth-wise. Centralization is utterly dependent on the distribution of the hash rate.

[n0nce]

[~] thus still serving new nodes almost half of the blockchain, which isn't unsubstantial and still relatively easy on hard disk space. [~]

I may be wrong but the P2P protocol needs to change for this to work. For now the pruned node can only announce that they are pruned and not able to tell the other node how many blocks they have stored so any node that connects to a pruned node assumes they have the minimum number of blocks. This means it may not matter if you store 100 GB or 500 MB they won't ask for old blocks from your node.

Argh, damn, that's a pity! I thought new nodes could e.g. get the first 200GB from a full node and the last 100GB from a pruned one, for example, what a bummer. Does anything speak against adding a mechanism that would allow to accomplish this?

[~]
a pruned node keeps X amount of most recent block
new users wanting to IBD(initial block download) start from block 0 not from block 560,000
so a pruned node only keeping block 560k to 712k wont have blocks 0-559k.

Sorry franky, but your long post (including the parts I did exemplarily quote) is full of information we here probably all knew already, it's all quite trivial basic stuff (how a pruned node works). The only kicker was that I assumed a pruned node was able to 'seed' the blocks that it does keep in storage.

Let's assume that you have a 500GB HDD and the blockchain data folder reaches this limit; with a Bitcoin Core implementation that allows pruned nodes to seed blocks to new users, one may set the 'prune' setting to exactly 500GB and thus keep everything but the very first few blocks. New nodes would need to fetch these first few megabytes (later gigabytes) from full nodes, sure, but could get all the following blocks from large, pruned nodes. A big benefit to the network, in my opinion. What do you guys think?

[pooya87]

Argh, damn, that's a pity! I thought new nodes could e.g. get the first 200GB from a full node and the last 100GB from a pruned one, for example, what a bummer. Does anything speak against adding a mechanism that would allow to accomplish this?

I believe the arguments against letting the pruned node announce the number of blocks it has was a privacy related one. Implementation details have to be checked by someone who can read C++ better than I, but you can see the the BIP advises that to avoid fingerprinting pruned nodes shouldn't provide blocks deeper than 288 threshold
https://github.com/bitcoin/bips/blob/master/bip-0159.mediawiki#counter-measures-for-peer-fingerprinting

[franky1]

Sorry franky, but your long post (including the parts I did exemplarily quote) is full of information we here probably all knew already, it's all quite trivial basic stuff (how a pruned node works). The only kicker was that I assumed a pruned node was able to 'seed' the blocks that it does keep in storage.

i appreciate that some do know the basics, but it appears some others do not. also this forum is open to all people who may want to learn. so instead of using buzzwords only devs know meanings of, or creating a culture of talking in special ways that only those "in the club" would know like some elitist group, sometimes its best to actually teach readers and not presume that things must be avoided because 'special group already knows'
(ive noticed this club culture alot)

anyways

I believe the arguments against letting the pruned node announce the number of blocks it has was a privacy related one. Implementation details have to be checked by someone who can read C++ better than I, but you can see the the BIP advises that to avoid fingerprinting pruned nodes shouldn't provide blocks deeper than 288 threshold
https://github.com/bitcoin/bips/blob/master/bip-0159.mediawiki#counter-measures-for-peer-fingerprinting

its not a situation that pruned nodes should limit their depth to 288 blocks. its that nodes should have the latest 288 blocks. mainly to have 288confirm protection from chain re-orgs

some merchants/custodians with customers depositing once a month. may want to keep full block details of say 3 months, for their own personal accounting checks and algorithms.

the whole fingerprinting. is just those privacy guys wanting to not offer any specific info through their tor connection that can also be compared to their clearnet IP connection. EG if they all default to 288 blocks. they are then just a shadow in a crowd of shadows showing nothing unique that stands out

Let's assume that you have a 500GB HDD and the blockchain data folder reaches this limit; with a Bitcoin Core implementation that allows pruned nodes to seed blocks to new users, one may set the 'prune' setting to exactly 500GB and thus keep everything but the very first few blocks. New nodes would need to fetch these first few megabytes (later gigabytes) from full nodes, sure, but could get all the following blocks from large, pruned nodes. A big benefit to the network, in my opinion. What do you guys think?

many pruners(leachers) are not the type that stay online 24/7.. they are usually popping online connecting to a full node peer grabbing 2week-2month re-sync in a couple hours. doing their own transaction and going offline.
thus not being online 24-7 to only request 6 blocks an hour as a plod along stay uptodate scenario.
thus full node peers are getting more connections demanding thousands of blocks an hour per connection and then seeing that peer disconnect and not a plod along 6 blocks an hour stay online situation.
this off-on-off scenario also means pruners are not online long enough to be 'latest block' seeders to take pressure off the full nodes by not being online to accept connections from others so that full nodes dont have to.. but instead full nodes still end up with genuine users wanting to help the network plodding along at 6 blocks an hour PLUS all the leachers dropping in and out at thousands of blocks an hour. plus all the genuine IBD people.
which is why many supernodes blacklist the pruners. and not connect to them

blachhatcoiner
other people read this. other people can learn from this. its not a private message. so dont turn it into an opportunity to be a drama queen. even newbies read this topic!!
and no. dont poke the bear causing drama hoping i bite so you can play victim.

its not a situation to act like an elitist bussworder hiding basic info unless a newbie drops a message asking for clarity. its a situation that newbies are here. this topic is open to all. not just 3 people

side note: this topic was created by a newbie. so backdown thinking that mentioning flaws should only be held in private conversation with minimal people and anyone mentioning flaws should be shunned. people can learn from flaws, and the hope is that devs also learn so they can fix the flaws. hiding things helps no one

[BlackHatCoiner]

i appreciate that some do know the basics, but it appears some others do not. also this forum is open to all people who may want to learn.

It's open for all people to learn, but the way you spread information isn't in good faith. It always seem like you want to shut up your interlocutors. In other words, you're ill-intentioned.

Besides, we should teach or resolve an issue newbies have, if... There are newbies. The above posts clearly consist a dialogue between you, me, pooya87 and n0nce. Any newbie who has questions can obviously create a whole new thread dedicated for them.

[PrimeNumber7]

Only validating blocks past block x would add uncertainty to the validation process.

You mean certainty? Validating blocks prior x or just starting from the genesis block adds certainty that the blocks are valid.

You validate blocks from the past to current. So validating past block 700,000 gives me less certainty in the validation process as would only validating past block 600k, and both would give me less certainty as validating past block 1 (the genesis block). xtdev

Playing devils advocate (and seeing if I can either disprove or validate your point):

Where's devil's advocate? Judging by your post, I can only assume you agree with me.

A Devil's Advocate is to "take...a position they do not necessarily agree with...for the sake of debate or to explore the thought further..."

I was hoping to start a debate regarding not validating all blocks. However, I don't think there is a good argument in favor of not validating all previously found blocks, as there would result in it being trivial to execute a sybil attack, even if the adversary does not control all connections to a node.

[BlackHatCoiner]

So validating past block 700,000 gives me less certainty in the validation process as would only validating past block 600k, and both would give me less certainty as validating past block 1 (the genesis block).

Yes, that's what I said. The more the blocks you validate, the more the certainty. I'm still not sure I understand what you mean in here:

Only validating blocks past block x would add uncertainty to the validation process.

I assume x is an arbitrary number?

However, I don't think there is a good argument in favor of not validating all previously found blocks

I know the “Devil's advocate” idiom's meaning. It's just that I couldn't find one valid reason to not validate the previous blocks and therefore, couldn't consider it a Devil's advocate argument.

[PrimeNumber7]

I'm still not sure I understand what you mean in here:

Only validating blocks past block x would add uncertainty to the validation process.

I assume x is an arbitrary number?

Right. block X would be a block in which the chainstate (all unspent outputs as of that block) is in the block.

However, I don't think there is a good argument in favor of not validating all previously found blocks

I know the “Devil's advocate” idiom's meaning. It's just that I couldn't find one valid reason to not validate the previous blocks and therefore, couldn't consider it a Devil's advocate argument.

This is an interesting topic to me. If it was possible to not have to validate all blocks, it would greatly reduce the barriers to running a full node (as low as they are currently). Both the time and space required to start running a full node could be reduced to a constant, as opposed to a function of the number of blocks (or a constant for space and a function of the number of blocks for time if running a pruned node).

[franky1]

If it was possible to not have to validate all blocks, it would greatly reduce the barriers to running a full node

if you didnt validate all blocks. your not a full node
if you dont archive all blocks, to then provide to other peers. your not a full node
if you accept without validating transactions(eg missing witness). your not a full node
if you 'accept' stripped blocks (missing witness) and archive only the 1mb section. your not a full node

in all these cases cannot provide all info to be a good seeder for others.
years ago devs called this being a 'bridged' or 'downstream' node, though even that buzzword was not useful.

the adverts that the network is 'backward compatible' is a mis-information to pretend that old nodes are full. even when they dont validate all active rules and dont store all data per transaction/block

there are many many reasons why people should be proper full nodes to help the decentralisation of the network and avoid chain re-org risks due to many possible reasons.
hiding the risks of pruning, downgrading(or not staying uptodate), weakens the network

here is the thing.
bitnodes lists 14k nodes(at time of writing). but there are not 14k 'fullnodes' listed.
take just one thing. i just searched the nodes at current height and found only 3000 of them fully uptodate
then out of those 3000 nodes. not all of them are using versions 2X.X, meaning LESS than 3000 are even checking for taproot valid transactions
and i havnt even got to those LESS THAN 3000 that are set as prune
meaning right now at time of typing out of the 14k nodes listed on bitnodes. less than 3k of them are actually able to offer full node peer services. and be able to relay the latest block solved by a pool. all the others(11k) are missing blocks or have not fully validated taproot transactions properly using the new rules. or have the full blockchain for IBD

[PrimeNumber7]

If it was possible to not have to validate all blocks, it would greatly reduce the barriers to running a full node

if you didnt validate all blocks. your not a full node
if you dont archive all blocks, to then provide to other peers. your not a full node
if you accept without validating transactions(eg missing witness). your not a full node
if you 'accept' stripped blocks (missing witness) and archive only the 1mb section. your not a full node

in all these cases cannot provide all info to be a good seeder for others.
years ago devs called this being a 'bridged' or 'downstream' node, though even that buzzword was not useful.

If you are running a full node, you need to validate all blocks, and you need to validate all transactions in each block.

In many parts of the world, bandwidth is expensive, specifically uploading data to the internet. So there are likely many nodes that do not broadcast blocks/transactions (including old blocks) to the rest of the network, including nodes that are trying to sync.

If you have validated a particular transaction, including its signature/witness, you know the transaction is valid. You will continue knowing the transaction is valid regardless of if you retain the signature data.

Similarly, if you have validated that a block is valid, you will continue to know that a block is valid, regardless of what you do with the information in a block.

[franky1]

my premiss is this
if you just want to validate your own transactions and you dont care about being part of the full node network of peers for the benefit of the bitcoin network universe. then fine you prune or use a lite wallet, whatever you choose. just know and understand that your not part of the full node network.

understand that by you saving your bandwidth/hard drive by not being a seeder means your not helping the real seeders.. and acept that your ok with that 'less than' status, because it fits your lifestyle

but if you want to be a full node then be a full node.
none of this wishy washy 'everyones a full node' pish posh

the network needs to have a strong amount of actual fullnodes
not a 3:11 ratio of full:lesthan group thinking they are all offering a full decentralised service for the network
without even knowing that there is a difference

fullnodes have the entire blockchain all the way upto the latest block. where all blockdata is included and all transactions within are validated. that way chain re-orgs cant happen easily/at all

however pretending that the bitnodes listing of todays 14k nodes are all "full nodes" is not correct thinking and not good in respect of a strong network of available nodes to service other peers

[pooya87]

some merchants/custodians with customers depositing once a month. may want to keep full block details of say 3 months, for their own personal accounting checks and algorithms.

I honestly don't see any reason why anyone would store the whole blockchain for "accounting checks". The node can know it received a transaction with a certain amount and at a certain block height which means they also know the time it was received so they can fetch the equivalent price too. What additional information can the full block provide?

Besides their node can be set up to fetch any block at any time from other peers if they wanted to. As I said they already know which block contains their transaction so all it takes is a getdata message.

1. if you didnt validate all blocks. your not a full node
2. if you dont archive all blocks, to then provide to other peers. your not a full node
3. if you accept without validating transactions(eg missing witness). your not a full node
4. if you 'accept' stripped blocks (missing witness) and archive only the 1mb section. your not a full node

Although definition of certain terms are sometimes flexible but "full node" definition was always about "full verification" not "full storage and full verification". So I don't agree with #2, you are still a full node if you aren't storing the whole blockchain. #3 and #4 are the same thing and similar to #1 and I agree that if you aren't validating everything (eg. old nodes not verifying SegWit transactions) you are no longer a full [verifying] node.

bitnodes lists 14k nodes(at time of writing). but there are not 14k 'fullnodes' listed.
take just one thing. i just searched the nodes at current height and found only 3000 of them fully uptodate

That site lists listening nodes not all bitcoin nodes which is a lot more than 14k.
You don't have to listen for incoming connection to be a full node or verify blocks or provide the blocks you have to other peers, ...

less than 3k of them are actually able to offer full node peer services. and be able to relay the latest block solved by a pool. all the others(11k) are missing blocks ~ or have the full blockchain for IBD

Being a full node is more than just providing historical blocks for IBD though.
A node should also enforce consensus rules, verify and relay new blocks, have an active mempool and verify relay transactions, relay other peers addresses, ...

[tromp]

> Although definition of certain terms are sometimes flexible but "full node" definition was always about "full verification" not "full storage and full verification".

Agreed.

Furthermore, many of the nodes considered to be full nodes are actually not,
as they ran with the default assumevalid = true.

[Kakmakr]

When people work with a database.. they eventually reach a point where they do not need the "old" data and then they archive that data from a specific point in time. Why can someone not develop some side-chain that holds the "archived" Blockchain data that was archived and the rest of the people hold a pruned version of it for say the last 12 months?

I know the Blockchain data is not supposed to be a problem, because storage media size increase over time too... but it will just be more efficient to only use the last 12 months of active data ..and then have the other data archived onto another Blockchain or side-chain?

If a tx needs to address that data.. it goes to a side chain.... or something built for that purpose?

[n0nce]

~

How would a 'sidechain' (that's not how sidechains work but let's play devil's advocate) need less data than just leaving the old blocks on the 'main chain'?

[BlackHatCoiner]

Why can someone not develop some side-chain that holds the "archived" Blockchain data that was archived and the rest of the people hold a pruned version of it for say the last 12 months?

And who would hold the “archived” block chain data? There's no point to have a side-chain in order for people to hold a pruned version of the chain. Just run a pruned node. That way, you can verify the validity of the chain without having to keep it.

In short:  You can choose to keep or not keep the whole block chain. Either way, you have to verify every single transaction. If you don't (which would happen the way you describe), you create the problem the entire project relies on solving:

that would force you to trust the system.