Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: deadlift75 on November 30, 2021, 03:34:03 PM



Title: Blockchain is 360GB and growing, can it be consolidated?
Post by: deadlift75 on November 30, 2021, 03:34:03 PM
(Please forgive my lack of specific technical knowledge, my question is likely dumb, so help me understand why)

If the blockchain is a chain of transactions, and its purpose (among others) seems to be to avoid double spending, why can't it be regularly consolidated into a state condition (as in, "at this time, wallet1 has N bitcoins", "at this time, wallet2 has N bitcoins", etc. or however that information would be stored in the same/new blockchain) to save on space?

This consolidation would obviously have to be proven to be consistent with previous transaction history, but once it's done, it would be accepted and added to the blockchain.

Once a state is accepted and consolidated into the blockchain, I assume the entire history of transactions before that state could be discarded for most purposes.

If this consolidation happened, let's say, once a month, and only the record of transactions from the previous month and the previous state was kept, transactions would only need to be checked against this relatively small data set.

Does this make sense? What problems would arise from this that I'm not seeing? Thanks!


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: bitmover on November 30, 2021, 03:39:32 PM
You are basically talking about a pruned node.

It would take about 5GB only, according this post
https://bitcointalk.org/index.php?topic=5213552.0

Prune node
Pros:
  • Required about 5 GB by default (a little it more than 5 GB). [a]
  • You don't have store all blockchain database on your computer
  • You can run a prune node to get experience and pratice with Bitcoin core without significant pressure on your computer storage space.
[a]: In reality, you will prepare at least a little more than 5GB for your prune node with a minimum storage value for blocks (at 2 GB). They are included by:
  • Chainstate: 3.5 GB
  • Blocks: 2 GB
  • Initial setup: 52.1 MB (see the below image)

They are safe to use. The user basically download the whole blockchain, but compress most of the data, so it doesn't occupy so much disk space.

You still need the same bandwidth


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: BlackHatCoiner on November 30, 2021, 03:50:38 PM
Short answer:  Because, that would force you to trust the system.

Long answer:  You may have heard the term “Don't trust, verify” as it's said often in social media; it has a lot of point. Each block with all those transactions is valid, because your node verifies it. Because your node has checked that once you hash the content, you do, indeed, get a valid proof of work. This isn't something insignificant to just pass.

Once a person spends bitcoins, you don't trust the others that they've supposedly verified (and now discarded) the old blocks that contained their transactions. You don't trust anybody; you, instead, verify it yourself since the first block that this very guy has had that many bitcoins.

To give a simplified example:  Say you wanted to run a node when the chain height is 100,000 and the network, by consensus, discards any block after the 1,000th most recent one. Once your node synced, you'd have verified that the transactions in the last 1,000 blocks are valid, but you'd have to trust that the rest nodes verified the other 99,000. You see the problem?


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: deadlift75 on November 30, 2021, 03:53:33 PM
Hi, bitmover, thanks! I haven't heard of pruned nodes, I like the term "chainstate".

My question now is why does the network still need full nodes, I mean what's being checked against the full nodes that can't be checked or trusted on the information of the pruned nodes?


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: deadlift75 on November 30, 2021, 04:01:55 PM
BlackHatCoiner, very good point about "trust but verify", I agree.

However, what if this consolidation happened every month or year, leaving everybody plenty of time to publicly verify previous states and transactions?

Regarding your example, I see why syncing the node with a network that limits itself to the last 1000 transactions would be a problem, but since past nodes never change, can't a verification on past nodes produce some publicly verifiable hash that can just be checked and everybody can just move on?


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: BlackHatCoiner on November 30, 2021, 04:24:38 PM
My question now is why does the network still need full nodes
For the reason I told you. Due to the need for trust.

However, what if this consolidation happened every month or year, leaving everybody plenty of time to publicly verify previous states and transactions?
There'll never be enough time. Once the consolidation sets sail, every other person who will want to find out the previous transactions by running a node, will have to trust the other nodes.

I see why syncing the node with a network that limits itself to the last 1000 transactions
I said 1,000 blocks, but anyway. Give or take whatever you want, it's a similar analogy.

but since past nodes never change, can't a verification on past nodes produce some publicly verifiable hash that can just be checked and everybody can just move on?
Yeah, of course they can produce a hash. The problem is that you can't verify that hash if you don't have the content of the block. I may give you the following hash:
Code:
000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f

But, if you don't have the whole thing, which is this (https://pastebin.com/raw/xgGtCcwg), how can you confirm that I've, indeed, found a valid proof of work and not written those hexadecimal characters by myself with zero effort?


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: deadlift75 on November 30, 2021, 05:17:13 PM
The problem is that you can't verify that hash if you don't have the content of the block. I may give you the following hash:
I see your point. I agree with you that trust would be less self-verifiable and delegated to an extent. The idea is that the hash would be verified against previous transactions up to the previous state, but the problem remains: how to verify the previous state?

Correct me if I'm wrong, but I understand trust here as being statistical, as in, "highly unlikely or computationally too demanding to be otherwise". By the same logic, how likely can the entire network be fooled into accepting a wrong but publicly verifiable state for a month or a year?


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: n0nce on November 30, 2021, 05:28:50 PM
How to verify the previous state?
See, without such a consolidation and by instead verifying the whole blockchain, block by block starting from the genesis block, there is no previous state to verify or anything like that. That's why it's really the best way to do things. Also there have been ideas like yours in the past, such as inserting 'sync blocks' into the blockchain periodically, but all these changes even if just slightly, decrease the security, meanwhile the blockchain only grows slowly and storage gets cheaper and cheaper.

The motivation to change anything is really not strong enough. During Black Friday, there were 1TB SSDs for <80€, and it gets cheaper with HDDs. 1TB will last like 10 years-ish, if I remember correctly. I can save up another 80 bucks during 10 years if I'll then need to get a second drive lol.

Also, one thing you got wrong in your initial post:
Quote
"at this time, wallet1 has N bitcoins"

Bitcoin is not balance-based, it's UTXO-based. This makes some things better, some things harder.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: DaveF on November 30, 2021, 05:41:07 PM
Correct me if I'm wrong, but I understand trust here as being statistical, as in, "highly unlikely or computationally too demanding to be otherwise". By the same logic, how likely can the entire network be fooled into accepting a wrong but publicly verifiable state for a month or a year?

The entire network does not matter, it's if A node can be fooled into accepting it.
And it's just not worth the risk.

This conversation keeps coming up on the forum about "ways to shrink the blockchain" and the answer is you can't. If you want to verify everything then you need everything.
If you don't want to verify everything, then there are plenty of light wallets that you don't need to download the entire blockchain for.

n0nce even started a thread on how you can run a full node for under $50
https://bitcointalk.org/index.php?topic=5364742.0


-Dave



Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: HeRetiK on November 30, 2021, 06:10:48 PM
In addition to what the others already said, keep in mind that Bitcoin is also about choice. With the way it works now, it's completely up to you how much you want to trust and how much additional effort you want to expend to stay trustless:

Want to remain trustless and are willing to keep some extra HD space for easier re-syncing if something goes wrong? Use a full node.

Want to remain trustless but don't want to use more HD space than necessary? Use a pruned node.

Want to spend as few resources as possible while keeping your own keys? Use a SPV wallet.


Each approach comes with a tradeoff of security vs convenience and while SPV wallet suffices for most people it's still important to allow for maximum security and trustlessness, where required.



On a sidenote, transaction cut-throughs may be interesting to you:
https://bitcointalk.org/index.php?topic=281848.0

Some minor alts are using this approach, unfortunately implementation doesn't appear to be feasible for Bitcoin though.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: deadlift75 on November 30, 2021, 06:43:37 PM
Yeah, I'm pretty sure this was suggested before or some version of it, I was more curious about the objections, so thank you guys for helping me understand.  :)


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: BlackHatCoiner on November 30, 2021, 07:07:01 PM
Correct me if I'm wrong, but I understand trust here as being statistical, as in, "highly unlikely or computationally too demanding to be otherwise".
No, you should see this in another way. Alright, let's say that Bob who runs his own node receives 1 BTC and the consolidation happens each time a new block is mined, for the 1,000th block. This means that sometime, his node will get rid of the block that contains that transaction and will mark his balance with 1 BTC.

Now, let's say that Alice wants to run her own node and starts receiving the consolidated balances of all the bitcoin addresses. What will prevent Bob from changing his mark that he owns 1 BTC? How can Alice verify that Bob, indeed, owns 1 BTC without trusting his sayings? The answer is:  By verifying every single transaction from the very first block.

Each approach comes with a tradeoff of security vs convenience and while SPV wallet suffices for most people it's still important to allow for maximum security and trustlessness, where required.
There's no tradeoff in security when it comes to running a node or trusting an SPV server. There's only tradeoff in privacy for the latter.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: HeRetiK on November 30, 2021, 10:09:23 PM
Each approach comes with a tradeoff of security vs convenience and while SPV wallet suffices for most people it's still important to allow for maximum security and trustlessness, where required.
There's no tradeoff in security when it comes to running a node or trusting an SPV server. There's only tradeoff in privacy for the latter.

Depends on your threat model! For a regular user it might make no difference (unlike e.g. storing your coins in a hot wallet vs using cold storage or a hardware wallet), but if you run a crypto-heavy business like an exchange I'd argue that running your own full node is safer over relying on someone elses.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: PrimeNumber7 on December 01, 2021, 03:53:52 AM

To give a simplified example:  Say you wanted to run a node when the chain height is 100,000 and the network, by consensus, discards any block after the 1,000th most recent one. Once your node synced, you'd have verified that the transactions in the last 1,000 blocks are valid, but you'd have to trust that the rest nodes verified the other 99,000. You see the problem?
Playing devils advocate (and seeing if I can either disprove or validate your point):

If a chainstate was included in block x, and a node that didn’t validate any blocks prior to block x, would need to assume the block x they received is valid. If it is not, the chainstate in block x is invalid. Similarly, block x+1 would be invalid.

One part of validating a block is validating the total proof of work, that is the sum of the POW performed access all blocks to date, and that the current POW is beyond the threshold for the current difficulty. So the cost of producing a fake block x is not trivial and if a node is connected to multiple other random nodes, they should receive the valid blockchain. There are a variety of scenarios in which the total POW for blocks prior to block x could be faked to make it appear that an invalid block x has the highest total POW when it is not a valid blockchain.

Only validating blocks past block x would add uncertainty to the validation process. The chances of receiving an invalid long chain of blocks is low considering the cost of producing invalid blocks, however there are trivial “fixes” to this process that can reduce the chances of accepting an invalid blockchain to nearly zero.

If you are running a pruned node, you will validate the entire blockchain, but will not store more than z blocks on your hard drive. So you will validate all blocks but will not store all blocks. This means that, unless you are being subjected to a Sybil attack, you will not accept an invalid blockchain, but will only store a limited number of blocks. The current implementation of a pruned node is not ideal, for example if a new private key is imported, it will need to download the entire blockchain again in order to know the unspent outputs it can spend, even though it has the UTXO set stored.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: tromp on December 01, 2021, 09:00:26 AM
A full (as in fully verifying) node needs two things in order to sync to the state of some current header.

1) the UTXO set

2) a proof that this UTXO set results from a valid block history from genesis to the current header

Part 1) only accounts for a small fraction of the 360GB; the vast majority is taken by part 2).

But in principle, part 2) could be replaced by a so-called Succinct Non-interactive Argument of Knowledge with recursive verification, that would be vastly smaller than part 1).

See this book for more details:

https://zcash.github.io/halo2/concepts/proofs.html

This technology is still in its infancy, and formalizing the entire Bitcoin consensus model into a SNARK would be a monumental effort, not to mention constructing the proof (which would have to be redone at regular intervals, e.g. every few months). But verification would be very efficient.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: BlackHatCoiner on December 01, 2021, 02:54:42 PM
Depends on your threat model!
Oh, so you meant the safety of your physical integrity?

If a chainstate was included in block x, and a node that didn’t validate any blocks prior to block x, would need to assume the block x they received is valid. If it is not, the chainstate in block x is invalid. Similarly, block x+1 would be invalid.
Yes, that's what I said. You have to trust the rest of the nodes.

Only validating blocks past block x would add uncertainty to the validation process.
You mean certainty? Validating blocks prior x or just starting from the genesis block adds certainty that the blocks are valid.

Playing devils advocate (and seeing if I can either disprove or validate your point):
Where's devil's advocate? Judging by your post, I can only assume you agree with me.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: HeRetiK on December 01, 2021, 03:39:14 PM
Depends on your threat model!
Oh, so you meant the safety of your physical integrity?

That's a great way to put it but not quite what I meant :D


I was more thinking along the lines of the 3rd party node spoofing incoming transactions, but it looks like that's less of an issue than I initially thought:

First, while the SPV client can not be easily fooled into thinking a transaction is in a block when it is not, the reverse is not true. A full node can simply lie by omission, leading an SPV client to believe a transaction has not occurred. This can be considered a form of Denial of Service.

Obviously the 3rd party node hiding transactions is not ideal either, but less exploitable than the other way round.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: franky1 on December 02, 2021, 11:35:30 AM
pruned nodes are not a trust thing.
all nodes(pruned or full) initially see a complete copy of all blocks and all transactions within.
just a pruned node does not archive them all.

this means a pruned node is not a (torrent buzzword) 'seeder' peer to then give that full information to others. thus peers disconnect from them and search for proper full nodes to get the data

if everyone pruned then no new users can then see the full blockchain to then do a full verification of all transactions.
.

this means the more people that prune. the less archives are decentralised. meaning less peers available to get the full blockchain from.
.
if your a random user that doesnt do much transactions or spend much time with your node running, whereby being a seeder or wanting to be part of the network is not important to you then fine prune. but if you care about decentralisation and want to be a full node then dont prune.

but just be aware that being a pruned node does not make you part of the full decentralised network, you are more of the edge of the network, deemed a (torrent buzzword)'leacher' peer, not a seeder peer. by which you are shrinking/centralising the network by reducing the amount of full node(seeder) peers available, plus also putting those remaining seeder peers under more strain to transmit their full blockchains downstream to random peers by being relied on more as blockchain sources

this can lead to a paradigm where full node archive nodes (seeders) need high bandwidth(unlimited data) internet plans because they are getting more demand, just to service the leachers. leading to another drop off of full nodes due to expense of servicing leachers. or leachers/new peers having a delay in finding seeder peers due to demand because seeder peers limit their connections to not use excessive data. both of which can shrink/centralise the amount of full nodes available, or slowdown the speed of getting blockdata


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: DaveF on December 02, 2021, 01:26:36 PM
To add to what franky1 said, you can also look at people who don't leave their nodes on full time the same way.
You open your client, sync the blockchain, do your transaction(s) and shut it down.
Yes, you can send out old data while your node is syncing, but you are still, more then likely taking in way more data then you are giving out.
Outside of keeping your TX on your own node for privacy, there is really very little reason to run your own node that way.

-Dave


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: bitmover on December 02, 2021, 02:38:35 PM
My question now is why does the network still need full nodes
For the reason I told you. Due to the need for trust.

pruned nodes are not a trust thing.
all nodes(pruned or full) initially see a complete copy of all blocks and all transactions within.
just a pruned node does not archive them all.

I agree with franky1 here. A pruned node is downloading the entire blockchain and verifying each transaction. There is no trust involved. Pruned nodes just doesn't achieve the entire blockchain, and save HD space. which is exactly what the OP is talking about.

Personally, i don't know what is their limitation, but you can even use Bitcoin Core software to run a pruned node.

My question now is why does the network still need full nodes, I mean what's being checked against the full nodes that can't be checked or trusted on the information of the pruned nodes?

Imo, a pruned node is a full node.

Let's look what bitcoin wiki says:
Quote
https://en.bitcoin.it/wiki/Full_node
Any computer that connects to the Bitcoin network is called a node. Nodes that fully verify all of the rules of Bitcoin are called full nodes. The most popular software implementation of full nodes is called Bitcoin Core, its latest release can be found on the github page.

Imo, a pruned node fits that description.

Anyway, the network needs full nodes to verify all transactions and all blockdata without any trust.

For example, let's image you are a big exchange (like Binance or Coinbase), or that you are blockexplorer. you need to run a full node to verify each single UTXO, not just the ones that you are interested with.

If someone sends you a coin which was received in 2013, you need to be able to fully verify it was not spent without trusting anyone.

Full nodes are also important for privacy purposes. When you use an spv wallet, this wallet asks for the balance of your addresses to other full nodes. Those nodes may be tracking your requests, and they might link your IP with your addresses.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: BlackHatCoiner on December 02, 2021, 04:01:42 PM
this means the more people that prune. the less archives are decentralised. meaning less peers available to get the full blockchain from.
The more people that prune means more nodes, which definitely not means less decentralization or less peers available to get the full chain from.

but just be aware that being a pruned node does not make you part of the full decentralised network, you are more of the edge of the network
What do you mean the edge? That you'll need to redownload the block chain if you ever wanted to add another wallet? That you do not share any block and hence, not help for the bandwidth of the network? If it's the former, that has nothing to do with the network and if it's the latter, you do share those few blocks you have.

Those nodes may be tracking your requests, and they might link your IP with your addresses.
Or even worse:  They can link your addresses. What you're describing as a problem can be tackled by connecting through Tor.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: n0nce on December 02, 2021, 11:41:34 PM
but just be aware that being a pruned node does not make you part of the full decentralised network, you are more of the edge of the network
What do you mean the edge? That you'll need to redownload the block chain if you ever wanted to add another wallet? That you do not share any block and hence, not help for the bandwidth of the network? If it's the former, that has nothing to do with the network and if it's the latter, you do share those few blocks you have.
You're bringing up an interesting point here. I was always mostly discarding the idea of pruned nodes as helpful for the network, but one might actually prune for example to 150GB or 100GB; thus still serving new nodes almost half of the blockchain, which isn't unsubstantial and still relatively easy on hard disk space. Many games these days take 100GB, so I feel allocating 100 gigs to helping the Bitcoin network like '30%' utility, is better than nothing, if you don't have the space for a full blockchain copy.

Probably my pessmistic view of pruned nodes came from the idea of pruning down to like 5GB, but of course the limit can be set to anything you want.
Added benefit of e.g. keeping the last 100GB: You will probably have a copy of each block mined since you entered and owned BTC if you're still new to Bitcoin.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: pooya87 on December 03, 2021, 05:02:54 AM
You're bringing up an interesting point here. I was always mostly discarding the idea of pruned nodes as helpful for the network, but one might actually prune for example to 150GB or 100GB; thus still serving new nodes almost half of the blockchain, which isn't unsubstantial and still relatively easy on hard disk space. Many games these days take 100GB, so I feel allocating 100 gigs to helping the Bitcoin network like '30%' utility, is better than nothing, if you don't have the space for a full blockchain copy.

Probably my pessmistic view of pruned nodes came from the idea of pruning down to like 5GB, but of course the limit can be set to anything you want.
Added benefit of e.g. keeping the last 100GB: You will probably have a copy of each block mined since you entered and owned BTC if you're still new to Bitcoin.
I may be wrong but the P2P protocol needs to change for this to work. For now the pruned node can only announce that they are pruned and not able to tell the other node how many blocks they have stored so any node that connects to a pruned node assumes they have the minimum number of blocks. This means it may not matter if you store 100 GB or 500 MB they won't ask for old blocks from your node.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: franky1 on December 03, 2021, 09:04:14 AM
this means the more people that prune. the less archives are decentralised. meaning less peers available to get the full blockchain from.
The more people that prune means more nodes, which definitely not means less decentralization or less peers available to get the full chain from.
more that prune.. means more that DONT HAVE THE FULL BLOCKCHAIN.. why.. because they pruned
yep pruned nodes dont have the full blockchain. you can tell because they only have 7gb of space used not 360gb.
more people with only 7gb which is mostly only the last fortnight means more people that dont have all the data since genesis.
this then means more leachers are all then relying on less full nodes that do have 360gb full blockchain. and those less full nodes with full blockchain have to broadcast more data to those leachers meaning more centralised.

but just be aware that being a pruned node does not make you part of the full decentralised network, you are more of the edge of the network
What do you mean the edge? That you'll need to redownload the block chain if you ever wanted to add another wallet? That you do not share any block and hence, not help for the bandwidth of the network? If it's the former, that has nothing to do with the network and if it's the latter, you do share those few blocks you have.

its an old phrase 'supernodes', but its a thing. the big main fullnodes of pools and exchanges select who they want to connect with for anti-DDos security and also to ensure they are linked to other important nodes for transaction relay efficiency.  

imagine pools/exchanges at the centre of a network and they transmit data out. like an exploding star. where each peer(planets) expands the universe of nodes outwards.
the lite/pruned wallets/nodes wont be seen as a priority nodes which the planets near the explosion want to transmit to. instead lite/pruned nodes are far out at the edge of the universe.
[not to be confused with an alt-nets bad naming buzzword for a channel]


You're bringing up an interesting point here. I was always mostly discarding the idea of pruned nodes as helpful for the network, but one might actually prune for example to 150GB or 100GB; thus still serving new nodes almost half of the blockchain, which isn't unsubstantial and still relatively easy on hard disk space. Many games these days take 100GB, so I feel allocating 100 gigs to helping the Bitcoin network like '30%' utility, is better than nothing, if you don't have the space for a full blockchain copy.
a pruned node keeps X amount of most recent block
new users wanting to IBD(initial block download) start from block 0 not from block 560,000
so a pruned node only keeping block 560k to 712k wont have blocks 0-559k. and thus the new peer will disconnect and try finding another peer. meaning the pruned nodes are not getting any sustained downstream connections and there for they become the outer edge of the peer universe.

its the same for old version nodes that dont store all the witness data. new nodes will reject the data because its not full witness and disconnect from the node leaving the old node at the edge of the universe



Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: BlackHatCoiner on December 03, 2021, 02:56:52 PM
I may be wrong but the P2P protocol needs to change for this to work.
Yes, it's probably working as you say.

this then means more leachers are all then relying on less full nodes that do have 360gb full blockchain. and those less full nodes with full blockchain have to broadcast more data to those leachers meaning more centralised.
It only means that these nodes will have to transmit a lot more data as much more nodes will want to initially download the chain. The network does not become more centralized or decentralized as the number of full nodes rises; it only helps bandwidth-wise. Centralization is utterly dependent on the distribution of the hash rate.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: n0nce on December 04, 2021, 02:32:06 AM
[~] thus still serving new nodes almost half of the blockchain, which isn't unsubstantial and still relatively easy on hard disk space. [~]
I may be wrong but the P2P protocol needs to change for this to work. For now the pruned node can only announce that they are pruned and not able to tell the other node how many blocks they have stored so any node that connects to a pruned node assumes they have the minimum number of blocks. This means it may not matter if you store 100 GB or 500 MB they won't ask for old blocks from your node.
Argh, damn, that's a pity! I thought new nodes could e.g. get the first 200GB from a full node and the last 100GB from a pruned one, for example, what a bummer. Does anything speak against adding a mechanism that would allow to accomplish this?

[~]
a pruned node keeps X amount of most recent block
new users wanting to IBD(initial block download) start from block 0 not from block 560,000
so a pruned node only keeping block 560k to 712k wont have blocks 0-559k.
Sorry franky, but your long post (including the parts I did exemplarily quote) is full of information we here probably all knew already, it's all quite trivial basic stuff (how a pruned node works). The only kicker was that I assumed a pruned node was able to 'seed' the blocks that it does keep in storage.

Let's assume that you have a 500GB HDD and the blockchain data folder reaches this limit; with a Bitcoin Core implementation that allows pruned nodes to seed blocks to new users, one may set the 'prune' setting to exactly 500GB and thus keep everything but the very first few blocks. New nodes would need to fetch these first few megabytes (later gigabytes) from full nodes, sure, but could get all the following blocks from large, pruned nodes. A big benefit to the network, in my opinion. What do you guys think?


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: pooya87 on December 04, 2021, 04:08:58 AM
Argh, damn, that's a pity! I thought new nodes could e.g. get the first 200GB from a full node and the last 100GB from a pruned one, for example, what a bummer. Does anything speak against adding a mechanism that would allow to accomplish this?
I believe the arguments against letting the pruned node announce the number of blocks it has was a privacy related one. Implementation details have to be checked by someone who can read C++ better than I, but you can see the the BIP advises that to avoid fingerprinting pruned nodes shouldn't provide blocks deeper than 288 threshold
https://github.com/bitcoin/bips/blob/master/bip-0159.mediawiki#counter-measures-for-peer-fingerprinting


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: franky1 on December 05, 2021, 08:36:13 PM
Sorry franky, but your long post (including the parts I did exemplarily quote) is full of information we here probably all knew already, it's all quite trivial basic stuff (how a pruned node works). The only kicker was that I assumed a pruned node was able to 'seed' the blocks that it does keep in storage.
i appreciate that some do know the basics, but it appears some others do not. also this forum is open to all people who may want to learn. so instead of using buzzwords only devs know meanings of, or creating a culture of talking in special ways that only those "in the club" would know like some elitist group, sometimes its best to actually teach readers and not presume that things must be avoided because 'special group already knows'
(ive noticed this club culture alot)

anyways
I believe the arguments against letting the pruned node announce the number of blocks it has was a privacy related one. Implementation details have to be checked by someone who can read C++ better than I, but you can see the the BIP advises that to avoid fingerprinting pruned nodes shouldn't provide blocks deeper than 288 threshold
https://github.com/bitcoin/bips/blob/master/bip-0159.mediawiki#counter-measures-for-peer-fingerprinting
its not a situation that pruned nodes should limit their depth to 288 blocks. its that nodes should have the latest 288 blocks. mainly to have 288confirm protection from chain re-orgs

some merchants/custodians with customers depositing once a month. may want to keep full block details of say 3 months, for their own personal accounting checks and algorithms.

the whole fingerprinting. is just those privacy guys wanting to not offer any specific info through their tor connection that can also be compared to their clearnet IP connection. EG if they all default to 288 blocks. they are then just a shadow in a crowd of shadows showing nothing unique that stands out

Let's assume that you have a 500GB HDD and the blockchain data folder reaches this limit; with a Bitcoin Core implementation that allows pruned nodes to seed blocks to new users, one may set the 'prune' setting to exactly 500GB and thus keep everything but the very first few blocks. New nodes would need to fetch these first few megabytes (later gigabytes) from full nodes, sure, but could get all the following blocks from large, pruned nodes. A big benefit to the network, in my opinion. What do you guys think?

many pruners(leachers) are not the type that stay online 24/7.. they are usually popping online connecting to a full node peer grabbing 2week-2month re-sync in a couple hours. doing their own transaction and going offline.
thus not being online 24-7 to only request 6 blocks an hour as a plod along stay uptodate scenario.
thus full node peers are getting more connections demanding thousands of blocks an hour per connection and then seeing that peer disconnect and not a plod along 6 blocks an hour stay online situation.
this off-on-off scenario also means pruners are not online long enough to be 'latest block' seeders to take pressure off the full nodes by not being online to accept connections from others so that full nodes dont have to.. but instead full nodes still end up with genuine users wanting to help the network plodding along at 6 blocks an hour PLUS all the leachers dropping in and out at thousands of blocks an hour. plus all the genuine IBD people.
which is why many supernodes blacklist the pruners. and not connect to them

blachhatcoiner
other people read this. other people can learn from this. its not a private message. so dont turn it into an opportunity to be a drama queen. even newbies read this topic!!
and no. dont poke the bear causing drama hoping i bite so you can play victim.

its not a situation to act like an elitist bussworder hiding basic info unless a newbie drops a message asking for clarity. its a situation that newbies are here. this topic is open to all. not just 3 people

side note: this topic was created by a newbie. so backdown thinking that mentioning flaws should only be held in private conversation with minimal people and anyone mentioning flaws should be shunned. people can learn from flaws, and the hope is that devs also learn so they can fix the flaws. hiding things helps no one


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: BlackHatCoiner on December 05, 2021, 08:53:06 PM
i appreciate that some do know the basics, but it appears some others do not. also this forum is open to all people who may want to learn.
It's open for all people to learn, but the way you spread information isn't in good faith. It always seem like you want to shut up your interlocutors. In other words, you're ill-intentioned.

Besides, we should teach or resolve an issue newbies have, if... There are newbies. The above posts clearly consist a dialogue between you, me, pooya87 and n0nce. Any newbie who has questions can obviously create a whole new thread dedicated for them.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: PrimeNumber7 on December 05, 2021, 09:44:20 PM
Only validating blocks past block x would add uncertainty to the validation process.
You mean certainty? Validating blocks prior x or just starting from the genesis block adds certainty that the blocks are valid.
You validate blocks from the past to current. So validating past block 700,000 gives me less certainty in the validation process as would only validating past block 600k, and both would give me less certainty as validating past block 1 (the genesis block). xtdev
Playing devils advocate (and seeing if I can either disprove or validate your point):
Where's devil's advocate? Judging by your post, I can only assume you agree with me.
A Devil's Advocate (https://en.wikipedia.org/wiki/Devil%27s_advocate) is to "take...a position they do not necessarily agree with...for the sake of debate or to explore the thought further..."

I was hoping to start a debate regarding not validating all blocks. However, I don't think there is a good argument in favor of not validating all previously found blocks, as there would result in it being trivial to execute a sybil attack, even if the adversary does not control all connections to a node.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: BlackHatCoiner on December 05, 2021, 09:53:46 PM
So validating past block 700,000 gives me less certainty in the validation process as would only validating past block 600k, and both would give me less certainty as validating past block 1 (the genesis block).
Yes, that's what I said. The more the blocks you validate, the more the certainty. I'm still not sure I understand what you mean in here:

Only validating blocks past block x would add uncertainty to the validation process.
I assume x is an arbitrary number?

However, I don't think there is a good argument in favor of not validating all previously found blocks
I know the “Devil's advocate” idiom's meaning. It's just that I couldn't find one valid reason to not validate the previous blocks and therefore, couldn't consider it a Devil's advocate argument.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: PrimeNumber7 on December 05, 2021, 10:06:52 PM
I'm still not sure I understand what you mean in here:

Only validating blocks past block x would add uncertainty to the validation process.
I assume x is an arbitrary number?
Right. block X would be a block in which the chainstate (all unspent outputs as of that block) is in the block.

However, I don't think there is a good argument in favor of not validating all previously found blocks
I know the “Devil's advocate” idiom's meaning. It's just that I couldn't find one valid reason to not validate the previous blocks and therefore, couldn't consider it a Devil's advocate argument.
This is an interesting topic to me. If it was possible to not have to validate all blocks, it would greatly reduce the barriers to running a full node (as low as they are currently). Both the time and space required to start running a full node could be reduced to a constant, as opposed to a function of the number of blocks (or a constant for space and a function of the number of blocks for time if running a pruned node).


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: franky1 on December 05, 2021, 11:34:22 PM
If it was possible to not have to validate all blocks, it would greatly reduce the barriers to running a full node

if you didnt validate all blocks. your not a full node
if you dont archive all blocks, to then provide to other peers. your not a full node
if you accept without validating transactions(eg missing witness). your not a full node
if you 'accept' stripped blocks (missing witness) and archive only the 1mb section. your not a full node

in all these cases cannot provide all info to be a good seeder for others.
years ago devs called this being a 'bridged' or 'downstream' node, though even that buzzword was not useful.

the adverts that the network is 'backward compatible' is a mis-information to pretend that old nodes are full. even when they dont validate all active rules and dont store all data per transaction/block

there are many many reasons why people should be proper full nodes to help the decentralisation of the network and avoid chain re-org risks due to many possible reasons.
hiding the risks of pruning, downgrading(or not staying uptodate), weakens the network

here is the thing.
bitnodes lists 14k nodes(at time of writing). but there are not 14k 'fullnodes' listed.
take just one thing. i just searched the nodes at current height and found only 3000 of them fully uptodate
then out of those 3000 nodes. not all of them are using versions 2X.X, meaning LESS than 3000 are even checking for taproot valid transactions
and i havnt even got to those LESS THAN 3000 that are set as prune
meaning right now at time of typing out of the 14k nodes listed on bitnodes. less than 3k of them are actually able to offer full node peer services. and be able to relay the latest block solved by a pool. all the others(11k) are missing blocks or have not fully validated taproot transactions properly using the new rules. or have the full blockchain for IBD


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: PrimeNumber7 on December 05, 2021, 11:52:37 PM
If it was possible to not have to validate all blocks, it would greatly reduce the barriers to running a full node

if you didnt validate all blocks. your not a full node
if you dont archive all blocks, to then provide to other peers. your not a full node
if you accept without validating transactions(eg missing witness). your not a full node
if you 'accept' stripped blocks (missing witness) and archive only the 1mb section. your not a full node

in all these cases cannot provide all info to be a good seeder for others.
years ago devs called this being a 'bridged' or 'downstream' node, though even that buzzword was not useful.
If you are running a full node, you need to validate all blocks, and you need to validate all transactions in each block.

In many parts of the world, bandwidth is expensive, specifically uploading data to the internet. So there are likely many nodes that do not broadcast blocks/transactions (including old blocks) to the rest of the network, including nodes that are trying to sync.

If you have validated a particular transaction, including its signature/witness, you know the transaction is valid. You will continue knowing the transaction is valid regardless of if you retain the signature data.

Similarly, if you have validated that a block is valid, you will continue to know that a block is valid, regardless of what you do with the information in a block.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: franky1 on December 06, 2021, 12:04:14 AM
my premiss is this
if you just want to validate your own transactions and you dont care about being part of the full node network of peers for the benefit of the bitcoin network universe. then fine you prune or use a lite wallet, whatever you choose. just know and understand that your not part of the full node network.

understand that by you saving your bandwidth/hard drive by not being a seeder means your not helping the real seeders.. and acept that your ok with that 'less than' status, because it fits your lifestyle

but if you want to be a full node then be a full node.
none of this wishy washy 'everyones a full node' pish posh

the network needs to have a strong amount of actual fullnodes
not a 3:11 ratio of full:lesthan group thinking they are all offering a full decentralised service for the network
without even knowing that there is a difference

fullnodes have the entire blockchain all the way upto the latest block. where all blockdata is included and all transactions within are validated. that way chain re-orgs cant happen easily/at all

however pretending that the bitnodes listing of todays 14k nodes are all "full nodes" is not correct thinking and not good in respect of a strong network of available nodes to service other peers




Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: pooya87 on December 06, 2021, 04:17:48 AM
some merchants/custodians with customers depositing once a month. may want to keep full block details of say 3 months, for their own personal accounting checks and algorithms.
I honestly don't see any reason why anyone would store the whole blockchain for "accounting checks". The node can know it received a transaction with a certain amount and at a certain block height which means they also know the time it was received so they can fetch the equivalent price too. What additional information can the full block provide?

Besides their node can be set up to fetch any block at any time from other peers if they wanted to. As I said they already know which block contains their transaction so all it takes is a getdata message.

1. if you didnt validate all blocks. your not a full node
2. if you dont archive all blocks, to then provide to other peers. your not a full node
3. if you accept without validating transactions(eg missing witness). your not a full node
4. if you 'accept' stripped blocks (missing witness) and archive only the 1mb section. your not a full node
Although definition of certain terms are sometimes flexible but "full node" definition was always about "full verification" not "full storage and full verification". So I don't agree with #2, you are still a full node if you aren't storing the whole blockchain. #3 and #4 are the same thing and similar to #1 and I agree that if you aren't validating everything (eg. old nodes not verifying SegWit transactions) you are no longer a full [verifying] node.

Quote
bitnodes lists 14k nodes(at time of writing). but there are not 14k 'fullnodes' listed.
take just one thing. i just searched the nodes at current height and found only 3000 of them fully uptodate
That site lists listening nodes not all bitcoin nodes which is a lot more than 14k.
You don't have to listen for incoming connection to be a full node or verify blocks or provide the blocks you have to other peers, ...

Quote
less than 3k of them are actually able to offer full node peer services. and be able to relay the latest block solved by a pool. all the others(11k) are missing blocks ~ or have the full blockchain for IBD
Being a full node is more than just providing historical blocks for IBD though.
A node should also enforce consensus rules, verify and relay new blocks, have an active mempool and verify relay transactions, relay other peers addresses, ...


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: tromp on December 06, 2021, 08:27:06 AM
> Although definition of certain terms are sometimes flexible but "full node" definition was always about "full verification" not "full storage and full verification".

Agreed.

Furthermore, many of the nodes considered to be full nodes are actually not,
as they ran with the default assumevalid = true.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: Kakmakr on December 06, 2021, 02:42:53 PM
When people work with a database.. they eventually reach a point where they do not need the "old" data and then they archive that data from a specific point in time. Why can someone not develop some side-chain that holds the "archived" Blockchain data that was archived and the rest of the people hold a pruned version of it for say the last 12 months?

I know the Blockchain data is not supposed to be a problem, because storage media size increase over time too... but it will just be more efficient to only use the last 12 months of active data ..and then have the other data archived onto another Blockchain or side-chain?

If a tx needs to address that data.. it goes to a side chain.... or something built for that purpose?


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: n0nce on December 06, 2021, 04:15:30 PM
~
How would a 'sidechain' (that's not how sidechains work but let's play devil's advocate) need less data than just leaving the old blocks on the 'main chain'?


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: BlackHatCoiner on December 06, 2021, 04:53:00 PM
Why can someone not develop some side-chain that holds the "archived" Blockchain data that was archived and the rest of the people hold a pruned version of it for say the last 12 months?
And who would hold the “archived” block chain data? There's no point to have a side-chain in order for people to hold a pruned version of the chain. Just run a pruned node. That way, you can verify the validity of the chain without having to keep it.

In short:  You can choose to keep or not keep the whole block chain. Either way, you have to verify every single transaction. If you don't (which would happen the way you describe), you create the problem the entire project relies on solving:

that would force you to trust the system.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: franky1 on December 06, 2021, 06:23:48 PM
I honestly don't see any reason why anyone would store the whole blockchain for "accounting checks".
just one example:
keeping the taint of movements of a utxos previous transactions is important for KYC and AML stuff
some services still want to avoid accepting transactions that have silkroad taint.
(pre-empt rebuttal:
unless you want to be one of those selfish leachers that request getdata of a new tx received previous taint blocks and then all blocks previous to that to then refind the full taint)

Although definition of certain terms are sometimes flexible but "full node" definition was always about "full verification" not "full storage and full verification". So I don't agree with #2, you are still a full node if you aren't storing the whole blockchain.
FULL means everything. a full node should do everything. hense the word FULL. by personally not wanting one feature, and pretending its not important because you dont want it. does not then still make the word full, full.. when its not doing the full job.
i get it its ok you dont want to do a full job.. but its not just about archiving. its about being a peer to seed other peers for IBD. its about having the full data to even be able to offer random blocks which as you say 'all it takes is the 'getdata' message'
sorry but if the peers dont have all blocks then your pruned peer cant then 'getdata' from that other peer of  random block X months ago
also if you are offline for a week and all pruned nodes are set to 288 blocks you also cant just resync just the last bit of time you were offline because there is a gap of 5 days of that week between when you went off line until the latest 288 blocks
you also cant offer any of this to others. literally making you as good as a lite wallet for the peer network

Quote
bitnodes lists 14k nodes(at time of writing). but there are not 14k 'fullnodes' listed.
take just one thing. i just searched the nodes at current height and found only 3000 of them fully uptodate
That site lists listening nodes not all bitcoin nodes which is a lot more than 14k.
You don't have to listen for incoming connection to be a full node or verify blocks or provide the blocks you have to other peers, ...
theres another thing nodes not listening means they are not offering a full service to the peer network.
but with that said i never said the 14k nodes were all nodes. but just an example selection by which even that selection are not even full nodes

Quote
less than 3k of them are actually able to offer full node peer services. and be able to relay the latest block solved by a pool. all the others(11k) are missing blocks ~ or have the full blockchain for IBD
Being a full node is more than just providing historical blocks for IBD though.
A node should also enforce consensus rules, verify and relay new blocks, have an active mempool and verify relay transactions, relay other peers addresses, ...
yep being a full node is more then just one thing. correct and thank you for correcting yourself after your earlier defence that you think that being a full nude is only a selected features you thought were important to you. EG you thought a full node didnt need to archive, without realising the other features not available by not archiving. so im glad to see you might have seen the light whereby having all data and all verification done is actually important not just to you, but to the peer network

In short:  You can choose to keep or not keep the whole block chain. Either way, you have to verify every single transaction. If you don't (which would happen the way you describe), you create the problem the entire project relies on solving:
gosh darn it. i think even blackhatcoiner is starting to get it too. finally seeing the light of the problem
just one note. although it means you by not verifying everything put you at risk by not independently trusting the data. being able to get the data from other peers because they have the data to give is step one of that process. and the more important feature. after all you cant verify data if there are limited/no sources of said data to then independently verify

anyways
like i said if you want to prune. thats your choice, but dont be deceived into thinking your a full node on the peer network helping the network

oh, last note:
Why can someone not develop some side-chain that holds the "archived" Blockchain data that was archived and the rest of the people hold a pruned version of it for say the last 12 months?
And who would hold the “archived” block chain data? There's no point to have a side-chain in order for people to hold a pruned version of the chain. Just run a pruned node. That way, you can verify the validity of the chain without having to keep it.

sounds like what kakmakr is hinting is a scenario where altnet like LN is refered to as 'bitcoin' where the actual blockchain network is defined as the 'sidechain'. so im surprised Blackhatcoiner has not sided with kakmakr thinking its a good idea. nice to see blackhatcoiner seeing the faults in kakmakrs idea. and takes note of this moment to realise what it actually means


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: BlackHatCoiner on December 06, 2021, 07:14:53 PM
gosh darn it. i think even blackhatcoiner is starting to get it too.
Do I interpret an aggressive behavior?

although it means you by not verifying everything put you at risk by not independently trusting the data. being able to get the data from other peers because they have the data to give is step one of that process. and the more important feature. after all you cant verify data if there are limited/no sources of said data to then independently verify
The underline parts don't make sense. If you really want to help a reader learn, you're failing miserably.

like i said if you want to prune. thats your choice, but dont be deceived into thinking your a full node on the peer network helping the network
Full nodes help the network. Did you possibly mean pruned node? Your text is hard to read.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: franky1 on December 06, 2021, 08:45:36 PM
gosh darn it. i think even blackhatcoiner is starting to get it too.
Do I interpret an aggressive behavior?
ignoring your poke, very obvious

although it means you by not verifying everything put you at risk by not independently trusting the data. being able to get the data from other peers because they have the data to give is step one of that process. and the more important feature. after all you cant verify data if there are limited/no sources of said data to then independently verify
The underline parts don't make sense. If you really want to help a reader learn, you're failing miserably.
dont start poking the bear with grammar debates as that just lessens your stance and makes you look like you are scraping the bottom of the barrel of argument just to poke the bear.

if you dont understand that to get data, other people need to have the data to give. so the first important step is that people have the data.
so again "being able to get the data from other peers because they have the data to give is step one of that process"

like i said if you want to prune. thats your choice, but dont be deceived into thinking your a full node on the peer network helping the network
Full nodes help the network. Did you possibly mean pruned node? Your text is hard to read.
yes pruned nodes as highlighted deceived into thinking they are full nodes, need to stop being deceived that they are full nodes helping the network..

now stop trying to make grammar/personal arguments for sole purpose causing drama.
your not the first that tried. and yes although i do usually bite. your not worth it.

..
point is archiving is important to being a full node for many reasons
archiving helps other peers:
Initial block download(IBD)
random block retrieval 'getdata'
resync more then just latest couple days
prevent easy chain re-orgs

it also helps yourself by:
being able to offer all of the above plus:
being able to follow taint of new transactions back to their original coin reward creation
any virus targetting utxoset doesnt require re-IBDing
plus more

addressing post below
when my post clearly addresses the pruners, by beginning the sentence 'if you want to prune thats your choice' then there is no argument about if i am addressing pruners in that sentence.

if i say that those needing data first, have to get data and that data can only be obtained if other users have the data. then clearly having the data is a important first step.

if however you ignore the context of whats been said and just want to knitpick HOW it was said. then you have missed the points entirely and just want to convert this topic into some social drama pretending your the victim and im the instigator. maybe next time concentrate on the context of WHAT is being said. not HOW.
this is not the first time grammar pokers try to get bit then cry victim. even when the original quotes they pretend not to understand did actually make sense. especially when they are willing to spend hours social drama arguing grammar but not spending 20 seconds just reading the context.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: n0nce on December 06, 2021, 08:52:00 PM
dont start poking the bear with grammar debates as that just lessens your stance and makes you look like you are scraping the bottom of the barrel of argument just to poke the bear.
My 2 sats: it's not about poking; it's just extremely hard to read your posts. Sorry, but they're so bad that I would need to put in an unreasonable amount of time and energy to try decrypting them; thus skipping your posts entirely.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: BlackHatCoiner on December 06, 2021, 09:00:31 PM
dont start poking the bear with grammar debates as that just lessens your stance
As n0nce pointed out, it's not about poking. I barely understand what you write and you should consider it justified if one does not understand a part of your text. Besides simple grammar mistakes such as improper usage of punctuation etc., some of your sentences don't even make any sense at all.

It seems like you're writing your posts in a rush while you want to shut me up.

And again, I don't understand what you're trying to mean in here:
if you dont understand that to get data, other people need to have the data to give. so the first important step is that people have the data.
so again "being able to get the data from other peers because they have the data to give is step one of that process"
If I don't understand that to get data, other people need to have the data to give what? Then you conclude that others need to have that “data”. Should I assume the data is the block chain? Then, of which process is this the step one?


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: franky1 on December 06, 2021, 10:48:45 PM
yes blockchain data.
this is not some science website that is asking for a thesis on particle manipulation related to the suns radiation.
of course when discussing bitcoin nodes/peers and the data they receive relating to the blockchain, that data means blockchain data(whole or part is no specific).

this whole topic is about blockchain data.
it does not mean data involving the spectral range of ultraviolet radiation from the sun. or any other data imaginable. it means on this website about bitcoin. in this category of technical discussion about bitcoin. in this topic of bitcoins blockchain.. yes data=blockchain data
1+1+1=3

now stop the social drama tactics.

if you are trying to play chicken(2) and egg(1)
the answer is egg.
step 0 duck+turkey create an egg(1) egg hatches and its a chicken(2) chicken lays an egg(1) another chicken comes out.. and so on. .. egg then chicken(rinse and repeat)

so to get data(2) someone needs to have data(1)... 1 comes before 2
pools create the data(1) peers get data(2) peer stores data(1) other peers get data(2)

if your pretending to be an egg farm but none of your chickens can produce eggs to feed customers. your not an egg farm.
just because you received an egg and verified for your own purposes that it is indeed a chicken by letting it hatch in your farm. but then not allowing that chicken to then produce eggs for others. does not make you a full egg farm


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: pooya87 on December 07, 2021, 03:53:23 AM
keeping the taint of movements of a utxos previous transactions is important for KYC and AML stuff
some services still want to avoid accepting transactions that have silkroad taint.
Wrong example because just having historical blocks isn't going to tell them the "taint" they also have to have implemented a taint analysis code which is not something merchants ever do. Instead they simply use a centralized third party.

Quote
yep being a full node is more then just one thing. correct and thank you for correcting yourself after your earlier defence that you think that being a full nude is only a selected features you thought were important to you.
I never said "only a selected feature" I said "full verification" makes a node full node not "full storage" and my reply here doesn't contradict that.

Your confusion and defensive behavior seems to be because you think I disagree with "pruned nodes contribute less to the network", but I'm not. I'm saying that you can't say they are not full nodes just because they don't store historical blocks.

Most important thing about pruned nodes is that (using the numbers you posted earlier) it is so much better to have a total of 14k nodes (3000 full archival nodes and 11k full pruned nodes) than to have 3000 full nodes in total and nothing else contributing to network.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: franky1 on December 07, 2021, 09:06:39 AM
(my last post on this discussion of less or full, as its a shame that i even have to debate such a basic concept to people that rebutt that basics dont need to be mentioned because it should be assumed they know)

to pooya:

you are right merchants need extra code to sieve through the block data to show the taint path. but its impossible to use their extra code to read the blocks to get the taint path of new incoming transactions unless... wait for it. they have the blocks of previous transactions to check the path.
you will be surprised that any decent merchant that cares enough about taint and customer transactions will actually want to be a full node. and have extra code ontop to do more beautiful things with the block data for their own independent purposes.(emphasis fully independent)

much like any decent business will have its own spreadsheet software and tax accountant software, instead of relying on 3rd party accountants

so try not to assume that all merchants are lite wallets and use third parties

after all the whole trust a third party thing you assume all merchants do, is not what bitcoin is about.
and its a shame that you think that users should trust third parties and use them because a decision to prune means they have to use third parties.
i think you shot yourself in the foot by admitting that those that do prune should and do trust third parties. it kinda rips apart your whole argument about 'full' if you are saying pruned nodes then use third parties for things a real full node can offer within the merchants own system..
.. doesnt really sound much of a rebuttal for your version of 'full' if you admit that they need to use third parties.
...
you did indirectly say selected features. because by your own admission you are not archiving so you have selectively chosen that certain features and services and options of a node are not to be used, hense no longer full
if i was to number all the features and the total was say 1037. defining FULL node. and your node wasnt offering all of them. even say 1036. then thats not full. thats 'less than'

by your own admission you say:
"you think I disagree with "pruned nodes contribute less to the network", but I'm not"
but then you pretend pruned nodes dont offer less because you believe pruned nodes are full.
less is not full.. less is less

if you went to a bar fo a pint of beer. there are 14 barman. the first you approach poured a pint. and then spilled 95% of it, but still gave you the glass of the remaining 5% froth. he can prove he poured a full pint into the glass. but is only offering you the 5% froth. would you accept that the glass is still a full pint. would you accept the barman offered you a full pint service.

in reality, you should think that the barman is useless if everytime you ask for a pint you only get 5% froth and go find another barman that can offer a full pint service. a fully trained barman that does contribute to the bar
in reality, you might even complain to someone that the barman is not contributing to the bars success.
in reality, you would NOT go on some site and leave a 5star review for the useless barman saying he is a fully skilled barman.
what if out of the other 13 barman you had the same problem with 10more barman not being fully trained..
and its impossible to get served by the 3 fully trained barmen because they are super busy serving regulars that came before you that know which barmen are fully trained. and also the queue of customers that got a bad service and are now queuing up with what little talent remains in the bar. making your experience miserable and slow, making you not want to have a pint altogether and making you not even want to be a fully trained barman in a future career as you see the pressure the 3 fully trained guys are under

as for your last statement that you think the scenario should be 'better to have 3k full 11k pruned rather than just 3k full'
actually no. because in the 3k full 11k pruned. those 3k are under more intense data pressure contributing to leachers. meaning the other peers within the 3k full nodes are getting slower speeds amongst themselves
and those leachers are not then contributing to help new full nodes get their IBD either.
the real scenario is to grow the full node population.


because yet again it needs to be mentioned. by trying to deceive people that being a prune is the same network contribution as being full, can and has made some full noders downgrade. thus instead of having 10k full nodes. its now ~3k.. because. wait for it.. people(like yourself) think that there is no harm and no loss of contribution to the network by being a pruner

sorry to inform you again. pruned nodes dont fully contribute to the network, they actually put more pressure on the ones that do.

so lets just stop the flip floppy changing the answer just to have a reason to debate.
make the decision
a. pruners dont help contribute towards the network and being pruned lessens what you can do/offer
b. remain in the disbelief that being pruned is full ability and full service offering to the peer network

i dont need the answer. the answer is for you to settle your own mind with.
less is not full

anyways, enjoy your 5% froth. just dont tell people your drinking a full pint. your a lightweight, cheap date


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: BlackHatCoiner on December 07, 2021, 09:36:24 AM
It's really a challenge to have a conversation with you, franky. I confess.

and its a shame that you think that users should trust third parties and use them because a decision to prune means they have to use third parties.
Running a pruned node means you don't have to trust anyone! As I said, whether you run a node who does keep the chain or not, you validate every single transaction you receive from your peers. The system works trustless-ly. What is this you dislike pruned nodes so much?

I said "full verification" makes a node full node not "full storage" and my reply here doesn't contradict that.
Huh? Don't pruned and full nodes differentiate only in the size of the storage they require? They both make a full verification.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: franky1 on December 07, 2021, 09:43:26 AM
verification of transactions is only one part of what a node does.
just because its the one part you only care about or the only part you want. does not mean its the only thing a node does
a full node does many many many other things.
and needs to perform and offer ALL those MANY things to be classed as full.

assuming a node just has to verify to be "full" is not full

if you think that being "full" is only about verification. then you need to redefined your assumption of what being a full node actually involves.

the real challenge seems you be certain peoples grasp of the concept of full, as oppose to 'less than, but pretends to be full'.

getting back to a technical discussion of this topic after the myriad of social drama discussion about less=full.

having the full blockchain means you can contribute to the network to offer(along with other things) not just Initial block download to peers, not just 'getdata' of random blocks, not just ability to sync up more than the last 2 days.
also theres are things like having the blockdata distributed so there are many copies in many countries to avoid any country 'pulling the plug'. having the data distributed to allow more choice of peer connections. having the data distributed so peers dont rely on a small cluster of attackable nodes

if the whole network (as the topic creators thoughts mention) only had the last month of transactions and a (UTXOSET) list of unspent transactions. where there was no taint or proof of that values initial creation from its original coin reward. then that puts the network at risk of UTXO virus attacks to rewrite sets. it puts the whole true accountability of all true coin at risk. along with other problems.

there is a reason that blockchains have become a success compared to spreadsheet accounting of 'live balance'. no one can just slip in a transaction with an old date and pretend its valid simply because they managed to send everyone their edited utxoset

its ok you you personally want to prune for any personal reason. just dont then pretend to be a full contributor to the network and dont tell others they are fully contributing when you talk them into pruning. just be honest and tell them you and them will be offering less then full network contribution and less than fulll features for you and them you take advantage of


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: pooya87 on December 07, 2021, 11:13:50 AM
so try not to assume that all merchants are lite wallets and use third parties
I made no such assumption. Many merchants run their own nodes (full or pruned) and many go through third parties but almost all those who need some sort of "taint analysis" rely on a third party (like chainanalysis) because this is despite their own preference like an exchange that is forced by the regulators to implement such nonsense.
Implementing such a system themselves is a burden and would cost more than using the third party not to mention that the responsibility would be with that third party not the user which is favorible.

Quote
after all the whole trust a third party thing you assume all merchants do, is not what bitcoin is about.
That's about receiving bitcoin through third parties not some regulation about KYC/AML needing chain analysis!

Quote
and its a shame that you think that users should trust third parties and use them because a decision to prune means they have to use third parties.
I made no such claims.
Again you claimed that merchants need to run archival nodes for accounting, AML and KYC and I said for those things that already have centralization involved they use third parties.

Quote
by your own admission you say:
"you think I disagree with "pruned nodes contribute less to the network", but I'm not"
but then you pretend pruned nodes dont offer less because you believe pruned nodes are full.
less is not full.. less is less
You are very confused about all concepts. A full node has a simple definition, a client that verifies everything. That's all. It doesn't have to do anything else. It can discard the entire blockchain, it can refuse to relay blocks or transactions, it can refuse to provide any services including replying to getheaders message which every client including SPVs could reply to but none of that changes the fact that it is a full node, it just means this particular client is not contributing to the network.

Quote
by trying to deceive people that being a prune is the same network contribution as being full,
I made no such claims. I clearly stated (and you quoted it) that pruned nodes contribute less to the network but they still contribute. You seem to be having a hard time keeping your thoughts in one place!


I said "full verification" makes a node full node not "full storage" and my reply here doesn't contradict that.
Huh? Don't pruned and full nodes differentiate only in the size of the storage they require? They both make a full verification.
That's exactly my argument. When your client downloads and verifies everything from block 0 that makes it a full verifying node. Whether your client discards the blockchain later (and stores less blocks) doesn't change that.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: PrimeNumber7 on December 07, 2021, 11:36:44 AM
my premiss is this
if you just want to validate your own transactions and you dont care about being part of the full node network of peers for the benefit of the bitcoin network universe. then fine you prune or use a lite wallet, whatever you choose. just know and understand that your not part of the full node network.

understand that by you saving your bandwidth/hard drive by not being a seeder means your not helping the real seeders.. and acept that your ok with that 'less than' status, because it fits your lifestyle

but if you want to be a full node then be a full node.
none of this wishy washy 'everyones a full node' pish posh
I guess you can call a pruned node whatever you want. However, a pruned node provides all the same security that a full node provides, and that is what is important to the end-user. The same is true if a non-pruned node disregards signature data after it has validated the signature of a transaction.

the network needs to have a strong amount of actual fullnodes
not a 3:11 ratio of full:lesthan group thinking they are all offering a full decentralised service for the network
without even knowing that there is a difference
Unfortunately, we don't live in a utopia, and users will prioritize their limited resources to things that are important to them. If someone does not have the resources to store all blocks, or to send all block/transaction data to the internet 8+ times, they should not be forced into doing so.

Even if a centralized service were to "sell" the ability to download the entire blockchain for new nodes, a node would be able to validate the entire blockchain, and know if they are receiving the correct version of the blockchain because they would be validating all blocks, including the POW, and would be able to validate the POW of new blocks they receive from other nodes. It is not possible to fake POW of old blocks.

fullnodes have the entire blockchain all the way upto the latest block. where all blockdata is included and all transactions within are validated. that way chain re-orgs cant happen easily/at all
A pruned node keeping 5 GB of the most recent block data would be storing a minimum of 1,250 blocks if they kept the block and signature data. This works out to approximately 8.5 days worth of blocks with no difficulty growth. A reorg anywhere near that size (even 1% of that size) would signal very serious problems, that far outweigh any potential issue about not having enough nodes.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: franky1 on December 07, 2021, 11:43:44 AM
seems pooya is stuck on the 'only verification'=full features..

That's exactly my argument. When your client downloads and verifies everything from block 0 that makes it a full verifying node. Whether your client discards the blockchain later (and stores less blocks) doesn't change that.

actually although you think you have a full verified UTXO set of all unspents from block 0 to say block 712k.. can you verify that today? right now! without requesting a new IBD to compare it to???
(answer is no)

heres the thing. you might have been online for 12 years and stayed synced and decided to prune say 6 months ago. you might have just come online 6 months ago and IBD and then pruned that month.
but here is the question for you.
what if your computer glitched 4 months ago or a hacker got in and added his own false utxo to your utxo set.

can you now with just your UTXO set verify today that no one slipped in a utxo into your set.
no. you cant
can you prove that a tx in your utxo set is today valid, when some chain re-org happens.. no you cant

your UTXO set is only as good as the minute you verify it. anything after that, the utxo set by itself cant be fully trusted. loss of data, virus, hacks. and boom your utxoset is compromised

yet having the blockdata to support it allows you to seek the utxo from the set(like reading an index of a book). and then compare it to the blockchain.
a utxo set is only useful to save having to find the tx via reading all blocks every time a transaction appears.

its just like an index of a book to save you having to read the whole book every time just to find a reference.
a utxo set is not good for verified accounting. its not good for security. its not good for proving the transaction has a real utxo.. because your utxo set can be compromised at any time and there is no way you can tell

again there is a reason why blockchains become popular compared to spreadsheet balance accounting


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: PrimeNumber7 on December 07, 2021, 11:59:19 AM
heres the thing. you might have been online for 12 years and stayed synced and decided to prune say 6 months ago. you might have just come online 6 months ago and IBD and then pruned that month.
but here is the question for you.
what if your computer glitched 4 months ago or a hacker got in and added his own false utxo to your utxo set.

can you now with just your UTXO set verify today that no one slipped in a utxo into your set.
no. you cant
If someone is able to maliciously add a UTXO that is not in the UTXO set, they have potentially (and likely) added malware to your computer. You would not be able to trust any output your computer provides. If your computer is infected with malware, validating the entire blockchain (again) is not going to show you that you have received a valid transaction.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: franky1 on December 07, 2021, 11:59:24 AM
If someone does not have the resources to store all blocks, or to send all block/transaction data to the internet 8+ times, they should not be forced into doing so.

its not about being forced to.

lets use a mcdonald employee analogy. maybe something more understandable for some
its about not pretending you deserve a 5star gold badge, when you only offer a 1 star service. its about being honest with yourself and others that if you choose to only offer a 1 star service, accept that fact that you are a 1 star service member. be ok that you are a one star member. and stop trying to pretend your 5star contributor.

it doesnt matter if you make 1000 more posts saying the 1star training is all that is needed to be 5star.. reality is there are more things then the 1 star offering involved to be given the 5star badge.
yes 5star contributors also do 1star tasks... along with many other tasks. but that does not make a 1 star member a 5 star member just because "but 5star guys does my job too'. the 1star member still needs to do more then a 1 star task to be promoted. ignoring all the other jobs a 5star member does, pretending they are insignificant. does not earn you a promotion
just because your a burger flipper and then you see the manager also flips burgers. does not then mean you get to call yourself a manager because you both flip burgers. the manager does other tasks too. which is why he is the manager


its about stop deceiving others that your a full service 5 star contributor. its about realising what you are not contributing to, and thus accepting that lesser position and the hazards to you and others  for not being able to do it all.

its simply that you can be a pruner. but call yourself a pruner not a 'full node'.
and no, dont demote a full node description to being 'just verification'. because a full node is not just about verification. there is alot more too it. and if you are still unsure of all the features.. learn

there are many many many things that involve being a full node. not just that at some point in time you verified something.. because months later that set cant be trusted
it has no UTXOset hash to compare to other peers UTXOset to verify the set has not been edited

nodes with nothing more then the lastest 288 blocks and a utxo set cannot in their true heart be 100% sure that their utxo set will never be compromised. they cannot also truly know for sure if it ever was. nor can they help other nodes.
maybe people need to revise and study and research what blockchains offer that spreadsheet accounting doesnt

its not about forcing people to be full nodes. its about getting pruners to accept their limited ability due to their own choice to limit what they do.

sorry but pruners do not get a 5star gold badge. they have not earned the promotion.

again to try bringing the discussion back on topic to the topic creators question
having a whole network of nodes with just 288 latest blocks and then a dataset of unspent values. puts the network at big risk. blockchains were invented for a good reason, doing something that spreadsheet balance accounting cant
it doesnt matter if at some point in the past you checked the balances were real. once verified if you are only storing a balance sheet of what you believe is still unspent. that sheet can become compromised and you have nothing on your system to check it against.
a dataset of unspents should never be used as "same security as blockchains" because it lacks the evidence of origins to ensure no compromise happened. a dataset of unspents should only be used as a quick index to find the real unspent on the actual blockchain. to then double verify the unspent is real when new transactions appear.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: tromp on December 07, 2021, 05:50:53 PM
seems pooya is stuck on the 'only verification'=full features..

Yes, pooya, along with most others in this discussion, as well as the vast majority of bitcoin experts,
is stuck on the notion that full node is short for fully verifying node.

You are one of a small minority of people that have miraculously overlooked this established terminology.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: BlackHatCoiner on December 07, 2021, 07:12:54 PM
actually although you think you have a full verified UTXO set of all unspents from block 0 to say block 712k.. can you verify that today? right now! without requesting a new IBD to compare it to???
Why should you verify an already-verified set of outputs?

what if your computer glitched 4 months ago or a hacker got in and added his own false utxo to your utxo set.
That has nothing to do with pruning. Alright, let's take it in another way:  What if a hacker replaced your UTXO with his UTXO while you're running a full node? Once you had verified the outputs you'd have no way of knowing if your verified outputs changed.

The only difference is that full node does the IBD once.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: franky1 on December 07, 2021, 09:27:55 PM
seems pooya is stuck on the 'only verification'=full features..

Yes, pooya, along with most others in this discussion, as well as the vast majority of bitcoin experts,
is stuck on the notion that full node is short for fully verifying node.

You are one of a small minority of people that have miraculously overlooked this established terminology.

ok there we have it.
it looks like tromp took a secret vote and elected bitcoin full nodes EG 'core' as just a litewallet, thats job is nothing to do with decentralising the network by offering IBD, getdata to peers.
(facepalm)

[sarcasm]
guess thats case closed then..
core devs dont need to code any parts of core relating to archiving the blockchain or providing peers with blocks older than the latest 288blocks

guess tromps community decided. bitcoin does not rely on blockchains. death to the un-needed blockchain i guess.. tromp has decided that blockchains are not important.

praise be tromp, the election committee director, the master of what core is responsible for, praise be
[/sarcasm]

ill leave tromp with one question.
if full means only certain features. then what terminology is there for a node that offers not just verification, but the full bitcoin offering of features such as verification AND archiving AND full peers services....

here is the funny thing. 'full node' was a term that existed before pruning existed. so pretending that pruning is full, is a false idea and a rewrite of history

lets await tromp to take another secret poll and come up with a new terminology
(isnt it weird that in my first post on this topic i said that too many people like to argue buzzwords, rewriting reality and thus making the buzzwords meaningless in their definition)

what if your computer glitched 4 months ago or a hacker got in and added his own false utxo to your utxo set.
That has nothing to do with pruning. Alright, let's take it in another way:  What if a hacker replaced your UTXO with his UTXO while you're running a full node? Once you had verified the outputs you'd have no way of knowing if your verified outputs changed.

if you have the blockchain. (important) you can check utxo to find its block, then its blockhash, and compare against the block contents. find the transaction in question, see it exists. then check the entire chain of blockhashes match up to what they suppose to be, and yep. you can see if there were edits or not.
EG edit one transaction from 2010 edits that blocks hash and all blockhashes there after meaning the latest blocks hash wont be the same as your edited latest blocks hash

you cant do all this if you prune, unless you continually prompt a third party to re-send you all the data again
(isnt it weird that one of the first rebuttals to me was that certain things didnt need explaining because it was assumed they knew the basics already.. guess not)

anyways, seems this topic is going no where fast where buzzwords dont mean what they intend because half a job is a full job, as voted.  and blockchains are not important and not part of the protocol that fullnodes are involved in. as voted by the established community (facepalm)

bored now. moving on


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: zbig001 on December 08, 2021, 10:29:20 AM
Paradoxically, pruning and validating only the current blocks may be sufficient for security, but only if it doesn't become too common.

It is better not to provoke the evolution of the protocol towards increasing scalability, which would not only threaten decentralization, but also security.
There can always be a bug that validators cannot detect and that will not be disclosed until a full chain audit by inquisitive, independent researchers.
As was the case with the Stellar coin once
https://coinmetrics.substack.com/p/coin-metrics-state-of-the-network-37f


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: n0nce on December 08, 2021, 01:45:05 PM
seems pooya is stuck on the 'only verification'=full features..
Here's the thing. In Bitcoin wiki (https://en.bitcoin.it/wiki/Full_node) and pretty much anywhere else, full node is used interchangeably with or even defined as fully verifying node.
You're actually stuck on believing 'full' means 'full features'; but that's not the definition. 'Full' in this context is more a short-form for 'fully verifying' instead of meaning 'all features'. By your definition, I could say, if you don't run Electrum and Lightning on your node and don't have a giant txindex, you're not providing 'full functionality' to the network. So as you see, such a definition would be very vague: what is everything? This is not clear-cut. But anyway: full = fully verifying. It's defined like that, after all.

Nodes that fully verify all of the rules of Bitcoin are called full nodes.

For the record, I run multiple non-pruned full nodes with Electrum and Lightning, so you're not gonna tell me to 'be happy with your mangled non-full node', please ;)



what if your computer glitched 4 months ago or a hacker got in and added his own false utxo to your utxo set.
That has nothing to do with pruning. Alright, let's take it in another way:  What if a hacker replaced your UTXO with his UTXO while you're running a full node? Once you had verified the outputs you'd have no way of knowing if your verified outputs changed.
Didn't you know franky reindexes his blockchain folder every day? :P No really, it makes no sense. Verified once, it's verified, that's it LOL; similar as if someone hashed the same thing 1000x and expected the result to change after some while.



if you have the blockchain. (important) you can check utxo to find its block, then its blockhash, and compare against the block contents. find the transaction in question, see it exists. then check the entire chain of blockhashes match up to what they suppose to be, and yep. you can see if there were edits or not.
EG edit one transaction from 2010 edits that blocks hash and all blockhashes there after meaning the latest blocks hash wont be the same as your edited latest blocks hash

you cant do all this if you prune, unless you continually prompt a third party to re-send you all the data again
(isnt it weird that one of the first rebuttals to me was that certain things didnt need explaining because it was assumed they knew the basics already.. guess not)
Replying to the underlined: actually this is exactly what you do during the IBD of a full node (pruned or not - no difference). You fully verify every single utxo from the beginning to the current state. After it's verified by you, you're good. There is no benefit in repeating it later on. That's why the old blocks can be discarded after verification. Of course, the big downside of not archiving all blocks is you can't seed to new nodes. But the direct benefit (security, privacy, ...) to you as a user is exactly the same if you prune or not.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: franky1 on December 08, 2021, 03:37:04 PM
i understand that certain self-assumed 'community' electorates who feel empowered that they are the voice of all, think that full node is an invented term after pruning became a thing and refers to nodes that only verify.. just to ego boost that their less than offering is still felt as full offering. although they lack responding by mentioning the term of what a node is called that does do all the jobs(true full).

but by the looks of their forum activity numbers they have not been around since the days of proper full nodes terminology creation.. you know the days before full nodes even had the ability to prune and where by reality was that archiving by default and un-optionable was actually part of being a full node.

i understand in recent years the paradigm has changed where pruning is now the default and archiving is now optional.. . but that does not change what full meant right from the start(when full nodes meant full job).

as for quoting wiki.. you might want to read a little more then your prefered snippet
Quote
By default full nodes are inefficient in that they download each new transaction at least twice, and they store the entire block chain (more than 165 GB as of 20180214) forever, even though only the unspent transaction outputs (<2 GB) are required. Performance can improved by enabling -blocksonly mode and enabling pruning

again even the wiki is a few month out of date because now the default is that prune is ticked and to now archive, the option needs to be unticked. so as you can see things have changed where definitions have moved the goal post RECENTLY

it only became efficient to not archive SINCE pruning became an option. because full nodes before pruning was even a thing, was default store entire blockchain because there was no choice within a full node not to archive.

so yea. your basically saying the default definition of storing the blockchain has been re-defined since pruned was invented. and so you now want to say and pretend its always been a less=full scenario even when real history shows that 'full' was in its original terminology full=full

..
with all that said. if you want to be a pruned node. thats ok for you. just be honest that you are a pruned node and happy being one. no need for deception. no need to boost your level of involvement while not actually being able to perform the higher level of involvement. just be honest and happy with your choice to be limited part of the networks security and decentralisation. then be happy to learn what that limited level of network support your node is or is not helping with. and not assume you are fully supporting the network in full, by simply redefining your version of 'full'


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: n0nce on December 08, 2021, 06:50:43 PM
but by the looks of their forum activity numbers they have not been around since the days of proper full nodes terminology creation..
If you're referring to my account age, maybe consider a person can be in Bitcoin many years before creating a Bitcoin Talk account.

i understand in recent years the paradigm has changed where pruning is now the default and archiving is now optional.. . but that does not change what full meant right from the start(when full nodes meant full job).

[..]
now the default is that prune is ticked and to now archive, the option needs to be unticked. so as you can see things have changed where definitions have moved the goal post RECENTLY
Oh, you're such a self-proclaimed know-it-all but use Bitcoin Core only via GUI? I recently found out that the GUI changed; however never used it except to do the IBD on a fast computer once. All my nodes run on Linux CLI as a service (as they should for best uptime and best serve the network), and in CLI mode it's still default not to prune. Pruning is deliberately for the 'n00bs' that want to run Core on their laptop and similar. So it makes sense that for personal machines which aren't always on and such, people might be more disk space restricted, whereas someone setting up a dedicated Bitcoin node (who uses CLI) most probably has enough disk space allocated for running it in full archival mode.

it only became efficient to not archive SINCE pruning became an option. because full nodes before pruning was even a thing, was default store entire blockchain because there was no choice within a full node not to archive.
Even before pruning you could be not fully verifying or fully verifying. That's why the terminology exists longer than the pruned mode itself.

so yea. your basically saying the default definition of storing the blockchain has been re-defined since pruned was invented. and so you now want to say and pretend its always been a less=full scenario even when real history shows that 'full' was in its original terminology full=full
Now tell me: what is 'full' in your opinion? Fully archiving? What about Electrum? How can something that does not have all the bells and whistles be considered full? So in your terms, a full node must at least contain: Core with full archive + Electrum + Lightning; now when we get into LN you have to define how many and how large channels you need to be considered 'full'. I could argue that you're not a 'full node' if you don't offer me to transmit 1BTC to another person through your channels. We can go down this hole further and further; such as requiring you need to offer all services both through Tor and clearnet etc etc.

In the end, your definition is completely wrong (and has always been! full refers to the validation of the blockchain...), and I never heard anyone use your definition in almost 10 years now that I speak to people about BTC. It would never work anyway; because the definition of what is 'everything' (which seems to be your definition of 'full') always changes e.g. with the introduction of LN. There may be people considering you not running 'everything' if you're not running the sidechains as well, for example.


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: franky1 on December 08, 2021, 09:53:12 PM
no its not about the noob GUI..
its about that since pruning even became an thing entirely. the whole definition has changed.
take cores system requirements of versions pre prune invention date. hard drive specification for running a core full node was excess of blockchain size. now since pruning that 'min system requirement' has altered.

Even before pruning you could be not fully verifying or fully verifying. That's why the terminology exists longer than the pruned mode itself.
..
Now tell me: what is 'full' in your opinion? Fully archiving? What about Electrum? How can something that does not have all the bells and whistles be considered full?

electrum? oh please!(facepalm). everyone refers to electrum as lite wallet/spv.
(just google 'electrum litewallet' or google 'electrum spv' and you will see millions of references)
also
other wallets and nodes that are not fully verifying or not archiving are classed as lite/spv/thin

seriously.. you really thought electrum was a full node?(facepalm)

not every wallet/node should be classed as a full just to give noobs the deceptive and false belief they are helping, when they are not fully helping. its ok to use bitcoin without being a full helper. be ok that you use bitcoin without fully helping the network if you are choosing your lifestyle doesnt require/want to help.


ok. this topic has derailed into debate that pruners and litewallet users want to call them selves full noders just to have some namebadge of importance, like they are the top guys and supporting the network... shame though that reality has shown they actually chosen not to be supporting the network.
hey its ok that you chose not to support the network. just be honest with yourself about your choice and be happy that your not.

anyways yea this topic has derailed into some social debate of redefining old terms to fit noobs sentiment of false level of network support. by trying to class bitcoins blockchain as non-important and insignificant..
such a shame that soo many people think blockchains are insignificant.

im moving on. this topic is dead.

edit:
to answer below.. as he still cant quite grasp it.
"electrum server" is not a stand alone software that works byitself. it requires also using bitcoind.

even when electrum server is running but not doing all the "backbone of the network" stuff. its not a full node
emphasis backbone of the network.
(unless you think theymos is wrong wrong too.. and you as a lite wallet user know better than theymos)

as for previous stuff about nodes that support alternative networks all in one. those are called 'master nodes'

and i never wanted or tried to claim myself as important. im just shocked that some people want to pretend there is a social hierarchy of non-devs where they deserve to be top guy amungst the dev group, while not supporting the backbone of the network.
my premiss is not to be a top guy or a 'buzzword definition election committees director'. im just calling out reality to people that are deceived or have been deceived. its called just spelling out the obvious. it requires no rank system.

maybe mentioning 'backbone of the network' 3 times isnt subtle enough hint about the wiki definition


Title: Re: Blockchain is 360GB and growing, can it be consolidated?
Post by: n0nce on December 08, 2021, 10:16:04 PM
~
Dude, I'm obviously talking about running Electrum server (ElectrumX or electrs). Running either of those allows people to use SPV wallets without trusting the preset server; thus providing more services to the Bitcoin network. I was just showing two examples (Electrum & Lightning) of software that some may define as necessary to achieve 'full functionality'. This was all just to show that a definition of 'full node = full functionality' is super vague, since there is no hard set definition of 'full functionality' itself.

All in all, this whole argument is pointless, since it's obvious (and has always been) that the definition of full node is fully verifying node.

For example, take this page from December 2014:
Nodes that fully enforce all of the rules of Bitcoin are called full nodes.

Pruning was introduced in Bitcoin Core 0.11.0 (https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.11.0.md#block-file-pruning). The release notes for 0.11.0 were written in August 2015, but I can't find that release in GitHub releases (https://github.com/bitcoin/bitcoin/releases?page=4). The 0.11.1 release came out in November 2016 (https://github.com/bitcoin/bitcoin/releases/tag/v0.11.1).

So all in all, the definition was written down and accepted by everyone for 7 years now; while pruning only exists for effectively 5 years. You can even see in the Wiki diff, that that sentence has barely changed and its meaning is retained to this day. (https://en.bitcoin.it/w/index.php?title=Full_node&diff=67995&oldid=53494) You even find a more thorough definition in the second element of the diff; where we can see they just removed the word 'core'.

https://i.postimg.cc/NfrjWmhq/image.png

Also what about this?
pruners and litewallet users want to call them selves full noders just to have so namebadge like they are the top guys
You're not a 'top guy' just because you run a full (pruned or not) node :D