Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: takuma sato on October 12, 2022, 04:09:08 AM



Title: Is anyone working in a "pruned download"?
Post by: takuma sato on October 12, 2022, 04:09:08 AM
What I mean by this is that, instead of needing to download the whole blockchain before you can enable the pruned node, that you download and delete blocks as you go. I understand the technological limitations of this and I don't see how this is possible, but I've seen things being implemented that at first seemed impossible. Would it be possible to establish an option that sets a safe amount of blocks and keep this threshold of blocks and download and delete as you go?

Of course being a purist myself I only consider doing real Bitcoin with a full downloaded blockchain, I don't even enable pruned mode, but I was wondering if this is something someone is actively researching at all.


Title: Re: Is anyone working in a "pruned download"?
Post by: nc50lc on October 12, 2022, 06:13:09 AM
What I mean by this is that, instead of needing to download the whole blockchain before you can enable the pruned node, that you download and delete blocks as you go.
-snip-
Would it be possible to establish an option that sets a safe amount of blocks and keep this threshold of blocks and download and delete as you go?
I didn't get the question, quite.
But "prune block storage to n MiB" is working similar to this, your node doesn't have to download the whole blockchain before you can enable pruning.
You can start Bitcoin Core with clean datadir with prune setting enabled and it will download and delete the older blocks after reaching the set size.

Or do you mean: A way to skip the older blocks and just download the "threshold" (latest blocks)?
If so; go to this issue and read its comments: github.com/bitcoin/bitcoin/issues/15605 (http://github.com/bitcoin/bitcoin/issues/15605)


Title: Re: Is anyone working in a "pruned download"?
Post by: PawGo on October 12, 2022, 06:42:23 AM
Of course being a purist myself I only consider doing real Bitcoin with a full downloaded blockchain, I don't even enable pruned mode, but I was wondering if this is something someone is actively researching at all.

I think it is good solution if you just want to use Bitcoin Core as a private wallet and if you do not care about about running your machine 24/7. That way you limit the disk space needed and still your wallet is fully functional. That kind of configuration could be replaced by SPV wallet, but using Core you verify blocks yourself and you do not need to rely on 3rd pary solution.


Title: Re: Is anyone working in a "pruned download"?
Post by: NotATether on October 12, 2022, 06:57:41 AM
You can't do that without introducing a check to make sure the UTXO set is not tampered with while verifying the transactions in newer blocks, as the older blocks that can cross-reference the UTXO are no longer on the disk.


Title: Re: Is anyone working in a "pruned download"?
Post by: DaveF on October 12, 2022, 07:01:11 PM
Rant.....
I would really like to see the entire concept of the pruned node removed. Either run a node or don't. There seem to be more and more people who want to have a node running on drives that don't have enough space that wind up going the pruned node route and then wonder why they can't import a private key or are having some other issue that comes back to the fact that you can't go back to a full node without re-downloading everything. Spinning 1TB drives are under $40.00 as are 512GB SSD it's no longer that big a deal.
...end rant

-Dave


Title: Re: Is anyone working in a "pruned download"?
Post by: pooya87 on October 13, 2022, 05:38:14 AM
Spinning 1TB drives are under $40.00 as are 512GB SSD it's no longer that big a deal.
The thing is people don't buy computers or their hardware to be dedicated all to bitcoin. For example my 4 TB HDD space is used for my files and it doesn't have that much space left for blockchain, same as my SDD which I use for 2 operating systems and their installed programs and their related files which take up a lot of space. So not much space is left for me to dedicate to bitcoin's full blockchain, this is where pruned mode is very useful.


Title: Re: Is anyone working in a "pruned download"?
Post by: NeuroticFish on October 13, 2022, 05:46:15 AM
Spinning 1TB drives are under $40.00 as are 512GB SSD it's no longer that big a deal.
The thing is people don't buy computers or their hardware to be dedicated all to bitcoin. For example my 4 TB HDD space is used for my files and it doesn't have that much space left for blockchain, same as my SDD which I use for 2 operating systems and their installed programs and their related files which take up a lot of space. So not much space is left for me to dedicate to bitcoin's full blockchain, this is where pruned mode is very useful.

You are both right. Pruning is very useful for those who know what they're doing, but, on the other hand, it gives (wild) ideas to people who have missed or misunderstood the "don't trust, verify" part of bitcoin's blockchain.

What I mean by this is that, instead of needing to download the whole blockchain before you can enable the pruned node

Afaik, you can enable pruning also before starting IBD. However, this doesn't means you won't download the whole blockchain, you will still download it all. And if you have an old wallet and forgot to put it in place before the sync (in pruned mode), you may need to download the whole blockchain again for the new wallet. Of course, only a small part will be stored; but is DaveF said, storage is cheap; even internet may go cheap, but our time usually isn't.


Title: Re: Is anyone working in a "pruned download"?
Post by: Little Mouse on October 17, 2022, 06:21:18 AM
There must be something open source application which would be able to verify the downloaded blockchain history file and tell whether anything is tampered with. I haven't seen anything like that. You can't simply trust other.

which obviously require you to trust that snapshot.

[1] https://ezno.de/packages/#fast-sync (https://ezno.de/packages/#fast-sync)
That's where the problem is. I don't know how much risky it is, can someone explain the possible manipulation one can do if I trust their file? Last year, I wanted to try such a method from my friend but later decided not to go with although there was nothing wrong with that.

or are having some other issue that comes back to the fact that you can't go back to a full node without re-downloading everything.
That's it. It doesn't make sense to download the blockchain again and again. I got tired when I first tried lol.


Title: Re: Is anyone working in a "pruned download"?
Post by: NeuroticFish on October 17, 2022, 09:11:13 AM
That's it. It doesn't make sense to download the blockchain again and again. I got tired when I first tried lol.

Pruned node makes sense to
* those who have an old wallet and want to get their money off without getting into more complex things (getting private keys out and so on)
* to the average Joe who will use only one wallet

Whoever wants to play a little more and do some tests, either buys a HDD and keeps the blockchain, either goes for a SPV wallet.


Title: Re: Is anyone working in a "pruned download"?
Post by: Wind_FURY on October 17, 2022, 11:08:04 AM
Rant.....
I would really like to see the entire concept of the pruned node removed. Either run a node or don't. There seem to be more and more people who want to have a node running on drives that don't have enough space that wind up going the pruned node route and then wonder why they can't import a private key or are having some other issue that comes back to the fact that you can't go back to a full node without re-downloading everything. Spinning 1TB drives are under $40.00 as are 512GB SSD it's no longer that big a deal.
...end rant

-Dave


I don't know if I can agree, but yeah, the more actual archive nodes out there, the better for IDB for those new users. Especially for a decentralized network, the Bitcoin Blockchain must be made as redundant as possible. BUT pruned nodes also help Bitcoin to decentralize further by giving those computers with low available disk space to join the network. Sometimes office/home/school computers can also double as a  Bitcoin node, no?


Title: Re: Is anyone working in a "pruned download"?
Post by: BlackHatCoiner on October 17, 2022, 03:52:16 PM
I would really like to see the entire concept of the pruned node removed.
That isn't going to happen, no matter if you or the developers want it. Pruning the chain is possible, and if something's possible it can and will happen.

Spinning 1TB drives are under $40.00 as are 512GB SSD it's no longer that big a deal.
But it's sometimes unnecessary. A person who firstly acquires bitcoin, and only wants to be able to spend it privately and securely, doesn't have to hold ~750,000 blocks. Even if it costs him little money. It's storage waste.

Sometimes office/home/school computers can also double as a  Bitcoin node, no?
There's no reason a school runs a Bitcoin node, unless they teach it. Businesses and offices, and any other entity that uses bitcoin should be running one (preferably non-pruned (https://bitcointalk.org/index.php?topic=5414284.msg60980561#msg60980561)).


Title: Re: Is anyone working in a "pruned download"?
Post by: NotATether on October 17, 2022, 04:15:11 PM
Spinning 1TB drives are under $40.00 as are 512GB SSD it's no longer that big a deal.
But it's sometimes unnecessary. A person who firstly acquires bitcoin, and only wants to be able to spend it privately and securely, doesn't have to hold ~750,000 blocks. Even if it costs him little money. It's storage waste.

Network bandwidth is more scarce than disk space, so we should be optimizing pruned nodes more in that direction instead of disk space.

Actually, we already did: It's called maxuploadtarget.


Title: Re: Is anyone working in a "pruned download"?
Post by: BlackHatCoiner on October 17, 2022, 04:27:02 PM
Network bandwidth is more scarce than disk space
From an individual's perspective, or from a community's perspective? A Bitcoin user's priority should be to protect himself against surveillance and keep his money safe. Helping the network bandwidth-wise is secondary.

Actually, we already did: It's called maxuploadtarget.
Wait, what does this matter? A pruned node doesn't upload any blocks.


Title: Re: Is anyone working in a "pruned download"?
Post by: NotATether on October 17, 2022, 05:28:22 PM
Network bandwidth is more scarce than disk space
From an individual's perspective, or from a community's perspective? A Bitcoin user's priority should be to protect himself against surveillance and keep his money safe. Helping the network bandwidth-wise is secondary.

This is from an individual perspective, because Comcast/Spectrum/Three/<household ISP> limit upload speed and capacity so thoroughly that it becomes impossible for people to complete verification in a reasonable time (a few days to a week or two).

The network as a whole does not benefit from reduced bandwidth usage, it's those who run nodes from their house who stand to gain from such a reduction in bandwidth.

Let's not forget that for IBD, the network usage is proportional to the blockchain aka. disk size. This is where I'd think alternate clients would come in handy as I don't see Core ever implementing this. Perhaps Libbitcoin Lite?  ;)

Actually, we already did: It's called maxuploadtarget.
Wait, what does this matter? A pruned node doesn't upload any blocks.

Unless I'm mistaken, I read the annotated bitcoin.conf file generator (https://jlopp.github.io/bitcoin-core-config-generator/) and it said: "Tries to keep outbound traffic under the given target (in MiB per 24h), 0 = no limit."

Nodes can upload both blocks and transactions.


Title: Re: Is anyone working in a "pruned download"?
Post by: Wind_FURY on October 18, 2022, 06:38:32 AM

Sometimes office/home/school computers can also double as a  Bitcoin node, no?

There's no reason a school runs a Bitcoin node, unless they taught it. Businesses and offices, and any other entity that uses bitcoin should be running one (preferably non-pruned (https://bitcointalk.org/index.php?topic=5414284.msg60980561#msg60980561)).


Academia can run a node for any purpose, whether financial, educational, or experimental. But my point is running a node by using computers not "officially" supposed to run a node. If a rebellious school university employee secretly installed Bitcoin nodes in the computers at work for example. I believe pruned nodes will be quite useful for that. I know it's a bad example, but you get the point. It helps the network to give users the option how much disk space they want to allocate for their nodes.


Title: Re: Is anyone working in a "pruned download"?
Post by: BlackHatCoiner on October 18, 2022, 08:20:16 AM
This is from an individual perspective, because Comcast/Spectrum/Three/<household ISP> limit upload speed and capacity so thoroughly that it becomes impossible for people to complete verification in a reasonable time (a few days to a week or two).
Which, how's relevant again? Using extra disk space isn't going to make their verification faster. It can make their possible and probable future re-syncing faster though.

The network as a whole does not benefit from reduced bandwidth usage, it's those who run nodes from their house who stand to gain from such a reduction in bandwidth.
Granted. But, as I said, if someone doesn't want to spend a few hundreds of gigabytes for these extra benefits, helping the network bandwidth-wise is a very indirect incentive. (And if he does it, it's mostly going to be for his own benefit)

Nodes can upload both blocks and transactions.
My bad. Since BIP159 (https://github.com/bitcoin/bips/blob/master/bip-0159.mediawiki), pruned nodes can advertise themselves as having these few blocks:
Abstract

Define a service bit that allow pruned peers to signal their limited services

Before that, it was disabled by default (since 0.11.0):
For now, block pruning disables block relay. In the future, nodes with block pruning will at a minimum relay “new” blocks, meaning blocks that extend their active chain.


Title: Re: Is anyone working in a "pruned download"?
Post by: NotATether on October 18, 2022, 11:12:58 AM
Nodes can upload both blocks and transactions.
My bad. Since BIP159 (https://github.com/bitcoin/bips/blob/master/bip-0159.mediawiki), pruned nodes can advertise themselves as having these few blocks:
Abstract

Define a service bit that allow pruned peers to signal their limited services

NODE_NETWORK_LIMITED is is used in practice by nodes, but then why is this particular BIP still marked "Draft"? Might be a good question to ask the mailing list and in particular Luke Dashjr (as Kalle doesn't seem to be online these days).


Title: Re: Is anyone working in a "pruned download"?
Post by: DaveF on October 18, 2022, 02:21:54 PM
Spinning 1TB drives are under $40.00 as are 512GB SSD it's no longer that big a deal.
But it's sometimes unnecessary. A person who firstly acquires bitcoin, and only wants to be able to spend it privately and securely, doesn't have to hold ~750,000 blocks. Even if it costs him little money. It's storage waste.

I guess I look at it as most people who are just getting BTC to use are more likely to just run a SPV client. Keep in mind that if your PC is not on 24/7 and you do not always keep the wallet open when it is on then you will probably have to wait a while after opening up your wallet at that point to be able to spend your coins.

I don't know if I can agree, but yeah, the more actual archive nodes out there, the better for IDB for those new users. Especially for a decentralized network, the Bitcoin Blockchain must be made as redundant as possible. BUT pruned nodes also help Bitcoin to decentralize further by giving those computers with low available disk space to join the network. Sometimes office/home/school computers can also double as a  Bitcoin node, no?

Home yes, school possibly, office only if you own the company. Putting non work software on your work PC is a good way to get fired. Also, unless you are remote work and connecting to an office PC, more and more offices are shutting down PCs and everything else at night. It's a bit more secure and saves power.

-Dave



Title: Re: Is anyone working in a "pruned download"?
Post by: dkbit98 on October 18, 2022, 05:42:44 PM
I guess I look at it as most people who are just getting BTC to use are more likely to just run a SPV client. Keep in mind that if your PC is not on 24/7 and you do not always keep the wallet open when it is on then you will probably have to wait a while after opening up your wallet at that point to be able to spend your coins.
It's not that hard to run a node using RaspberryPi or some other small format low power computers, but I agree that most people won't do it without a good reason.
If they are not sending and receiving coins daily or on regular basis, then they probably don't need to run full node and download full blockchain.
Any business that deal with Bitcoin should run a full node and Lighting Network node.

Home yes, school possibly, office only if you own the company. Putting non work software on your work PC is a good way to get fired. Also, unless you are remote work and connecting to an office PC, more and more offices are shutting down PCs and everything else at night. It's a bit more secure and saves power.
Unless you are government official of some corrupt country, than you can run anything you want :D
I know several examples of people like that doing Bitcoin for years in some countries (maybe they still are), and story even broke out in news.


Title: Re: Is anyone working in a "pruned download"?
Post by: LoyceMobile on October 18, 2022, 05:46:42 PM
I just found a pruned download link in another topic, but I'm not at my desk. I wouldn't trust it though, for good reasons.

Years ago, I created my own, but abandoned it due to lack of processing power. I've got that covered now, so if there's a demand LoyceV can set this up with daily updates. I don't like the trust part though.


Title: Re: Is anyone working in a "pruned download"?
Post by: Wind_FURY on October 19, 2022, 11:50:18 AM

Home yes, school possibly, office only if you own the company. Putting non work software on your work PC is a good way to get fired. Also, unless you are remote work and connecting to an office PC, more and more offices are shutting down PCs and everything else at night. It's a bit more secure and saves power.

Unless you are government official of some corrupt country, than you can run anything you want :D
I know several examples of people like that doing Bitcoin for years in some countries (maybe they still are), and story even broke out in news.


OR if you work for a government office, which 100% has dirty politicians, then you definitely SHOULD install a Bitcoin node in ALL those government computers. Make them mine unprofitably too! It will be considered a public service. Ot's probably one of the demonstrations of the importance of pruning, you can stealth-install Bitcoin nodes in computers you don't own. 8)


Title: Re: Is anyone working in a "pruned download"?
Post by: DaveF on October 20, 2022, 01:32:52 PM
OR if you work for a government office, which 100% has dirty politicians, then you definitely SHOULD install a Bitcoin node in ALL those government computers. Make them mine unprofitably too! It will be considered a public service. Ot's probably one of the demonstrations of the importance of pruning, you can stealth-install Bitcoin nodes in computers you don't own. 8)

Drifting OT, but only if they don't have proper security and allow people other then the IT staff / administrators to install and run software.

Back OT, other then the savings of long term storage space I still don't think it's worth it. If you can get the d/l from what you think is a trusted source, then fine. But as has been mentioned, that goes against the verify it yourself principle of BTC

Going to leave it at that, I don't think either side is really going to change the other sides mind at this point.

-Dave