Can I just run a pruned node without downloading historical data?

[PrimeNumber7]

Also part of the problem is that as we have discussed in other threads, you can put a complete node together for well under $200, NEW.
Used PCs and stuff are much much less. https://www.ebay.com/itm/284407553576
At what point does doing the lets try 100 different things so you don't have to download the entire blockchain or have that extra storage space not worth it because if you ever want to do anything else with the node you are going to have to download it anyway and all you did is waste time and effort to get to the same place you were going to wind up at anyway.

I agree; some people are ready to fully trust 3rd parties, while others want to change the Bitcoin protocol itself.... for what? For impatience and / or not wanting to save up for a few pieces of basic hardware (which can always come in handy in case you abandon the full node project).
If price is really such a big concern, they can even get hardware for free, like look on eBay & similar for trashed laptops where mostly the screen is broken - there you go, saved 100 bucks off a Raspi setup. SSD / HDD you might want to buy new, but at that point it's just one single component left to buy. If you can't afford an SSD, 1TB HDDs are dirt cheap in 2021 and people might give them away too, if you ask around. I find them used everywhere for 20€.

The cost of hardware is not the only cost of running a full node. Users must also have an internet connection that is sufficient to handle block downloads and the initial blockchain sync. For most of the "first world", this is not an issue as users will have an existing broadband connection they use and need for other reasons. There are "data caps" in some parts of the world that limit the amount of data a customer can transfer per month without paying an overage fee. If you have a data cap, there is a good chance the 350 GB initial blockchain download will exceed your cap, as there are very few legitimate reasons to need to download that much data in one month (the initial blockchain download is probably the only legitimate reason for most non-business users of an internet connection).

[1714101633]

1714101633

[1714101633]

1714101633

[n0nce]

The cost of hardware is not the only cost of running a full node. Users must also have an internet connection that is sufficient to handle block downloads and the initial blockchain sync. For most of the "first world", this is not an issue as users will have an existing broadband connection they use and need for other reasons. There are "data caps" in some parts of the world that limit the amount of data a customer can transfer per month without paying an overage fee. If you have a data cap, there is a good chance the 350 GB initial blockchain download will exceed your cap, as there are very few legitimate reasons to need to download that much data in one month (the initial blockchain download is probably the only legitimate reason for most non-business users of an internet connection).

You're right; if we look at sites like https://bitnodes.io/ it is clear BTC full nodes should be more decentralized, especially in developing countries.

https://bitnodes.io/

This is now going a bit off-topic, we could create a new topic for 'bringing BTC full nodes to developing countries' to gather ideas But a few thoughts:
If they don't run a full node (this topic is about pruned nodes), the decentralization is not really better; it's just maybe better for them (anonymity when looking up blockchain information). If there's no way to get an uncapped data connection, it might be an idea to sell / gift HDDs with blockchain sync on them to such countries. They should download Core and verify it themselves; then the software will verify all blocks when starting up, so the users don't really run a risk there of 'trusting someones IBD'.
With regards to legitimate use of bandwidth: way back, when I started my first Bitcoin node, I checked my family's monthly data consumption to see if it would jump up significantly with a node running, and it was ~300GB of all legitimate data So it's not unreasonably much. There are also schools, student dorms etc. which share a connection and go terabytes per month at least. But I agree; some places indeed don't offer unmetered, so we need a solution for that, which still allows people to have the full blockchain locally.

Edit: As I like to do, I had a quick peek at eBay / craigslist - people sell 250GB HDDs for 5-10 bucks! It may be an idea to even ship around semi-synced blockchains; so people don't have to download 370GB (current IBD size) but just the last ~120. It would be cheaper, probably more secure and 100ish GB in one month may be more feasible for lots of people than almost 400.
Just throwing ideas out there

[aliashraf]

OP,
you are absolutely entitled to what you are asking for: a full node with fast sync and little HDD waste.

Although it is technically possible for Bitcoin to offer such a critical utility without sacrificing essential security/decentralization measures, unfortunately in the Bitcoin community we are dealing with a sort of sectarian culture which is not tolerant in trade-offs, it is typical for cults, you know, they don't trade off!

In fast sync problem case, it is easily achievable by:
1- Implementing a more smart internal representation for the UTXO
2- Applying a minor extension to the Bitcoin consensus, making it mandatory for blocks to commit to the valid UTXO state.
3-Adding required messaging protocols for fast syncing through UTXO acquisition.

Costs/trust issues are negligible because using a proper prune factor could make it to an unfeasible level expensive for a scammer to fake an illegitimate UTXO. For instance a prune factor of 5000 blocks implies like 2.5 Billion dollars of costs/missed-opportunities for attackers and requires the victim to be Sybil attacked at the same time because of fame issues. It is true that faking the whole blockchain takes orders of magnitude higher costs,  anybody who needs or actually utilizes such a level of security could always go with the legacy boot-from-scratch option.

In SPV case you have zero security against the server you've put trust in because it doesn't need to do any work on the fake blockchain as long as you are "Sybil"ed, so arguments that compare UTXO commitment with SPVs are misleading.

 

[DaveF]

The cost of hardware is not the only cost of running a full node. Users must also have an internet connection that is sufficient to handle block downloads and the initial blockchain sync. For most of the "first world", this is not an issue as users will have an existing broadband connection they use and need for other reasons. There are "data caps" in some parts of the world that limit the amount of data a customer can transfer per month without paying an overage fee. If you have a data cap, there is a good chance the 350 GB initial blockchain download will exceed your cap, as there are very few legitimate reasons to need to download that much data in one month (the initial blockchain download is probably the only legitimate reason for most non-business users of an internet connection).

Lots of vides streaming services easily push more data then that per month to customers.
There are other solutions for the IBD and connection to the blockchain in general

https://store.blockstream.com/product-category/satellite_kits/

But they are not cheap, if you are having trouble getting the $200 together to build a node then $500 to get a satellite kit is probably too much.

-Dave

[PrimeNumber7]

The cost of hardware is not the only cost of running a full node. Users must also have an internet connection that is sufficient to handle block downloads and the initial blockchain sync. For most of the "first world", this is not an issue as users will have an existing broadband connection they use and need for other reasons. There are "data caps" in some parts of the world that limit the amount of data a customer can transfer per month without paying an overage fee. If you have a data cap, there is a good chance the 350 GB initial blockchain download will exceed your cap, as there are very few legitimate reasons to need to download that much data in one month (the initial blockchain download is probably the only legitimate reason for most non-business users of an internet connection).

You're right; if we look at sites like https://bitnodes.io/ it is clear BTC full nodes should be more decentralized, especially in developing countries.
[img width=512 ]https://i.postimg.cc/nzrTR66C/image.png[/img]
https://bitnodes.io/

This is now going a bit off-topic, we could create a new topic for 'bringing BTC full nodes to developing countries' to gather ideas But a few thoughts:
If they don't run a full node (this topic is about pruned nodes), the decentralization is not really better; it's just maybe better for them (anonymity when looking up blockchain information). If there's no way to get an uncapped data connection, it might be an idea to sell / gift HDDs with blockchain sync on them to such countries. They should download Core and verify it themselves; then the software will verify all blocks when starting up, so the users don't really run a risk there of 'trusting someones IBD'.

A pruned node is technically a full node, although with slightly less functionality, such as the ability to rescan the blockchain for transactions.

When I purchase hardware with the intention of using it for anything to do with bitcoin/crypto, I will purchase hardware that could potentially be (and most often is) used by someone who is not using the hardware to handle bitcoin/crypto transactions. This reduces the risk of someone tampering with the hardware who is targeting someone who is a bitcoin user. If someone is selling HDDs with the blockchain pre-loaded, there is the risk to the buyer that the seller has put some kind of malware on the HDD designed to steal the buyer's private keys.

I suggested upthread that blocks contain a hash of the UTXO set. This could potentially be done via a softfork, for example by requiring coinbase transactions to have a OP_RETURN transaction with this hash. Nodes must already store the current UTXO set in order to check if a particular transaction is valid or not.

A node can obtain the actual UTXO set as of block x from a third party, calculate the hash of the UTXO set it was provided, and compare this to the hash contained in block x it receives from its peers. The node will also look at subsequent blocks to see how the UTXO set changed according to the blocks, calculate the hash, and validate accordingly. As long as the node is not being subjected to a Sybil attack, I would confidently say the node has sufficient information to know if a transaction is valid or not.

With regards to legitimate use of bandwidth: way back, when I started my first Bitcoin node, I checked my family's monthly data consumption to see if it would jump up significantly with a node running, and it was ~300GB of all legitimate data So it's not unreasonably much. There are also schools, student dorms etc. which share a connection and go terabytes per month at least. But I agree; some places indeed don't offer unmetered, so we need a solution for that, which still allows people to have the full blockchain locally.

I am not sure where you live, or how big your family is. My guess is that a big chunk of your data usage is from various streaming services. Streaming a video in 720p (HD) uses about 0.9 GB per hour, and streaming in 2k (higher quality HD) uses about 3 GB per hour. That works out to about 2.7 hours of 720p per day (assuming a family of four), or around 50 minutes of streaming per day at 2k. I don't think spending 2-3 hours/day watching Netflix is something I would want to be spending my time doing, but I also don't think it would be totally unreasonable.

I don't think there are a lot of people in third-world countries spending $14/month on Netflix though.

[n0nce]

If someone is selling HDDs with the blockchain pre-loaded, there is the risk to the buyer that the seller has put some kind of malware on the HDD designed to steal the buyer's private keys.

That's true, I agree! While I fully agree this poses some risk, I don't think many people will (or should) actually use the wallet in Bitcoin Core. The goal would mostly be to have a better node distribution. The ones that can run Electrum as well, might serve thousands of local Bitcoin users who all hold their private keys on their own devices.

I suggested upthread that blocks contain a hash of the UTXO set. This could potentially be done via a softfork, for example by requiring coinbase transactions to have a OP_RETURN transaction with this hash. Nodes must already store the current UTXO set in order to check if a particular transaction is valid or not.

In my opinion, this is a more significant change than you might think. Lots of nodes might decide to use this method, leaving only a small number of nodes holding the entire blockchain. While with a perfect implementation of your above suggested idea, it might not be an issue if one day nobody has the whole blockchain anymore, I believe such a big change (and possible risk) can't be justified since HDDs are so, so cheap.

I am not sure where you live, or how big your family is. My guess is that a big chunk of your data usage is from various streaming services. Streaming a video in 720p (HD) uses about 0.9 GB per hour, and streaming in 2k (higher quality HD) uses about 3 GB per hour. That works out to about 2.7 hours of 720p per day (assuming a family of four), or around 50 minutes of streaming per day at 2k. I don't think spending 2-3 hours/day watching Netflix is something I would want to be spending my time doing, but I also don't think it would be totally unreasonable.

I don't think there are a lot of people in third-world countries spending $14/month on Netflix though.

I don't like to share too much personal information; in fact I don't even know myself what constitutes how much data usage. I'm not a Netflix subscriber though. I would like to watch more movies but I never have the time. However, there are many more data hogs apart from Netflix binge-watching. Non-exhaustive, generic list: video calls / meetings (they were a thing before COVID ), software updates, cloud backups, working via VNC, simply large file downloads like ISOs, watching documentaries, high-bitrate audio streaming, downloading games (modern games can easily hit 100GB each), playing those games, cloud-connected security cameras - just to name a few. I don't know how much this list applies to developing countries or how much their average bandwidth usage is, though, to be honest.

[PrimeNumber7]

I suggested upthread that blocks contain a hash of the UTXO set. This could potentially be done via a softfork, for example by requiring coinbase transactions to have a OP_RETURN transaction with this hash. Nodes must already store the current UTXO set in order to check if a particular transaction is valid or not.

In my opinion, this is a more significant change than you might think. Lots of nodes might decide to use this method, leaving only a small number of nodes holding the entire blockchain. While with a perfect implementation of your above suggested idea, it might not be an issue if one day nobody has the whole blockchain anymore, I believe such a big change (and possible risk) can't be justified since HDDs are so, so cheap.

Storing the entire blockchain will provide businesses the opportunity to profit, for example by serving ads to users viewing transactions on a block explorer, analyzing previous transactions (similar to the work done by chainanalysis).

I think my proposal would remove nearly every barrier someone has to running a full node, and that people using light nodes would go down to nearly zero. I also think over the long run, if bitcoin were to see widespread adoption, it is going to see a much higher percentage of transactions conducted via their phone, either on-chain transactions or transactions on an L2 type network such as LN. I don't see any reasonable way to run anything resembling a full node on a phone without the ability to not download the entire blockchain.

Further, there is no reason why I need to know about someone buying coffee in 2014, or about every single transaction in the 2015 spam attack, or any other subsequent spam attack. The reason why I need to validate these transactions today is that these transactions must be validated in order to validate subsequent blocks.

I don't like to share too much personal information; in fact I don't even know myself what constitutes how much data usage. I'm not a Netflix subscriber though. I would like to watch more movies but I never have the time. However, there are many more data hogs apart from Netflix binge-watching. Non-exhaustive, generic list: video calls / meetings (they were a thing before COVID ), software updates, cloud backups, working via VNC, simply large file downloads like ISOs, watching documentaries, high-bitrate audio streaming, downloading games (modern games can easily hit 100GB each), playing those games, cloud-connected security cameras - just to name a few. I don't know how much this list applies to developing countries or how much their average bandwidth usage is, though, to be honest.

I think a lot of your examples would probably constitute "business" use, although most residential ISPs would probably not especially care if someone was doing any of the above except for the ISO files if they are downloaded on any kind of regular basis. Perhaps someone who is working from home might reasonably use 350 GB+ of bandwidth using his work computer, depending on the specific setup, and how much of his day is made up of meetings that must take place via video.

I think, for the most part, most of your above list is probably not very commonly used in third-world countries.

[pooya87]

You're right; if we look at sites like https://bitnodes.io/ it is clear BTC full nodes should be more decentralized, especially in developing countries.

Actually it is not clear at all from this picture. The only conclusion you can make here is that more listening nodes are coming from those areas and when you check where they are you realize that most of them belong to data servers. In other words there is a lot of listening nodes on data servers.
Otherwise this site does not show all bitcoin nodes, majority of which don't listen for incoming connections.

[LoyceV]

When I purchase hardware with the intention of using it for anything to do with bitcoin/crypto, I will purchase hardware that could potentially be (and most often is) used by someone who is not using the hardware to handle bitcoin/crypto transactions. This reduces the risk of someone tampering with the hardware who is targeting someone who is a bitcoin user.

This is always on my mind too, but I couldn't know if some (Chinese) manufacturer tampered with all (millions of) devices they sell.

I suggested upthread that blocks contain a hash of the UTXO set. This could potentially be done via a softfork

Why does this hash need to be included in the block? While you're downloading your pruned node, you could calculate this hash on your own. I see no reason to increase the size of the blockchain by a soft fork for this.

A node can obtain the actual UTXO set as of block x from a third party, calculate the hash of the UTXO set it was provided, and compare this to the hash contained in block x it receives from its peers. The node will also look at subsequent blocks to see how the UTXO set changed according to the blocks, calculate the hash, and validate accordingly. As long as the node is not being subjected to a Sybil attack, I would confidently say the node has sufficient information to know if a transaction is valid or not.

You've kinda lost me here: what can you do with this information that you can't currenly do (assuming you have a pruned node with full UTXO information)?

I don't think there are a lot of people in third-world countries spending $14/month on Netflix though.

Netflix charges substantially lower fees in poorer countries and the average download speed is higher than I expected. Who still has bandwidth limits?
With multiple devices in the family, 300 GB per month doesn't seem like much of a challenge even without downloading the full blockchain.
And thanks to Net neutrality, it's none of my ISP's business what I download.

[PrimeNumber7]

I suggested upthread that blocks contain a hash of the UTXO set. This could potentially be done via a softfork

Why does this hash need to be included in the block? While you're downloading your pruned node, you could calculate this hash on your own. I see no reason to increase the size of the blockchain by a soft fork for this.

Nodes do not calculate this hash presently.

A node can obtain the actual UTXO set as of block x from a third party, calculate the hash of the UTXO set it was provided, and compare this to the hash contained in block x it receives from its peers. The node will also look at subsequent blocks to see how the UTXO set changed according to the blocks, calculate the hash, and validate accordingly. As long as the node is not being subjected to a Sybil attack, I would confidently say the node has sufficient information to know if a transaction is valid or not.

You've kinda lost me here: what can you do with this information that you can't currenly do (assuming you have a pruned node with full UTXO information)?

Presently, nodes review information from each block starting from block 1 to keep track of all outputs, and if they have been spent. The only way to know if an input is valid and if it has been previously spent is to look at all blocks, or to use information derived from looking at all blocks. Nodes currently keep track of the UTXO set by storing it in RAM, but the only way they authoritatively know the UTXO set is because the node previously looked at every block.

For example, a node will know the following is part of the UTXO set:
txid,index,amount,block
2a6a13ffb7146ef05a009d65d83b08bf48d53b13c471d13113345b4c9953611b,1,13.55748202,704548

Nodes store the above information, along with similar information about every other unspent output.

If a node were to store the UTXO set in a very specific order, and in a very specific format, two arbitrary nodes would calculate the hash of this information and the calculated hash would be the same.

If the above hash were to be stored in blocks, someone could spin up a node, obtain the UTXO set from an arbitrary untrusted third party, and immediately know if an arbitrary transaction they receive is valid or not (assuming they know the current consensus rules). Your average user would not need to download and store the blockchain, nor would they need to wait for their node to catch up to the current tip if their node has been offline for a period of time.

I would also point out that dissenters who are using bitcoin will want to minimize their data use as much as they can. No amount of data use would be "safe", but being able use a full node after has been offline for days/weeks without having to download all blocks subsequent to the last time it was online would reduce the risk of using bitcoin, and would make it so they would not have to rely on third-party services to send transactions or verify if a transaction has been received.

[vapourminer]

I don't think there are a lot of people in third-world countries spending $14/month on Netflix though.
Netflix charges substantially lower fees in poorer countries and the average download speed is higher than I expected. Who still has bandwidth limits?
With multiple devices in the family, 300 GB per month doesn't seem like much of a challenge even without downloading the full blockchain.
And thanks to Net neutrality, it's none of my ISP's business what I download.

i have really really sucky internet. it would take months (literally) to redownload the full blockchain. and my internet was worse back in 2011 when i started. if i leave bitcoin core up i cant really use the internet at all for anything else, it stalls everything. so its generally offline and i run it to sync every couple weeks or so.

so i still have the original blockchain i started in 2011. full and unpruned. i back it up to a raidz2 (5 drives any 2 can fail)  nas whenever i sync it and also to multiple single drives, some offsite, monthly. if i lose it it would suck. hard.

downloading the full blockchain from scratch is a real pita for someone like me but as i use it to verify my own transactions i need to trust my node fully, ergo i need a full chain from genesis block to now. the hardware/storage is no problem. the bandwidth is.

it would be nice to have a better way but i cant see how. i trust no node but my node.

[HCP]

Theoretically... if you have a (trusted) "friend" that has a full unpruned node, one can simply copy the blocks/chainstate and drop them into a new node installation and "voilà", you have your full node without having to download anything, you just have to trust your "friend".

You can even set the new node to be pruned (after you have loaded and scanned your wallets) and it will discard all the unneeded blocks.

Of course, if storage space is the issue (and not bandwidth)... then this might not be an option either.

I just want to run a full node where I can just type in a certain public key, and it tells me which UTXOs are associated with this key. Basically a local version of blockchain.com/explorer

In any case, the OPs requirement to "be able to enter a random public key and view the balance" might preclude pruning anyway... as most (all?) blockexplorer apps require txindex, which is not compatible with pruning.

Alternatively, a bespoke software solution might be possible.

[PrimeNumber7]

Theoretically... if you have a (trusted) "friend" that has a full unpruned node, one can simply copy the blocks/chainstate and drop them into a new node installation and "voilà", you have your full node without having to download anything, you just have to trust your "friend".

Dont do this. The greatest president the US has had says to “trust but verify” and using any hardware from a third party will make it very difficult to be certain you are in fact verifying.

It might be possible to verify by connecting to your friends node locally, receiving and verifying blocks as if you would if you were receiving blocks via the internet. This would obviously require you to use your friends resources. It would not solve the problem of needing to catch up to recent blocks if your node goes offline after the initial sync.

[LoyceV]

Theoretically... if you have a (trusted) "friend" that has a full unpruned node, one can simply copy the blocks/chainstate and drop them into a new node installation and "voilà", you have your full node without having to download anything, you just have to trust your "friend".

Dont do this.

You can just do a full reindex, right? If your own Bitcoin Core installation verifies all the blocks you copied from your friend, you don't need to trust him anymore.

Code:

Usage:  bitcoin-qt [command-line options]                     

Options:

  -reindex
       Rebuild chain state and block index from the blk*.dat files on disk

I expect this to take a day, so I don't want to test the command.

[KIZILAGA]

I am trying to run Bitcoin core, but when I started the software, it says that I can either:

1) Download the entire blockchain and keep it on my disk, 350GB
2) Download the entire blockchain, verify historical data, and only keep the most recent 2 GB of blocks.

Either way, I have to download 350GB of historical data.

Forgive me, Toxic Maxis, but is there an implementation of Bitcoin Core where I can only download the most recent 2GB of blocks?

Furthermore, I have another idea.

Is it possible to build a full node that stores only the latest UTXO set and the latest 2016 blocks for difficulty adjustment?

If I store all current UTXOs and the latest 2016 blocks, is it to all intents and purposes, the same as storing the entire 350GB block chain?

If no such software exists I will build it from scratch. I am currently reading Programming Bitcoin by Jimmy Song.

yes you can with Grin ツ

[PrimeNumber7]

Theoretically... if you have a (trusted) "friend" that has a full unpruned node, one can simply copy the blocks/chainstate and drop them into a new node installation and "voilà", you have your full node without having to download anything, you just have to trust your "friend".

Dont do this.

You can just do a full reindex, right? If your own Bitcoin Core installation verifies all the blocks you copied from your friend, you don't need to trust him anymore.

The issue is not with verifying the blockchain. The issue is you are using hardware that is potentially infected with malware.