Bitcoin Forum

To begin with, I don't have deeper technical knowledge about the blockchain as many others here have. Which is why I would like to get opinion for a solution which I think might be possible to reduce the blockchain size.
Again, I am no expert and just expressing my views to get some better reviews and understanding of the blockchain.

Considering that the size of blockchain has already crossed 200GB+ I think the simple solution here would be to archive this data from the Blockchain and store it on some particular locations from where the archival is easy to access.

Let us consider that we have archived all the blocks until 200GB and have stored on some secure place. Also, we have attached a hash/signature to the archive and modified the consensus such that every new block that is generated includes a hash/signature from the archive. So now instead of validating all the previous blocks, the miner will validate only the hashes of archived blockchain, previous block and the current block.

This will save the 200GB of data that a miner has to validate and reduce the size of the blockchain. I know there must be some flaws in this system. Let me know in the comments.

This might a good useful idea but I have no idea how it can be achieved. I have the same problem like you. I am no expert in the Blockchain technology. :-(

What you are suggesting was proposed and discussed before, actually.
And the answer is no, this is not going to work nor solve the blockchain size problem.

First, if we are going to "store" an archived copy of the blockchain in a particular place then it can't be considered decentralized nor trustless system anymore.

Second, each block is cryptographically linked to the previous block. Thus, if we need to validate a block/transaction a full node has to go all the way back to the genesis block.

=> we need full nodes, and we need more of them.

Apart from what khaled0111 already wrote, disk space is not the main bottleneck. 200 GB are not much, it's almost nothing for modern HDDs.

The main bottleneck is the propagation of new blocks and new transactions.

You can already prune your blockchain to less than 1 GB (AFAIK 550 MB was the minimum).

There was a proposal in 2014 called the "Mini-blockchain scheme (https://www.weusecoins.com/assets/pdf/library/The%20Mini-Blockchain%20Scheme.pdf)", which is something very similar to what you're proposing. It has changed some rules to the code so that the fraction of the blockchain which has to be validated by full nodes is much shorter than in the current codebase, and from older blocks only the headers are stored. The proposal, however, hasn't got acceptance in the Bitcoin community, but there are altcoins using the proposed modifications, or variations of it.

Among the problems with this approach are:
- an 51% attack could not only double spend, but replace all complete blocks of the mini-blockchain. A "checkpoint" (issued in the code) would be necessary to make it usable again (see here (http://cryptonite.info/wiki/index.php?title=Weaknesses_and_attack_vectors#Attack_Vectors))
- there is no way to validate most contracts, LN, for example, would be probably impossible.

Do we really need more? Is there a statistical study showing which is the ideal and necessary number? Perhaps we need much more of a diversity of locality where there are full nodes than necessarily a much larger number of them.

My answer: "yes, we do"
Now, you have to believe me as I am the only one who answered, right!

What if someone else joins the conversation and replies: "no, we dont" ?
In this case you need answers from as many members as possible to know who is telling the truth.

The same applies to blockchain. Blockchain is more secure with more full nodes.
Also, full nodes make sure that miners are following the consensus.

Apart from that, it is always advised to run your own full node so you don't have to rely on other nodes.

It is worth mentioning that a full node is helping the network only if it accepts inbound connections.

If the size of the blockchain is the only concern in this discussion then instead of focusing on storing it elsewhere (which sounds more like centralization) focus on bitcoin transactions at byte level and see how you could "compress" them instead. FYI inside a transaction there are lots of wasted bytes that don't need to even exist. For your storage and even communication over P2P network you can come up with a compression technique that reduces the size by a lot.

I have actually been working on this recently and been able to "compress" a transaction up to 15%. That's just a start though, it needs a lot more work and study of other people's work to see what they have done, but you get the idea. With a block this can be more by simply dropping a lot of things and adding a "contract" in your compressor based on some factors such as block height. I expect the final result to be 30% compression.

I kind of knew this question would arrive and I was prepared for the same. The thing is we would not store it on one particular location but a group os 'trusted' locations.
Example: If we identify 50 or more trusted locations and store the archival there then it shouldn't be called centralized as the archival is still spread across trusted locations.

There is no need to validate all the previous blocks until the genesis block. Only the hash of the archival and the hashes of the previous blocks until the archival would be sufficient.

For example: The blocks until 200GB are archived and from the next block(first block after the archival) it would be considered as second genesis block or genesis block after archival.
So we would need to validate only blocks until second genesis block and not all the blocks until the real genesis blocks.

Yes, pruned blockchain does sort out the issue of storing the whole blockchain on your system however it still has to validate all the blocks until the genesis block.
Whereas my proposed system would eliminate the need to validate all the blocks until the genesis one and will only need to validate blocks until second genesis block/genesis block after archival whatever we name it.

Well that's a nice thing to start with. I haven't thought about it yet henceforth I can't comment on it. It's good that you have reached 15% compression and I hope you achieve more so that we all could get better at it.

The whole point of Bitcoin is for it to be trustless. Someone still has to manage the files and everyone's definition of who is trustable is different. Who should be the one assigning the trusted people? It's fairly easy for that person to push an agenda through and manipulate the entire system. If I needed to trust someone, I might as well go for Paypal.
How would you get the chainstate without indexing the blockchain? Trusting another person to obtain the chainstate is considered fairly unsafe as the information inside can be manipulated. You need the entire blockchain to ensure the validity of the chain. If not, you might as well just use an SPV client.

SPV client would be a better alternative. If you don't validate the blocks fully, you are essentially an SPV client and you don't have to download any blocks at all.

issues:
if you want to be a full node. then take the big boy pants, put them on and be a full node. if your just gonna leach the blockdata from someone else and then strip signatures, prune and filter to be left with pretty much just utxo's your no longer a full node. no longer able to seed the full data to others.
so just give up starting as a full node if next week your going to be a useless litewallet.
make the choice be a full node or a lite wallet.

solutions:
the main concern for full nodes is not the physical space. nor the actual time to download the full sync. but the fact that the main reference software is coded that you cant even see a estimated balance or send out a transaction until the sync is complete. thus making the sync appear slow because your sat there doing nothing twiddling your thumbs waiting for it.

but, you dont actually have to wait. so the solution is simple.
instead of starting as full node leaching off others to then downgrade to spv/litewallet. wasting others resources. do the opposite
get a UTXO set first to get users active and making transactions with a 'independently unverified balance' as a spv/lite wallet which is still safe because if your personal utxo is spent the network wont allow it to relay/confirm in a new block. then. the syncing to get the full archive and verify all transactions becomes a background task, thus no twiddling idle thumbs.

tl:dr
dont start as full and downgrade to spv
start as spv and upgrade to full

well if core developers inteoduced schnorr signatures you would save 25% space

/kx

As a security measure, I believe it's better to have more than it is to have less full nodes in the network. Because if there was a change in the network that causes some people not to run full nodes due to higher costs, the network will centralize by some "number". We want the contrary that, more decentralized = more security.

I like this. And the beauty of it is that it can be possible without altering the protocol; each and every client stores the data in the way it prefers or can, the useful data is the same.
But I fear that this will not solve much. Size-related discussions are here since the chain data was under 100GB. And I feel like the complaining ones are the newcomers that have to do the initial setup and download everything.

And then Franky's idea becomes even more interesting.


Since many use the Core wallet only because it's the official one and also praised to be the safest one, it could have the option - which could be selected by hand or asked whenever the wallet is more than a couple of blocks out of sync - to run as a SPV wallet (with reduced functionality, compared to the full wallet, obviously)
The only problem here is that it has to be implemented very well to avoid issues like Electrum had and also make it clearly visible what mode is currently running.

No you won't. This topic is about the size of "blockchain" not size of signatures or transactions. Since block size will remain the same, the blockchain size will remain the same and grow at the same rate as long as new blocks are full.

I've been trying to finish it up and release it but other things keep coming up and I sidetrack.

knowing the network is centralised code to the "reference client" it does not matter if its 100 nodes or 100,000 nodes. if there is a bug in the code (https://bitcointalk.org/index.php?topic=5032443.0) its not going to be limited to a single user. it will affect everyone running the code.

if you are talking about DISTRIBUTION rather than decentralisation. then having a node connect to 5 other nodes concentrates the bandwidth on 5 nodes. thus faster speed/propagation. but if the node is connected to 100 other nodes than the speed/bandwidth is diluted by 20x making each connection slower and propagation worse. so sometime distribution especially if nodes foolishly then strip/prune/filter data make it so theres more leachers than seeders

having some non-interested basement user that isnt really interested in bitcoin personally but has the naive assumption that they NEED to push their computer and bandwidth to the limit purely assuming it 'helps the network' is not what a decentralised network should be veiwed as.

wats better is for those people who NEED to verify more than a few transactions a day, have more reason to be a full node. and it becomes in their own benefit of needing to verify that would self motivate them to want to invest in self verifying.

there is no need to stifle bitcoin to make average uninterested joe into a full node. its just better to let nature do its thing, better to have 100,000 businesses and regular users be full nodes because their self motivated by the need to verify.. rather than 100,000 randomers thinking they should be full nodes because 'fear of network security'

You know well what I was talking about, franky1. Stop bringing your social drama in all of the discussions.


Those hat wearing basement dwellers were successful in pushing for Segwit's activation. Remember UASF.


But it shouldn't be that the ability to run a full node should be taken away from anyone.


Scaling out Bitcoin is stifling?

Plus why should it matter? 100,000 nodes are 100,000 nodes from the network's standpoint, whether they are owned by ordinary users or merchants, or miners. They all validate to make sure every transactions and blocks follow the rules.

Validate historic transactions and ensure nobody has violated the Bitcoin Protocol. However, there are proposals for more efficient ways to transmit and store the information currently in the block chain.
total amount of information in the block chain can't be reduced without making it impossible for new nodes.

I doubt there would be a difference there. At least without the change, the network would be decentralised.

That would likely occur only if there is some form of protocol rule change which somehow results in higher costs for those having nodes. In this case, there would likely be fairly little support from the community and the fork wouldn't even happen in the first place. If it is just a modification to the reference client that doesn't affect Bitcoin at a protocol level, people would be fine running the older nodes which wouldn't matter.

here is a lesson
by only limiting bitcoin to ~3ktx a block =500k a day
means without day traders(spammers) if people only transacted 1tx a day. thats 500k users needing bitcoin daily.
(with day traders/spammers, the numbers of users is less)

forget imagining millions of users wanting to b full nodes. because your not even going to get 500k users wanting to be full nodes

infact if we said 50k transactions were from people doing 5tx a day(10k users) and 450k were from 1tx a day.. psychologically the stats would presume there is only 10k users that NEED to be full nodes because they are handling more than 1 tx day to want to be more involved.

after all who would want to run a full node when they are not even using bitcoin.
..
people need to think about the more they deburden users actually using the blockchain.. the number of people wanting to secure the blockchain diminishes too

Maybe not, but it's not a good design-decision for a decentralized network if you have made it to discourage users from running their own full nodes.

Do you want a real decentralized network, or a network centralized towards the miners?

And what is the incentive for a home user to run a full node? I have to spend storage, bandwidth and other costs of keeping a server online, yet i'm not getting a single satoshi for it.

Even worse, i know miners earning plenty don't even bother setting one in an old pc next to their hundred or thousand asic miners...

you got no clue
1. ASICS(miners) have no hard drive. so thinking th number of nodes would centralise around miners is stupid. again ASICS dont even have hard drives to even be full nodes.

2. ASICS(miners) communicate to pools. there are not 100,000 pools. there are only 20 main pools

3. pools cant do much if there are 5k, 50k 500k nodes to reject any blocks that a particular pools releases.. a pool has to not only have large hashrate to stay ontop to keep its blocks active but also needs thousands of nodes to sway consensus over to "th more popular network". but even so merchants and users have to agree to the rules of such.
much like the NYA agreement(BSgroup) being onside with cores 2017 controversy

but here is the main point
deburdening bitcoin of utility by shrinking th blocks.. purely to promote an alternative network is going to discourage users from running their own node MORE

it seems people dont actually care about bitcoin utility or security. its just the lazy unemployed basement dwellers that want free coin from just running a computer.

its these lazy people we dont need. we ned people that care about security and community. not greed

oh and last lesson..
if th hope of stifling bitcoins utility to promote LN in the hopes of getting LN routing income. atleast do some math
if your charging 1msat per route to route something like a coffee.
you will have to process hundreds of thousands of payments just to make a couple pennies
meaning you will have to have hundreds of thousands of collateral locked to make them payments

in short. stifling bitcoin utility.. becoming an LN route WONT make you rich
if you dont care for bitcoin enough to research the code/utility.. atleast use your greed to do some maths to realise you wont get rich with the foolish rhetoric that stifling and deburdening bitcoin is good network decision

Then the more for the need for the users to run their own nodes, and validate. The ability to run full nodes should not be taken away because a small minority in the commumity want bigger blocks, faster transactions, or lower fees.

That small minority will follow Bitcoin's model after three halvings.


Stop the social drama, tinfoil hat conspiracy theories.



Hahaha. What are you talking about?


Then how would bigger blocks help?

1. which requires MORE transaction count on the BTC network. to allow more people to want and use bitcoin more
not less transactions where users play with other networks
not more expensive fee's to push people out of wanting to use the btc network
not cutting down transaction count to make it appear exclusive in the hopes it makes it more popular/energised

analogy: if you want more people to use the airline network to travel internationally.. you dont reduce the number of planes

2. if you think 1mb legacy, 4mb wishywashy is too much to handle. then are you also crying that call of duty wont run on your old sega genesis(megadrive). or have you upgraded your games console and realise millions of people CAN cope with playing
we are in the fibre, 5g internet age. not dialup (56kbit/s=4.2mbyte/10min)
we are in the terrabyte hard drive era. not the floppydisk 4gb HD era

3. nothing stops users being full nodes. but users have to put an effort into it.
nothing stops users playing call of duty. but users have to put an effort into it.

if lazy people want to get rich but pay nothing. they wont get what they want

what does truly make me laugh though
those that want to get rich for fre via stiffling bitcoin. and pretend its about cost of running bitcoin is too much. are the same lazy fools that are happy to push UP the price of transactions.

the obvious reason is they want to deburden bitcoin. and make users use other networks in the hope for lazy people to get rich free. its got nothing to do with improving bitcoin utility. its about greed via other networks

I'm confused. But the people already want to use Bitcoin more.


Then good. Maintain regulating the block size, let the network scale out, and decentralize further.


The higher costs will.

I think a snapshot sort of idea could be a good thing.

Code in so when you hit block X;  the entire blockchain is broken down into its constituent records;  and keep only the addresses and values along with checksums for each that contains a balance; make it as a new base block and have the network find a hash to sign that package (no reward); use that package in a manner to seed the start of the first block..  From there, continue business as usual.    When the blockchain grows up again;  it can all be done again.  Basically delete the chain of record; and make it contain addresses with a balance and a hash of the two as a checksum.

But it would have to be done with consensus.  100% must be.

Each node at the appropriate blockheight will begin assembling the new "genesis block" which could be of infinite size, and once built;  nodes start hashing away to sign that package.   Literally mine out the reset of the blockchain.... not only that, but at the time of the network change;  node consensus should be in the magnitude of [some number deemed appropriate] reporting in with an individual verifiable hash.   Estimated hash difficulty requirement can be worked out in the process... maybe make it variable based on hash time and network block consensus count.  once that threshold is passed;  start processing tx's as normal.

The early records can be saved independently; and re-verified as is needed; but once its truly known everything is verified with the replacement blockchain;  it would basically be delete-able unless you wanted to traverse the full record in the future.

I know there are many ways for it to possibly go wrong;  but we have ample time to test and re-test and get it right before we do such a thing.... and I really think this could be a reality for making the blockchain smaller.    


I also remember something about the merkle root being able to take place of a lot of block data;   like a side-ledger....   I need to do more diligent research about that [from what i recall] unused aspect.

sorry i'm rambling a little;  its late but i wanted to comment.

people already use bitcoin via exchanges, webwallet, phone apps, lite wallets.
but you yourself were saying
"Then the more for the need for the users to run their own nodes, and validate."

but if your reducing the blocksize, LESS users will use bitcoin regular
those irregular users wont want to be full nodes because they are not using it regular enough

and again if you want more users. you need more transactions.
users playing with other networks and only broadcasting to the btc ntwork once a month, are not people that will be full nodes
users broadcasting to the btc ntwork once day atleast are the people that will be full nodes

deburdening the btc network of utility (smaller blocks, less transactions) doesnt help promote the btc network
deburdening the btc network of utility (increased fee's, costing$1+ per use) doesnt help promote the btc network

infact those 'once a month broadcasters' wont even be regular users, but 'factories' so dont expect promoting other networks to also promote users to be full nodes

complaining that running a full node. based on hardware that can be spread over 4 years meaning the daily cost is under $1
but then promoting fee wars, based on fee/reward equilibrium of $33 a tx (12.5*$8k=$100k... $100k/3000tx=$33)

i find it shocking how people presume hundreds of thousands of people would pay $33 a tx but wouldnt pay under $1 a say for a PC that they can also use for many things while using bitcoin. (hint people already do spend under $1 a day on their pc)

Stop making the debate very confusing for everyone. Simply, the people already want to use Bitcoin more. Can you make a debate that the people want to use Bitcoin Cash more?

go take your bitcoin cash derailments elsewhere

yes normal users want to use bitcoin more. but devs and a certain corporation group want people to use alternative networks by making bitcoin transactions expensive, less transaction count. all in the aim of the HOPE of getting rich via other networks.


but the point is reducing transaction count from 3000tx a block(~500k a day) down. means less people will be able to use bitcoin regularly.
by having less people regularly using bitcoin, less people will ned/want to be full nodes. they would just use lite/web wallets

i said it before having users that only validate 1 tx a day/week just dont see the benefit of validating everything 24/7.
i said it before having users that only validate 1 tx a day/week just dont see the benefit of running a system 24/7
its these people that then complain that by running their system 24/7 its costing them a few extra cents of electric or having to upgrade their internet package. which for 1tx a day aint worth it to them.

but if they could use it daily and often without large tx expense, then they would be happy
also more transactions would mean more users.
EG if there were say 500k daily bitcoiners. and 100k are fullnodes(20%)
imagine if there were 2.5m daily bitcoiners.. that might mean 500k fullnodes(20%) +
imagine if there were 250k daily bitcoiners.. that might mean 50k fullnodes(20%) -

It's not an a derailment, nor was it my intention. It's a valid question in my opinion, asking from your previous post to clarify.


Thank you. That's the answer we needed.


But why would you want to take away a user's, or anyone's ability to run their own nodes, to validate their own transactions themselves, from their own nodes? Bitcoin is still a decentralized network, isn't it?

You can achieve something very similar even without modifying a single line of Bitcoin's code: Simply download an archived blockchain copy - this is much faster than validating it - and then prune it. (There seem to be even services for that: getbitcoinblockchain DOT com - dunno if it's trustworthy so I disabled the "hot" link).

You node then would not contribute to the validation of the early parts of the blockchain (in your system, you wouldn't either). However, it still would be better than a SPV node because it would validate everything after the point where you downloaded the copy.

It should even be possible to download already pruned archived blockchains (I haven't found a download repo however). You can also create a tool for distribution of these blockchain copies using a network like BitTorrent, although you will probably never be 100% secure that you have a "clean" blockchain copy.