Solving the Scalability issue for Bitcoin

[shemtov]

Solving the Scalability issue for Bitcoin  

What am I trying to solve? Currently Bitcoin’s blockchain is around 140GB.
In 2011 it took 1GB, and it was predicted back then that in 2020 that size would be 4GB.
As you can see it is not yet 2020, and we are way over that predicted size.
At our current time, prune nodes which make the block smaller, but they can not be validated without the full node. And this full node is getting exponentially bigger, we need to stop that. Because if we don’t no private citizen will have the capability of storing the full node in his computer, and all full nodes will be at private multi-million dollar companies. That would literally be the end of decentralization (or non-centralization).
What I am proposing also makes sure the blockchain has a maximum finite size, because today the blockchain can grow to any size without limit while it approaches an infinite size!

The linear growth illusion of Segwit
-------------------------------------------------------
It is true Segwit and Segwit2x are agreements which promise a linear growth in blockchain size. But Segwit and Segwit2 are only temporary solutions. The main factor in exponential growth isn't policies like Segwit, but the growing popularity of Bitcoin. Segwit was only thought up, in order to meet with growing demand. And in the near future when the demand keeps growing, the community will be forced to change policy, by either agreeing to larger blocks, or shortening the duration time between blocks. Because if it doesn't, average transaction time will hike up to 10 hours, or even days. Or, if you are in a hurry, then you will have to pay a transaction fee of $5 or more, so that your transaction will be on the next block. And then bitcoin's value will most likely drop, because it will become cheaper to use paypal, or any other altcoin which is still small enough to be managed. So Segwit as good as it is, can only manage a fixed amount of Transactions in a specified period of time. Once the demand grows, Segwit will not have a choice but be amended, and that is how linear growth becomes exponential.

Today our blockchain is growing at speed which is much faster than Moore’s law! This proposal will help set storage growth at a reasonable number.


A short list of what I am about to explain:   Steps that need to be taken:
---------------------------------------------------------------------------------------------------------------------
(The details are not described in this order)
1) Create a pair of keys, called the Genesis Pair, or Genesis Account, a private and public key which will be publicly known to all and yet it’s use will be restricted and monitored by all. The key will be the source of all funds (Point A).
2) Preserve the Genesis Block, its hash code is needed. And personally I think its of historical value.
3) Combine all Blocks up to the most recent (not including the Genesis Block), and cut out all intermediary transactions, by removing All transactions, and replacing them with new transactions sent from A to every public key which has funds in the most recent block, in the amount they have. And sign these transactions with A’s private-key. And create a new block with this information.
4) This Combined/Pruned Block should point to the Genesis Block hash, and the next block created should point to the Pruned Blocks hash. The random number used for this pruned block will be predefined, this random number normally used to meet the hash difficulty requirement in this case is not needed, since no difficulty setting is necessary for this block, and by predefining it, this block can be easily identified.
5) Download the pruned block from another node or create it yourself, the hash code will be identical for everyone, since the block will be created exactly the same everywhere.
6) Preserve a certain amount of the most recent blocks, just in case a longer fork of the blockchain is discovered, and then the Pruned Block should be recalculated.

---------------------------------------------------------------------------------------------------------------------
Now for a more detailed description:
I have this idea to solve the scalability problem I wish to make public.
If I am wrong I hope to be corrected, and if I am right we will all gain by it.
Currently each block is being hashed, and in its contents are the hash of the block preceding it, this goes back to the genesis block.

What if we decide, for example, we decide to combine and prune the blockchain in its entirety every 99,999 blocks to one block (Genesis block not included in count).

How would this work?: Once block 100,000 has been created, the network would be waiting for a special "pruned block", and until this block was created and verified, block 100,001 would not be accepted by any nodes.
This pruned block would prune everything from block 2 to block 100,000, leaving only the genesis block. Blocks 2 through 100,000, would be calculated, to create a summed up transaction of all transactions which occurred in these 99,999 blocks. (This number n=100,000 can be any number agreed upon by the community, it is not essential for this proposal to work)

And its hash pointer would be the Genesis block.
This block would now be verified by the full nodes (or created by them), which if accepted would then be willing to accept a new block (block 100,001, not including the pruned block in the count).

The new block 100,001, would use as its hash pointer the pruned block as its reference. And the count would begin again to the next 100,000. The next pruned block would be created, its hash pointer will be referenced to the Genesis Block. And so on..
In this way the ledger will always be a maximum of 100,000 blocks.

 A bit more detail:

All the relevant outputs needed to verify early transactions will all be preserved in the pruning block. The only information you lose are of the intermediate transactions, not the final ones the community has already accepted. Although the origin of the funds could not be known, there destination is preserved, as well a validation that the transactions are legitimate.
For example:

A = 2.3 BTC, B=0 BTC, C=1.4 BTC. (Block 1)
If A sends 2.3 BTC to B.  (Block 2)
And then B sends 1.5 BTC to C. (Block 3)
The pruning block will report:
A->B = 0.8 BTC and A->C=2.9 BTC.
The rest of the information you lose, is irrelevant. No one needs to know, what exactly happened, who sent who what, or when. All that is needed is the funds currently owned by each key.

Note:  The Transaction Chain would also need to be rewritten, to delete all intermediate transactions, it will show as though transactions occurred from the Genesis block directly to the pruned block, as though nothing ever existed in between. This will be described below in more detail.

You can keep the old blocks on your drive for 10 more blocks or so, just in case a longer block chain is found, but other than that the information it holds is useless, since it has all been agreed upon. And the pruning block holds all up to date account balances, so cheating is impossible.

Granted this pruning block can get extremely large in the future, it will not be the regular size of the other blocks. For example if every account has only 1 satoshi in it, which is the minimum, then the amount of accounts will be at its maximum. Considering a transaction is about 256bytes. That would mean the pruning block would be approximately 500PB, which is 500,000 Terra-bytes. That is a theoretical scenario, which is not likely to occur. (256bytes*21M BTC*100M (satoshis in 1 BTC))

A scenario which could be solved by creating a minimum transaction fee of  for example: 100 satoshis, which would insure that even in the most unlikely scenario, at worst the pruning block would be 5PB in size.
Which is still extremely large for today. But without implementing this idea the blockchain literally does not have a finite maximum size, and over time approaches infinity!

Also, this pruning block does not even need to be downloaded, it could be created by already existing information, each full node by itself, by:
1) combining and pruning all previous blocks
2) using the genesis block as its hash pointer
3) using a predefined random number "2", which will be used by all. A random number which is normally added to a block to ensure the block's hash-rate difficulty, is not needed in this case, since all information can be verified by each node by itself through pruning.
This number can also be used to identify this block as the Pruned/Combined Block since it is static.
4) Any other information which is needed for the SHA256 hash, for example a time-stamp could be copied off the last block in the block chain.
These steps will ensure each full node, will get the exact hash code as the others have gotten for this pruning block.

And as I previously stated the next block will use this hash code as its hash reference.
By creating a system like this, the pruning block does not have to be created last minute, but gradually over time, every time a new block comes in, and only when the last block arrives (block 100,000), will it be finalized, and hashed.
And since this block will always be second, it should go by the name "Exodus Block".

Above, I showed a way to minimize the blocks on the block chain, to lower the amount of space it takes on the hard drive, without losing any relevant information.
I added a note, saying that the transaction chain needs to be rewritten, but I did not give much detail to it.

Here is how that would work:

The Genesis Account (Key Pair):
---------------------------------------------------
The problem with changing the transaction and block chain, is that it cannot be done without knowing the private key of the sender of the of the funds for each account.
To illustrate the problem: If we have a series of block chains with a string of transactions that are A→B→C→D, and to simplify the problem, all money was sent during each transaction, so that no money is left in A or B or C. And I was to prune these transactions, by replacing them with A→D. Only this transaction never occurred, nor can anyone create it without A’s private key.
There is however a way to circumvent that problem. That is to create a special account called the “Genesis Account”, this account’s Private Key and Public Key will be available to everyone.
(Of course, accounts do not really exist in Bitcoin, when I say account what I really mean is a Private/Public Key pair)
This account will be the source of all funds
But this account will not be able to send or receive any funds in a normal block, it will be blocked--blacklisted. So no one can intentionally use it. The only time this account will be used is in the pruning block, a.k.a Exodus Block.
When creating the new pruned block chain and transaction chain, all the funds that are now in accounts must be legitimate, and it would be difficult to legitimize them unless they were sent from a legitimate private key, which can be verified. That is where the Genesis account comes in. All funds in the Exodus Block will show as though they originated and were sent with the Genesis private-key to generate each transaction.
The funds which are sent, must match exactly the funds existing in the most updated ledger in block 100,000.
In this way the Exodus Block can be verified, and the Genesis Account cannot give free money to anyway, because if someone tried to, it would fail verification.

Now the next problem is that the number of Bitcoins keeps expanding and so the funds in the Genesis Account need to expand as well. That can be done by showing as though this account is the account which is mining the coins, and it will be the only account in the Exodus Block which “mines” the coins, and receives the mining bonus. In the Exodus Block all coins mined by the real miners will show as though they were mined by Genesis and sent to the miners through a regular transaction.

I hope this proposal will be implemented as soon as possible so that we can avoid a problem which is growing by the minute. It was brought up about 6 years ago when the blockchain was only 1GB in size, nobody imagined back then that it would grow so quickly, and the problem was ignored.
Today all solutions implemented have been implemented by software, and not on the blockchain itself, these solutions are not helpful in the long run.

The full node needs to be publicly available to everyone, and at this rate, nobody will have the hard-drive capacity to store. This will make us more dependent on private corporation’s to store the blockchain, which will lead us quickly to a centralized currency platform. By then it will be too late, and the corporations will have complete control of what happens next.
Please take this problem seriously and work with me, to prevent it while we still have some time.
The exact details can be worked out at a later time, but for now we need at least an acknowledgment that this problem is dire, and needs to be solved in a year’s time. I have presented a solution, if someone has a better one, then let him/her step forward, but in any case a solution needs to be implemented as soon as possible.

I have given a basic proposal, I am sure there are those among us with more technical understanding to the nuances of how this idea should be implemented. I am counting on their help to see this through.

Adam Shem-Tov

[1714984472]

1714984472

[aleksej996]

I couldn't read such a big post right now, but based on the section names I see that you didn't talk about an exponential rise of hardware storage and decline in it's prices.
You know, there are already experiments on storing data on a DNA, which can hold 215 million terabytes on a gram of DNA (https://en.wikipedia.org/wiki/DNA_digital_data_storage)

Most of PCs now have a terabyte of free space, it is really cheap. There is no telling on how fast blockchain will rise compared to data storage development, but I wouldn't panic, since there are pretty good options in site, like DNA and decentralized cloud storage like Sia.

[defined]

Most users do not store the full chain anyway.

about 6 years ago when the blockchain was only 1GB in size, nobody imagined back then that it would grow so quickly

This is not true.
Moore's Law says the capacity doubles every 2 years. A new computer has much more space than a new computer 6 years ago.
In 2023, 6 years later, capacity is much bigger again.

[bitdude]

Your idea is very much degraded by the false urgency you present it with. Almost no one will seriously talk with you if you keep that attitude. Also your poll does not provide all options, so you are pushing the voter into one of your predefined scenarios.

As for the idea - it is interesting idea ... for an altcoin. For Bitcoin, it is no go for many reasons.

The primary problem I can see is with that it seems that you do not actually propose (or did I just miss it?) a way to verify that the presented block after pruning is the one that counts. What makes it impossible for someone to come with their own version of compacted state and claim that that is the right state?

[HeRetiK]

I may have missed something, but by the way you describe the problem you could just use pruned nodes as they are, without the extra steps of a Genesis account. Please correct me in my misconceptions.

The main problem being the following point:

[...]

6) Preserve a certain amount of the most recent blocks, just in case a longer blockchain is discovered, and then the Pruned Block should be recalculated.

[...]

That's exactly the reason why pruned nodes don't have the same status as full nodes -- because you can never be fully sure that you preserved the correct amount of blocks before moving on. What if you had to recalculate the pruned block about 10 blocks in? 30? 100? Also cyclical exodus blocks would be an interesting attack scenario for selfish mining.

Apart from that, how would you handle transactions that just entered the network during the exodus block? Ie. that reach their first (or second) confirmation at the exodus block and are thus not yet fully secured?

[...]

And in the near future when the demand keeps growing, the community will be forced to change policy, by either agreeing to larger blocks, or shortening the duration time between blocks.

[...]

Shortening duration time between blocks comes at the risk of decreased security, ie. if you'd reduce block time to 5 minutes you'd have to increase the recommended confirmation count to 12 to reach the same level of security. I'm aware that that's not what you're arguing for, it's just that I see this misconception of shorter blocktimes for faster transactions ever so often that I feel the need to point it out.

[cr1776]

...
and it was predicted back then that in 2020 that size would be 4GB.
...

But Segwit and Segwit2 are only temporary solutions. The main factor in exponential growth isn't policies like Segwit,
...
6) Preserve a certain amount of the most recent blocks, just in case a longer blockchain is discovered, and then the Pruned Block should be recalculated.

...
each account. ...  (Of course, accounts do not really exist in Bitcoin, when I say account what I really mean is a Private/Public Key pair)
...
transactions sent from A to every public key
...
nobody imagined back then that it would grow so quickly
...

I am sure there are those among us with more technical understanding to the nuances of how this idea should be implemented. I am counting on their help to see this through.
...

There seems to be a lot of hyperbole and assumptions in here that many who have a technical understanding of bitcoin would disagree with.  Including some of the following which I see as either misunderstandings of how bitcoin works or something else.  For example:

* Who predicted it would be 4GB in 2020?  Some references to claims would be nice for statements like that.  Many people imagined that the block chain would grow as it has.  Stating that no one did is disingenuous particularly since it is clear using basic multiplication that at a 1MB block size the chain could grow about 52.5GB/year (1MB/block * 6 blocks/hour * 24 hrs/day * 365.25 days).  :-)

* Things such as SegWit enable bitcoin transactions to grow exponentially, not linearly, if necessary via lightning, side chains etc.  Plus using lightning enables micro-transactions which one doesn't need to store on-chain.

* Presumably you mean a longer fork of the block chain?  Not a longer block chain.

* Accounts?  Bitcoin works on inputs and outputs, not accounts (which you did acknowledge), private keys or public keys.  Private keys allow you to spend outputs by demonstrating cryptologic proof that you are permitted to do so.  You can't send to a public key (as above), you have to create a new input or output which out kind of mention later, but referring to it as "sent to a public key" in a technical proposal is just wrong.

* You have to reference a UTXO and somehow all wallets need to know that this could change behind their back at some point and then have some method of tying the two old and new together.

* nLockTime transactions may be impacted which is problematic for people who rely on them.

* Segwit (and other solutions or those that rely on it - lightning, side chains etc) could be impacted if the UTXO set changes behind their backs.

Hopefully you can address some of the concerns that others have and update.  ;-)

[shemtov]

about 6 years ago when the blockchain was only 1GB in size, nobody imagined back then that it would grow so quickly

This is not true.
Moore's Law says the capacity doubles every 2 years. A new computer has much more space than a new computer 6 years ago.
In 2023, 6 years later, capacity is much bigger again.

This is the math, in 2011 we were at 1GB double it each year until 2017. Which is 2^6=64.
64GB is the size of the Blockchain in 2017 if it went at the speed of Moore's law, but we are much higher at ~140GB.

Not to mention Moore's Law is Obsolete, our systems don't grow that fast anymore.

[shemtov]

Thank you for responding intelligibly, I will try to do the same if I can.

Why the Genesis Key Pair?
First of all I should note, that accounts do not really exist, only the transactions themselves. So, there is no A or B or C. There is only A->B, B->C...
Now that I have gotten that out of the way.
If for example we had a series of transactions in which A->B->C->D, and we rewrote to look like A->D,
that would create a problem for us. Because A never signed a transaction sent to D.
And in order to create such a transaction we would need to sign it with A's private key. So A's private key must be known to all those signing with it, and since Bitcoin is meant to be decentralized, which means controlled by Public Consensus in this case, A's Private key must be at the disposal of the Public. Hence the Genesis "Account" (Key Pair).
There is however another way to achieve this result without a Genesis Key Pair, that would be to make it look as though all the coins in the Exodus Block were mined directly to their owners, as though all the bitcoins were newly minted in the Exodus Block, and no bitcoins existed previously. But I like my first idea better, because it seems to be more organized.

What if we had to calculate the Pruned Block ten blocks in?
That is a valid point. It could happen, highly unlikely. The "10" blocks can be a matter of debate, and changed to "n" blocks if it makes you more comfortable. If you want keep 1000 of the older blocks, it would only mean 1GB today, 2GB with Segwit2x, this is not a big deal.

Please Clarify:  What did you mean by Cyclic Exodus Blocks would be an interesting attack scenario?

How would we handle early transactions with one or two confirmations at the time of the Exodus Block?
That is easy, we would rely upon the older blocks that we still store. I proposed 10 in my post, that would mean we could check up to 10 confirmations before the Exodus Block.

Shortening duration time comes at a risk...
I have no disagreement with you on this subject. Some policy will be proposed which will make it possible for more people around the world to use Bitcoin. Highways expand in lanes, so more people can drive on them. Buildings are built taller, so more people can live in the city. This is inevitable, and it has its pros and cons. Bitcoin is no exception. And something must be proposed to allow more Transactions in X amount of Time, which will gradually expand with Bitcoin's Popularity. More Transactions, means more Data per X amount of Time, which means more Hardware Space per X amount of time.

Equation:                          (Transaction Per Block)*(Blocks per Hour)= (Transactions per Hour)

If we are in need of more (Transactions per Hour), either (Blocks per Hour) needs to increase, or (Transactions Per Block).
(Transactions per Hour) is directly correlated to Bitcoin's popularity, which I assume is growing. And I assume nobody in this forum would disagree with me about that.
HeRetik, as for this bothering you... I don't see any way around this, if you do then make sure you are listened to.
We would all gain by it.

[shemtov]

There are those among us who deserve a response, and those who don't.
These are the basic guidelines.

1) Feel free to ask any questions pertaining to my post, I will try to answer them as best as I can.

2) If you think there is a technical problem with my idea which won't allow it to work, please describe the problem as logically as you can, so that I can learn from my mistake and adapt.

3) If you think my idea can technically be applied, but do not think it should be applied:
      a) Clearly state that you think it could work
      b) Say you don't think it should be applied.
      c) Then give the reasoning behind why you think it shouldn't be applied. Provide a scenario, clearly, which shows my idea will not benefit us.[/b]

4) After careful consideration, I have decided not to respond anymore to:
   a) Ankle-Biters : Those who ignore the main idea of my post, concentrating on small petty details, which only disrupt a healthy conversation.
   b) Prophets and Fortune-Tellers : Who love sharing there believes about the future without pointing to any data that can back there belief logically.
   c) Any personal attacks : jokes or sarcastic remarks.
   d) Criticism on my vocabulary, and technical words: Keep your remarks directed toward the logic behind my idea, I don't need an English teacher.
       If you want to teach me to use Bitcoin jargon so that I can better express myself, you can do so privately, there is no need to publicly embarrass me.
       This is just another form of ankle-biting, and its disrespectful.

5) If you agree with my post, I welcome any support I can get.

Thank you for your support!

[shemtov]

1) Here are earlier posts speaking about this problem:

http://bitcointalk.org/index.php?topic=56226.0
https://bitcointalk.org/index.php?topic=505.0
https://bitcointalk.org/index.php?topic=473.0
https://bitcointalk.org/index.php?topic=52859.0
https://bitcointalk.org/index.php?topic=12376.0
https://bitcointalk.org/index.php?topic=74559.15

People who were ignored, back in 2011, you should read how they were speaking back then, and you will see it was an assumption the blockchain would not grow as fast as it did.

2) Segwit can allow linear or exponential growth, you are taking my words out of context. I was talking about the current agreement, and the current agreement of 1MB per block and then 2MB per block is Linear, not exponential. Except agreements change...

3) Reference UTXO?  The UTXO needs to be completely rewritten, it will also be empty, since there will not be any unspent transactions. All funds will be transferred, none will be left associated with the Genesis Private Key, and no other keys will have signed any coins.
Wallets are just Key Pairs, which havn't changed, so no change is needed. Wallets will need to be rescanned, that is all.

4) Transactions with nLockTime would be a problem if they were created before the Exodus Block and not added into a block until after the Exodus Block was created, because the Transactions would be pointing to a previous Transaction which now couldn't be found. I have not found a solution for that as of yet.

5) How don't see how this idea could affect Segwit, if you do, please clarify.

6) Saying "sent to a public key" is wrong.  Yes I was aware of that when I wrote it, but I am trying to keep the language simple, if you have a better way to say it, so that others can understand, I am open to it.

7) You have not said if you think the idea could work, nor have you given any opinion if it should.
    I am not sure what your goal is with your questions.
    Do you acknowledge there is a problem that needs to be solved?
    I think you should of started your reply with these basic concepts, before questioning my idea.
    Having a rational approach is just as important as having rational questions.
   

[defined]

about 6 years ago when the blockchain was only 1GB in size, nobody imagined back then that it would grow so quickly

This is not true.
Moore's Law says the capacity doubles every 2 years. A new computer has much more space than a new computer 6 years ago.
In 2023, 6 years later, capacity is much bigger again.

This is the math, in 2011 we were at 1GB double it each year until 2017. Which is 2^6=64.
64GB is the size of the Blockchain in 2017 if it went at the speed of Moore's law, but we are much higher at ~140GB.

Not to mention Moore's Law is Obsolete, our systems don't grow that fast anymore.

Moore's Law is not an exact number, it is an estimate.
See how much free space your new computer in 2011 had with Blockchain on it. Do it again for your new computer in 2017 and you have much more free space.

[shemtov]

How is it known the Exodus Block is the one that should be used as the previous block?
That is by consensus. By agreeing that every n blocks, we should be creating it. Everyone will create it the same, with the same hash code, and the next block in the chain must reference to it. If the next block in the chain does not reference to it, that block will be rejected.

What makes it impossible that someone will come with their own version of the Exodus block?
The Exodus block can be verified, by all previous blocks including the previous Exodus block. The balances must match, if they don't it won't be accepted as the new Exodus Block. The only way a hacker could come up with his own version, is if he owns more than 51% of the network, which is a problem even today.