Blockchain Compression

[mechs]

   Are there any plans at some point to add blockchain compression technology to the bitcoin client? The blockchain size will continue to increase exponentially as adoption increases and will become a victim of its own success.  It would seem to me adding some compression algorithms to the bitcoin-qt client would be of significant benefit and eventually an outright necessity to maintain the integrity of the system.

[nomailing]

Did you see the threads about Ultimate Blockchain compression?
Here is the concept: https://bitcointalk.org/index.php?topic=88208.0
And here is the current implementation in progress: https://bitcointalk.org/index.php?topic=204283.0

[mechs]

Why is this being done separately from the core development team?

[dillpicklechips]

Why is this being done separately from the core development team?

Probably busy! Always lot's of stuff to do.
And, why not? 

[mechs]

Wouldn't this work best if it was integrated into the official client as opposed to a secondary blockchain?

[Mike Hearn]

"Ultimate blockchain compression" does not compress the block chain. It's misnamed.

The most important piece of the chain is the unspent outputs set (UTXO set). This is already highly compressed using a custom compression algorithm. The rest of the chain is not compressed, but the solution to growth there is called pruning, which allows for deletion of old data.

[TierNolan]

Wouldn't this work best if it was integrated into the official client as opposed to a secondary blockchain?

The idea is to make it backward compatible with all other clients.  If it becomes popular, it could become part of the main standard.

From the thread, there is an intention to merge mine it.  This means that every so often, there would be a block on the main chain that references the alt-chain.  It also means that miners can mine both chains at the same time, so hopefully, it would have similar POW to the main chain.

Having said that, it can't give block rewards, so there is less incentive to merge mine it.

[mechs]

Has the main dev team discussed these ideas at all? I think we all see the blockchain size will increase exponentially as acceptance of bitcoin grows.

[colinistheman]

Would pruning delete bitcoin ledger entries if the person hadn't used it before the date of the pruning?

I guess the obvious answer is of course they wouldn't just make people's bitcoin disappear... but how would you avoid that if you were "pruning" the blockchain??

[hazek]

Has the main dev team discussed these ideas at all? I think we all see the blockchain size will increase exponentially as acceptance of bitcoin grows.

Unfortunately Gavin has different priorities. He intends to solve a problem that was already solved by private companies by implementing it into the reference client instead of focusing on scalability which he knows when push comes to shove he will likely be able to lobby and get enough support for a hard fork that removes the blocksize limit and he basically doesn't care how big the blockchain gets or how few full nodes we have.

[TierNolan]

He intends to solve a problem that was already solved by private companies by implementing it into the reference client

So, what are they working on?

[hazek]

He intends to solve a problem that was already solved by private companies by implementing it into the reference client

So, what are they working on?

An integrated secure invoice system. Something the functionality of which Bitpay and all the other similar platforms already offer.

[TierNolan]

An integrated secure invoice system. Something the functionality of which Bitpay and all the other similar platforms already offer.

Hmm, I would tend to agree that focusing on "core" functionality of bitcoin should be highest priority.  Would the intention be that this includes some kind of protocol change?

[kjj]

He intends to solve a problem that was already solved by private companies by implementing it into the reference client

So, what are they working on?

An integrated secure invoice system. Something the functionality of which Bitpay and all the other similar platforms already offer.

I think your understanding of the payment protocol is incorrect.

[Liquid]

   Are there any plans at some point to add blockchain compression technology to the bitcoin client? The blockchain size will continue to increase exponentially as adoption increases and will become a victim of its own success.  It would seem to me adding some compression algorithms to the bitcoin-qt client would be of significant benefit and eventually an outright necessity to maintain the integrity of the system.

+1 running out of space here 

also i have a question a bit off topic but can a wallet.dat file get corrupted ?

[solex]

"Ultimate blockchain compression" does not compress the block chain. It's misnamed.

The most important piece of the chain is the unspent outputs set (UTXO set). This is already highly compressed using a custom compression algorithm. The rest of the chain is not compressed, but the solution to growth there is called pruning, which allows for deletion of old data.

Yes, you are technically right, but "compression" captures the thrust of the Reiner-Maaku project for public consumption. I see this as the single most important piece of work being done on Bitcoin at the moment as it partly addresses the long-term scalability problem.

[hazek]

He intends to solve a problem that was already solved by private companies by implementing it into the reference client

So, what are they working on?

An integrated secure invoice system. Something the functionality of which Bitpay and all the other similar platforms already offer.

I think your understanding of the payment protocol is incorrect.

You know I really don't care if that's true. Because what I do know is that it's wasting time on a problem that we might have over spending vital time we don't have on a problem that we have RIGHT NOW which frustrates me to no end.

A CEO with his prioritization skills would have been fired long ago.

[kjj]

You know I really don't care if that's true. Because what I do know is that it's wasting time on a problem that we might have over spending vital time we don't have on a problem that we have RIGHT NOW which frustrates me to no end.

A CEO with his prioritization skills would have been fired long ago.

There is certainly room for honest disagreement here.  One could argue that the lack of authentication for addresses and transactions is a problem right now, while the size of the blockchain does not appear to be a problem today, but might become one in the future.

At any rate, Gavin isn't a CEO, he is an engineer.  The payment protocol is something that can surely be solved, while the solution to the blockchain size is unclear.  You can hardly fault an engineer that wishes to solve a problem he knows he can solve in preference to a spinning his wheels on something vague.

[Mike Hearn]

The solution to both problems is clear and Gavin's prioritization is correct. The block chain is only 8 gigabytes today. That's nothing. If you can't keep up with that then just switch to MultiBit and you'll not need to store the chain any more. It'll be much faster as well.

Again, the "ultimate blockchain compression" thing doesn't make any technical sense, that's why it's not happening. Pruning is something that Pieter Wiulle has said he wants to work on. Most of the work was already done. It requires some protocol upgrades and some other tricky work, then you will probably be able to tell Bitcoin-Qt/bitcoind how much disk space you want to give it. It'll store as much of the chain as it can within that space.

The payment protocol is foundational work, it is required for a lot of other features. That's why it's important. Pruning is a needed scalability fix but we're far from having some kind of crisis because people can't afford the disk space. A lot of people blow more than 8 gigs of disk space on apps they simply forgot about.

[hazek]

The solution to both problems is clear and Gavin's prioritization is correct. The block chain is only 8 gigabytes today. That's nothing. If you can't keep up with that then just switch to MultiBit and you'll not need to store the chain any more. It'll be much faster as well.

Yes, let's screw decentralization while we are at it, right? I know you wouldn't mind Bitcoin becoming controlled by a hand full of super nodes..