Is anybody working on pruning on the main client?

[MoonShadow]

Just curious, because I've been getting a lot more questions about the resident size of the blockchain.

[1714800527]

1714800527

[1714800527]

1714800527

[Sukrim]

The last I heard about active pruning was https://bitcointalk.org/index.php?topic=204283.0

Other than that supposedly bitcoind already can prune, but it is not really exposed out of fear that there might be too many people actually doing it which would hurt chain distribution. Also header-first sync might allow for some nice features (e.g. keep the last N blocks + a random sample of the older ones, up to size X). I don't know about progress there though, in general I would just advise to invest the 2 USD or whatever a few dozen GB of HDD space costs these days and just store all blocks.

If you have such an old PC that a few GB are of concern, you might want to think of switching out harddisks anyways, especially when storing wallet files on there...

[MoonShadow]

Other than that supposedly bitcoind already can prune, but it is not really exposed out of fear that there might be too many people actually doing it which would hurt chain distribution.

Can you point me to more details about this?  I'm having trouble finding it with the search functions.

[gmaxwell]

, but it is not really exposed out of fear that there might be too many people actually doing it which would hurt chain distribution.

Thats not _quite_ accurate. The P2P protocol has no way to communicate which nodes have which block other than a binary state for "full node or not" which implies all the block. To have pruning we first must change the p2p protocol to communicate nodes that are full nodes but can only serve recent blocks (+some named subset of the history, most likely). There are also a bunch of other minor details like refusing to serve blocks it doesn't have instead of just crashing on the request.

[Sukrim]

Well, a pruned node would anyways report as non-full node, right?

A first step could have been to have pruned nodes that store no history at all, then start on "hybrid" nodes, that store some history for the next iteration. Why the approach the other way round?

[nomailing]

Well, a pruned node would anyways report as non-full node, right?

A first step could have been to have pruned nodes that store no history at all, then start on "hybrid" nodes, that store some history for the next iteration. Why the approach the other way round?

I guess, because then you might loose a lot of full nodes completely in a very short time frame and then build them up again slowly. So you would have to bridge a time window of instability and untested conditions.
Therefore the other way around is better, where you try to slowly shift from full-nodes and slightly reduce their history step-by-step.

[piotr_n]

, but it is not really exposed out of fear that there might be too many people actually doing it which would hurt chain distribution.

Thats not _quite_ accurate. The P2P protocol has no way to communicate which nodes have which block other than a binary state for "full node or not" which implies all the block. To have pruning we first must change the p2p protocol to communicate nodes that are full nodes but can only serve recent blocks (+some named subset of the history, most likely). There are also a bunch of other minor details like refusing to serve blocks it doesn't have instead of just crashing on the request.

right.
changing the p2p protocol - that sounds like a really tough one.
seems like at least 10 years of work, for you guys.
but the good thing is that you have already spent at least the last two years on analyzing the problem, so there is a decent chance that it will get done within the next eight...

though, more likely scenario is that within the next 8 people will forget about the original bitcoin dev team and start making/using their own mods.
it's actually already happening - has happened while you were busy with designing black and red lists during a conference with US financial "authorities", otherwise known as the core of the wold's financial regime

[Peter Todd]

though, more likely scenario is that within the next 8 people will forget about the original bitcoin dev team and start making/using their own mods.
it's actually already happening - has happened while you were busy with designing black and red lists during a conference with US financial "authorities", otherwise known as the core of the wold's financial regime

Mike != the dev team.


Anyway, Litecoin contracted me to implement pruning, although there's no particular timeline on that contract; it depends on sipa's headers-first which has its own set of issues.

[piotr_n]

Mike != the dev team.

Just because you're looking at a guy raping a girl, while doing nothing to stop it - it does not make you innocent in the crime.
Unless he was keeping a gun aimed at you, while you were watching it, was he?

[piotr_n]

Anyway, putting strife away and focusing on the job that eventually has to be done and will be done by someone, some day.

There is only one ultimate purging solution which does not affect the network's decentralization and address the scalability issues.
Start distributing snapshots of UTXO database, with the snapshots' security protected by the blockchain and the miners.
But that's definitely too far fetched idea, as for this team - 25 years at least, 10 of which just to realize that all the other options suck...
I'll be using viagra, instead of the memory impairing drug, by then

[piotr_n]

You know, it's not that I care very much to increase my ignore digit... it just happens out of my control.
But let's be realistic and someone finally has to say it.

Almost five thousands years ago, it took Egyptians about twenty years to build the Great Pyramid of Giza.
If the current bitcoin elite, put into power by the satoshi himself, will manage to solve the ever known bitcoin scalability issues in less then that, then you guys will have a reason to be really proud of yourself... Though honestly, looking at your progress from the past couple of years, I have serious doubts about you meeting the deadline

[Peter Todd]

Mike != the dev team.

Just because you're looking at a guy raping a girl, while doing nothing to stop it - it does not make you innocent in the crime.
Unless he was keeping a gun aimed at you, while you were watching it, was he?

Nah, we're just subtle about it, the kind of subtlety that involves orchestrating an angry mob to stop the rapist rather than doing so ourselves. More concretely, remind yourself again about who's been behind the latest interest in CoinJoin; I personally just spent a week at the DarkWallet hackathon.

Anyway, putting strife away and focusing on the job that eventually has to be done and will be done by someone, some day.

There is only one ultimate purging solution which does not affect the network's decentralization and address the scalability issues.
Start distributing snapshots of UTXO database, with the snapshots' security protected by the blockchain and the miners.
But that's definitely too far fetched idea, as for this team - 25 years at least, 10 of which just to realize that all the other options suck...
I'll be using viagra, instead of the memory impairing drug, by then

Look up "MMR TXO commitments" among other things - we're way ahead of you mate.

FWIW I've been hired by Mastercoin to work full-time on crypto-coin research - scalability will definitely be one of the focuses of my work.

[piotr_n]

Look up "MMR TXO commitments" among other things - we're way ahead of you mate.

FWIW I've been hired by Mastercoin to work full-time on crypto-coin research - scalability will definitely be one of the focuses of my work.

Well, I would be surprised if you were not ahead of me, since I have not even worked on this.
And I cannot work on this, since it is literally impossible to change a peer-to-peer protocol while being  the only peer on the network.
There were times when I had my ideas and energy to work on such things, but you guys seem to always know better and it doesn't seem like you need anyone new in the team.
And now, after the latest red list revelations, it's even a team that I would be ashamed to join, so no thanks and good luck.

Anyway, all I am saying is that it goes very slowly and as much as I believe that some members of your team are sincere, competent and professional - there are also others who are intentionally disturbing to drive this project in the right direction, often using a dirty methods. But then you defend them when I point it out.
You obviously act as a team and you cannot blame people for judging you as a team.
Plus, you cannot expect others to not beat your team in the competition, considering that to beat you on a software development field does not seem like an extreme challenge, even for a single developer, not to mention an actual team.

So my advise to the bitcoin elite: you guys either take your shit together and start doing your job right, or you will be out of this business soon - meaning that even the admiration from people who have no idea how lame this project's development has recently been will be gone within a single day.

[wtogami]

The Litecoin Dev team did contract with Peter Todd to do all the work necessary to implement pruning and to have it submitted to Bitcoin github for code review.  This includes pruning and various supporting code that has been proposed by others.  For example, "archive nodes" would store a full copy of the blockchain.  "Fully verifying nodes" no longer need the entire blockchain, but for the network to be healthy and capable of syncing new clients they need to keep subsets of the blockchain. There also needs to be a way to tell peers what ranges of blocks you have available.

While it would be bad to allow peers to have less than a full blockchain, this is a balancing act.  With the rapid growth of the blockchain the quantity of listening nodes on the network has been continually shrinking.  Perhaps fully verifying nodes that store subsets of the blockchain would allow more listening nodes to exist comfortably on the global network.

[Mike Hearn]

A contract with no timeline?

[grv]

The last I heard about active pruning was https://bitcointalk.org/index.php?topic=204283.0

Other than that supposedly bitcoind already can prune, but it is not really exposed out of fear that there might be too many people actually doing it which would hurt chain distribution. Also header-first sync might allow for some nice features (e.g. keep the last N blocks + a random sample of the older ones, up to size X). I don't know about progress there though, in general I would just advise to invest the 2 USD or whatever a few dozen GB of HDD space costs these days and just store all blocks.

If you have such an old PC that a few GB are of concern, you might want to think of switching out harddisks anyways, especially when storing wallet files on there...

and this line of reasoning is why everything gets so unnecessarily bloated.
multiply 10 gigs by millions of users (and billions of devices) and the result is a HORRIBLE WASTE of resources
(client hardware, internet bandwidth, electricity for all hardware involved, etc)
this is a big problem which might actually prevent global bitcoin adoption
average user wants to run a simple client and start using it right away.
what he gets right now is hours/days of waiting for "sync" to finish.
unacceptable if you're looking for widespread adoption by non-tech population
not to mention its already 10 #@$ gigs and its been only few years,
with the volume of transactions only a fraction of what will be coming in the future.
do you expect every single client to store 100 terabytes and wait for months to sync??? lol

[behindtext]

right.
changing the p2p protocol - that sounds like a really tough one.
seems like at least 10 years of work, for you guys.
but the good thing is that you have already spent at least the last two years on analyzing the problem, so there is a decent chance that it will get done within the next eight...

hi there piotr!

i can't... stop... laughing. LOL

[gmaxwell]

do you expect every single client to store 100 terabytes and wait for months to sync??? lol

No, this is why SPV mode supporting clients exist. Perhaps you should refrain from posting in the technical subforum until you're a little more aware of how the Bitcoin system works. Also, please try to treat the other contributors with a bit more respect.  Responses like " lol" from clueless people drives away competent contributors.

[grv]

do you expect every single client to store 100 terabytes and wait for months to sync??? lol

No, this is why SPV mode supporting clients exist. Perhaps you should refrain from posting in the technical subforum until you're a little more aware of how the Bitcoin system works. Also, please try to treat the other contributors with a bit more respect.  Responses like "??? lol" from clueless people drives away competent contributors.

yea sorry about that, but arrogant and narrowminded people just annoy me sometimes.
(judging by the quote: "If you have such an old PC that a few GB are of concern").
i know about spv mode clients, but until every major client is such and featured prominently in download sections, this has the potential of driving away most of the average people.

[Sukrim]

Sorry, but if you want to run Bitcoin-QT on a local machine, it will be easily able to handle the few GB (current computer games are MUCH larger) of space and traffic. Otherwise you can also use prominently featured lighter clients like Electrum or Multibit.

Please tell me exactly why me reasoning that having a block chain that contains 4 years(!) of binary transaction data and still is half the size of a current computer game (random google, yes I know it's XBox One...: http://callofduty.digitalwarfare247.com/2013/11/call-of-duty-ghosts-is-a-39gb-install-on-xbox-one/) is "arrogant and narrow minded"?!

Currently the block size limit is ~1 MB, which gives a constant hard cap of about 55-60 GB of transaction data per year. This can be stored on a USB drive that you can get for 30 USD with free shipping (again random google, I am sure you can get it cheaper too: http://www.amazon.com/SanDisk-Cruzer-Flash-Drive-SDCZ36-064G-AFFP/dp/B007JR5304).

How the hell is this amount of storage going to drive away "most of the average people"?