Downloading pruned blockchain inefficient?

[SebastianJu]

Hello,

I just download the blockchain in pruned mode. I downloaded the unpruned blockchain some days ago. It was way faster and I notice that downloading a pruned blockchain now takes way longer. It looks like takes more cpu and it uses the harddisc more often. It isn't the swap file also and I gave bitcoin-qt 6gb RAM and 16 threads. The same setup. Antivirus is also not running.

So can it be that the pruning is constantly done and writing and reading from the harddisc in an inefficient way? The same way like it works with single blocks when it is already synchronized? If so then maybe it would make sense to make the prune work just every couple of minutes. This might save ressources then.

Greetings!

[TexasHuck]

This has been my experience as well.  I was thinking there could be some type of issue with the indexing of the values retained from the prune.  I would interested to see a play by play of the prune in action.  Even pseudo code would be helpful... thanks for posting!

Huck

[achow101]

Pruning will be more taxing on your disk as it has to constantly read and write to the disk far more than without pruning. This is because it deletes things on the fly. That means that, for each block, it will need to additionally compute what blocks should be deleted, look up where the block is, delete it, and remove it from the database. That will use extra CPU, RAM, and disk IO as there are simply more things that Core is doing in order to prune.

[tspacepilot]

I was away from following development on core for some time.  Last time I was up-to-speed pruning was just starting to be implemented.  @achow101: where I can read about the pruning implementation (obviously I can study the source, but I suppose there's a writeup somewhere)?

[achow101]

@achow101: where I can read about the pruning implementation (obviously I can study the source, but I suppose there's a writeup somewhere)?

AFAIK, there is no write up of how pruning is implemented. You will have to look in the source code for that. Since manual pruning is now available via an RPC command, I would recommend that you start looking there to see what it does to prune: https://github.com/bitcoin/bitcoin/blob/master/src/rpc/blockchain.cpp#L849

[-ck]

If you want to run a pruned node but have the storage capacity to download the whole blockchain it is indeed much faster to download the whole blockchain and then restart bitcoind in pruned mode to do the pruning afterwards since it is constantly being pruned during the download process otherwise.

[tspacepilot]

@achow101: where I can read about the pruning implementation (obviously I can study the source, but I suppose there's a writeup somewhere)?

AFAIK, there is no write up of how pruning is implemented. You will have to look in the source code for that. Since manual pruning is now available via an RPC command, I would recommend that you start looking there to see what it does to prune: https://github.com/bitcoin/bitcoin/blob/master/src/rpc/blockchain.cpp#L849

Thanks!  I also rhappened to re-look over the original whitepaper and I noticed (for the first time) the section on "reclaiming disk space".  I guess I'm trying to figure out what the disadvantages are (if any) over running a pruned node vs non-pruned.

EDIT: I found some further notes about pruning in the release notes for 0.11: https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#block-file-pruning

[SebastianJu]

Thanks for the answer. It might be useful for the future to implement something like
if more than 100 blocks behind then download normally 50 blocks then prune on those. Repeat.

[gmaxwell]

If you want to run a pruned node but have the storage capacity to download the whole blockchain it is indeed much faster to download the whole blockchain and then restart bitcoind in pruned mode to do the pruning afterwards since it is constantly being pruned during the download process otherwise.

Have you actually measured that? If it's true-- it's a bug. the pruning should delete a whole blockfile at a time (128MB) and shouldn't take much time at all.  It's totally plausible to me that there is a bug here, but I've not noticed it myself or heard it reported before.

[-ck]

If you want to run a pruned node but have the storage capacity to download the whole blockchain it is indeed much faster to download the whole blockchain and then restart bitcoind in pruned mode to do the pruning afterwards since it is constantly being pruned during the download process otherwise.

Have you actually measured that? If it's true-- it's a bug. the pruning should delete a whole blockfile at a time (128MB) and shouldn't take much time at all.  It's totally plausible to me that there is a bug here, but I've not noticed it myself or heard it reported before.

Only when pruning was first available did I experience this and I did not do a fair comparison of identical hardware so no I cannot say for certain. I've not downloaded the whole blockchain since.