Bitcoin Forum

Bitcoin => Bitcoin Technical Support => Topic started by: CyberDain on November 13, 2017, 04:24:28 PM



Title: BitCoin core writes too fragmented data
Post by: CyberDain on November 13, 2017, 04:24:28 PM
at most 100000 fragments after full sync

https://i.imgur.com/iPDBITz.png
can you fix it?


Title: Re: BitCoin core writes too fragmented data
Post by: achow101 on November 13, 2017, 04:49:23 PM
This is not something that Bitcoin Core can control. It is entirely based upon the filesystem and how it allocates space to store data. Bitcoin Core itself has no control over that and cannot control where data is physically located on disk.


Title: Re: BitCoin core writes too fragmented data
Post by: CyberDain on November 13, 2017, 05:20:27 PM
disagree. Writing many files simultaneously produce huge amount of fragmented files, that cause slow sync.


Title: Re: BitCoin core writes too fragmented data
Post by: achow101 on November 13, 2017, 05:34:53 PM
disagree. Writing many files simultaneously produce huge amount of fragmented files, that cause slow sync.
The files are not written simultaneously.


Title: Re: BitCoin core writes too fragmented data
Post by: CyberDain on November 13, 2017, 06:13:14 PM
may be.
But DB files too fragmented.
and for example uTorrent writes clean continuous blocks from beginning of drive as you can see on picture,  even thousand files in torrents.
So I think this is bug of bitcoin core .


Title: Re: BitCoin core writes too fragmented data
Post by: LoyceV on November 13, 2017, 07:55:13 PM
may be.
But DB files too fragmented.
and for example uTorrent writes clean continuous blocks from beginning of drive as you can see on picture,  even thousand files in torrents.
So I think this is bug of bitcoin core .
uTorrent allocates space upfront, for a file with a known file size. Bitcoin Core writes files year after year, it writes wherever your disk has some space available at that moment. And since you use your computer for other tasks too, the "empty spots" on your disk change location. It would make no sense to reserve 500 GB upfront when you first install Bitcoin Core, to prevent fragmentation in the years to come.

The easiest solution is simply not to worry about it, your computer doesn't need to defragment files all the time.
But if you really want to, move Bitcoin Core to a different disk, and it will be completely sequential.

The sync speed - in my experience - mainly depends on your system's memory, and how efficiently your computer uses it. For comparison: I get about 1 block per second, on an i3 with hdd, Linux and 12 GB RAM.


Title: Re: BitCoin core writes too fragmented data
Post by: achow101 on November 13, 2017, 10:51:32 PM
may be.
But DB files too fragmented.
and for example uTorrent writes clean continuous blocks from beginning of drive as you can see on picture,  even thousand files in torrents.
So I think this is bug of bitcoin core .
No, you just don't understand how Bitcoin Core works. It is not like uTorrent where the file sizes are known and it can allocate space accordingly. Rather what happens is that space is allocated for each block as it arrives. The blockchain is not one massive file that Bitcoin Core must download. It is a bunch of individual blocks that are recorded to disk separately. This occurs for both initial sync and during normal operation because it does not know the full blockchain size or even block sizes beforehand. It just allocates space as it needs it. Since it is constantly appending to files and requesting more space as it appends to each file. This is why it cannot use a continuous space: it does not know how much it will need and cannot request it all. Furthermore, as LoyceV points out, because Bitcoin Core is designed to run continuously and must write new blocks to disk as they are found, it will result in some fragmented data since it reserves space as it needs it, and presumably you are doing other stuff on the computer that also reserves space as it needs it.


Title: Re: BitCoin core writes too fragmented data
Post by: CyberDain on November 14, 2017, 04:05:50 AM
I supposed size of files are approximately known

https://i.imgur.com/Z3sALsK.png


Title: Re: BitCoin core writes too fragmented data
Post by: CyberDain on November 19, 2017, 08:11:52 AM
so I think best practice is shrink HDD to 200gb for BTC db, then BTC will not write to middle of whole drive and decrease overall system performance.


Title: Re: BitCoin core writes too fragmented data
Post by: LoyceV on November 19, 2017, 08:24:58 AM
so I think best practice is shrink HDD to 200gb for BTC db, then BTC will not write to middle of whole drive and decrease overall system performance.
Why do you think some fragmentation reduces your overall system performance? It only influences sequential reads, which is something you barely do, unless you're copying large files.
If you create a separate partition on the same disk, the read arm of the hdd has to travel to that location anway, in those cases is can be faster if the data is all closer together.
Your 200 GB partition will be filled soon anyway, after which you'll need to increase it again.

If you really want to increase your system performance, install a SSD.


Title: Re: BitCoin core writes too fragmented data
Post by: toastyhamster on November 19, 2017, 04:24:50 PM
What's typical sync time?  Due to space constraints and for availability I'm experimenting running Core on a low spec Celeron with RAID space for media etc, currently on day 3 of sync with 2 more left apparently!

Fragmentation looks ok but I did start with loads of disk space which would have helped.


Title: Re: BitCoin core writes too fragmented data
Post by: slate_main on November 21, 2017, 06:09:26 PM
What software is that fascinating looking drive mapping/sector tool?


Title: Re: BitCoin core writes too fragmented data
Post by: CyberDain on November 22, 2017, 10:56:24 AM
What software is that fascinating looking drive mapping/sector tool?
UltimateDefrag