Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: btc123thatthere on April 22, 2019, 05:34:10 AM



Title: Differing .dat file sizes on Pi node vs. Windows nodes
Post by: btc123thatthere on April 22, 2019, 05:34:10 AM
Hi

So I've got three Bitcoin Core nodes which are all the latest version:
1) Running on a windows 7 machine
2) Running on a windows 10 machine
3) Running on a Raspberry Pi running UbuntuMate (up to date - is always running. Created about 10 days ago with copied files from win 10 node. Works fine.)

Originally, I was using the windows 10 machine to keep an updated copy.
To create the Pi node, I copied the files over from the windows 10 machine which worked fine.

Today I thought I would update the windows 7 node (a few days behind) by copying the latest 'blocks' files and copying the whole 'chainstate' folder.
I noticed that not all of the previous .dat files were the same size (more than just the latest on the out-of-date node) and I have no idea why.

It appears that the Pi node changed some of the files that I copied over from the windows 10 node for it, and now the win 7 machine can't use them.
Anyone know why it changed the files?

It's a bit annoying really as it means it's no use as a source of up-to-date blocks & chainstate files.
Update: I managed to get the win 7 version working by wiping its blocks and chainstate folders and copying those folders from the Pi's hard drive (to the win 7's hard drive). I noticed that after blk01483.dat, all the files are different sizes on the Pi copy compared to the win 10 copy. I still don't know why this is, but it's good to see that the Pi copy is usable nonetheless.



Title: Re: Differing .dat file sizes on Pi node vs. Windows nodes
Post by: achow101 on April 22, 2019, 05:04:43 PM
Today I thought I would update the windows 7 node (a few days behind) by copying the latest 'blocks' files and copying the whole 'chainstate' folder.
That is guaranteed to not work. Nodes download blocks out of order so they will get written to disk in a different order which depends on the order that blocks were received. If you copy the database from one node to another, that database will be incorrect and say that blocks are exist at a particular location when they actually don't. Furthermore, you didn't copy the block index which is in the index folder inside of the blocks folder. That actually says where blocks are actually located and, IIRC, the chainstate database contains references to the block index. If you want to copy the blockchain from one node to another, you must copy the entirety of the blocks folder and the chainstate folder so that the correct databases and indexes  are included.

I noticed that not all of the previous .dat files were the same size (more than just the latest on the out-of-date node) and I have no idea why.
Not all blocks are the same size and not all .dat files contain the same blocks. As such, not all .dat files will be the same size. As I mentioned, blocks are downloaded out of order. Furthermore, one node may have received blocks that another node has not. Thus the size of the .dat files will vary based on the contents of the file.


Title: Re: Differing .dat file sizes on Pi node vs. Windows nodes
Post by: btc123thatthere on April 23, 2019, 09:54:13 AM
So what worked for me was:
- Closing bitcoin core on the Pi (bitcoin-cli stop).
- Accessing the 'blocks' and 'chainstates' folders on the Pi HDD from the Win 7 machine using Samba on the Pi (https://help.ubuntu.com/community/How%20to%20Create%20a%20Network%20Share%20Via%20Samba%20Via%20CLI%20(Command-line%20interface/Linux%20Terminal)%20-%20Uncomplicated,%20Simple%20and%20Brief%20Way! (https://help.ubuntu.com/community/How%20to%20Create%20a%20Network%20Share%20Via%20Samba%20Via%20CLI%20(Command-line%20interface/Linux%20Terminal)%20-%20Uncomplicated,%20Simple%20and%20Brief%20Way!)).
- Making sure bitcoin core wasn't running on the win 7 machine.
- Updating the win 7 copy from the Pi's copy using SyncToy (Pi left via network and Win 7 copy on right, using Echo mode.)

After which I started up the win 7 bitcoin core, let it do its thing and it's all synced up no problem.

Cheers