Making sync more efficient

[desired_username]

I wonder if it would be possible to make the synchronizing process faster, using torrent like services for example? I would provide server space for it if needed (I'm sure a lot more volunteers would show up).

I just set up an offline wallet with a presistent knoppix linux, but the sync run for about 17 hours and completed only 76% so far

The network connection is completely fine, getting 50mbit on speedtest.

[justusranvier]

I wonder if it would be possible to make the synchronizing process faster, using torrent like services for example?

Try searching the forum for the word "torrent".

[grue]

tl;dr version: no, it's not good to have a torrent of the blockchain, because it leads to centralization (who to trust for the torrent?). Also, blockchain is updated regularly, so the torrent will have to be changed quite often, which causes it to loose all seeders.

[TierNolan]

tl;dr version: no, it's not good to have a torrent of the blockchain, because it leads to centralization (who to trust for the torrent?). Also, blockchain is updated regularly, so the torrent will have to be changed quite often, which causes it to loose all seeders.

In fairness, he said "torrent like".

My understanding is that the default client downloads everything from 1 other client.

A better (torrent-like) process would be something like

Download header chain

Get 100 header hashes from each other node you are connected to
- You need to do this in order
-- ask first node for 100 hashes starting at block 0
-- ask second node for 100 nodes starting at 100 (you need to send the hash for 99)
-- ask next for 100 more etc

When you have a set of 100 header hashes from a node, you can then ask the node to send you the block headers in full.

When downloading the 100, you can verify that they make up a chain.

The result is that you get lots of 100 length sub-chains.

You then have to merge them and make sure they form a chain too.

When a node has sent you the 100 block headers, you can ask it for another 100.

This gets you all the block headers

Download blocks

Next you need to download the blocks.  Again, ask each other node for blocks in groups of 100.

When a block is received, you can check that it matches the header for that block.

Verify blocks

Once you have all blocks that occur before a given set of 100 blocks, you can then add them to the verified set.  This requires checking all the encryption.

This is apparently, the big processing bottleneck.

[TierNolan]

Opps .. double posted instead of editing.

[desired_username]

tl;dr version: no, it's not good to have a torrent of the blockchain, because it leads to centralization (who to trust for the torrent?). Also, blockchain is updated regularly, so the torrent will have to be changed quite often, which causes it to loose all seeders.

In fairness, he said "torrent like".

My understanding is that the default client downloads everything from 1 other client.

A better (torrent-like) process would be something like

Download header chain

Get 100 header hashes from each other node you are connected to
- You need to do this in order
-- ask first node for 100 hashes starting at block 0
-- ask second node for 100 nodes starting at 100 (you need to send the hash for 99)
-- ask next for 100 more etc

When you have a set of 100 header hashes from a node, you can then ask the node to send you the block headers in full.

When downloading the 100, you can verify that they make up a chain.

The result is that you get lots of 100 length sub-chains.

You then have to merge them and make sure they form a chain too.

When a node has sent you the 100 block headers, you can ask it for another 100.

This gets you all the block headers

Download blocks

Next you need to download the blocks.  Again, ask each other node for blocks in groups of 100.

When a block is received, you can check that it matches the header for that block.

Verify blocks

Once you have all blocks that occur before a given set of 100 blocks, you can then add them to the verified set.  This requires checking all the encryption.

This is apparently, the big processing bottleneck.

This.

I didn't know that it downloads it from 1 other client. :S

it's not that surprising to have 1+ days download time now.

What is the status of the fix? has it been considered to make it better?