Block headers download

[r.willis]

I have trouble implementing parallel blockchain download (headers only). I read wiki back and forth, but do not understand:
1) What exactly is block locator object.
2) How to download headers before certain block (towards genesis block)
Now I can start from genesis and work up blockchain (using 0 as hash_stop). Is it possible to move in opposite direction (from newest blocks to genesis)?

[TierNolan]

I have trouble implementing parallel blockchain download (headers only). I read wiki back and forth, but do not understand:
1) What exactly is block locator object.
2) How to download headers before certain block (towards genesis block)
Now I can start from genesis and work up blockchain (using 0 as hash_stop). Is it possible to move in opposite direction (from newest blocks to genesis)?

You are supposed to start at the start and work forward.  The locator object is just hashes.  You effectively have to give the hash of the start and finish.  You can give extra hashes.  You are sending a list of hashes that you know about.  The other node will scan them, and start at the hash that has the highest height and is on the main chain.

You could use the getblocks message.  They returns the hash of the blocks in an "inv" packet.  However, that is 33 bytes long per entry, so you aren't really saving much.

For parallel download what is needed is a message that returns the hash of a block for a set of heights.

[Zeilap]

I have trouble implementing parallel blockchain download (headers only). I read wiki back and forth, but do not understand:
1) What exactly is block locator object.
2) How to download headers before certain block (towards genesis block)
Now I can start from genesis and work up blockchain (using 0 as hash_stop). Is it possible to move in opposite direction (from newest blocks to genesis)?

Block locator is a list of block hashes that you know about, starting from your head, back towards the genesis block. The idea is that the remote peer can know which blocks/headers to send you. If it determines that your most recent blocks are actually now on a side chain, then it will work backwards through your block locator to find a block which is on the main chain and then send you blocks from there onwards. If it cannot find any blocks in your locator which are on the main chain, it will send you blocks from the genesis block.
Current implementations create a block locator roughly as follows,

Code:

 
var i = current head height
var stepSize = 1

while(i > 0) {
    insert the hash of the block at height i
    i = i - stepSize
    stepSize = stepSize * 2
}

So many more recent blocks are added, and as you work back, the steps get bigger (because of the decreased likelihood of deeper and deeper reorgs).
When downloading the headers before the last checkpoint, you can simply use the hash of the most recent header you have as the locator, i.e. only one hash.

EDIT:
As for working backwards, this is not possible. What is possible however is to hardcode checkpoint hashes, this way you can download the headers between each checkpoint separately enabling your parallel download. I have code to do exactly this, but I haven't bothered to test it to see if the speed improvement was worth it compared to downloading from a single peer. Others may see this as a waste of network resources perhaps.

[r.willis]

TierNolan, Zeilap - thanks. Now I understand better.
It seems the only option to implement parallel download is to provide multiple checkpoints - if I want N parallel requests, I must hardcode N somewhat evenly-spaced checkpoints.

[Zeilap]

TierNolan, Zeilap - thanks. Now I understand better.
It seems the only option to implement parallel download is to provide multiple checkpoints - if I want N parallel requests, I must hardcode N somewhat evenly-spaced checkpoints.

Yes exactly  - see edit

[r.willis]

Others may see this as a waste of network resources perhaps.

Why? It's the same total amount, isn't it?
Btw, even is sequential download I select peer to send getheaders randomly - this way I create more even load on peers. After I got headers, I can get blocks using getdata or getblocks with much better control.

[Zeilap]

Well it depends how lazy you are . If you share the downloads between the peers without connecting to more peers, then no problem. If you purposely connect to as many peers as there are checkpoints, then it's not the same, especially if you don't bother to disconnect from most of them after you've finished downloading.