The way the initial block download works, your client goes and asks somebody, "what's the current block", and they say, BLOCK ID XXXXX. And then since you don't have XXXXX, you ask them for that and they send it to you, but after you download it, then you look at its block header and discover that the previous block was WWWWW - and you don' t have that one either! So you ask, "Hey could you give me a copy of Block WWWWW?" And they send that, and you look at it, and its parent was VVVVVV, and you don' t have that either, so you ask..... and so on, back to the Genesis block.
No it doesn't.
If you had blocks from 0 to 10000, you would send something like.
I have (hashes of)
block 10000
block 9999
block 9998
....
block 9990
block 9900
block 9000
block 8000
block 5000
block 2000
block 500
block 0
The peer then scans the list and starts with the one first on the main chain. If your block 10000 wasn't on the main chain, but block 9990 was, then it would start at block 9990.
Normally, it starts from the most recent block you have, unless you are somehow on a fork of the main chain.
Then there's a CPU bottleneck to worry about. Your peer does all kinds of cryptographic checks on each downloaded block, so if your CPU is slow, that may even dominate the latency time.
This is generally the big one.