how to maximize block download speed on a single local node

[Nicolas Dorier]

I noticed that fetching blocks over the bitcoin protocol is slow in my setting.
My node is in my local network which have over 100MB speed with my dev computer.

I find no way of making it better, and I'd like your advice, here is my finding:

• Average Speed Achieved : 5MB/s
• Max speed Achieved (ponctual) : 15MB/s
• Fetch strategy : getdata of 50 000 blocks, Tip to Genesis
• Client's socket receive buffer is never full (indicate bitcoind does not send fast enough)
• Disk IO and CPU not capped
• When the rcv buffer of the client is empty, it takes 100ms with low variance before new data come
• Using 2 connections instead of one for fetching blocks speed things up a little bit (7.5MB/s average)

I tried to play with receive and send buffer on both the bitcoind and the client code, no effect.

I noticed that the 100ms time it take for my buffer to be filled again must come from ThreadMessageHandler.

However by reviewing the code, I have seen no mistake : if the send buffer of the bitcoin node is not full, I should not experience any 100ms delay.
Once again, trying to change the buffers size of bitcoind had no effect.

Any idea ?

My alternative is to parse the blk directory over SMB, which go near 75MB/S.
But this is a fragile solution I'd like to avoid.

[1713989050]

1713989050

[1713989050]

1713989050

[1713989050]

1713989050

[1713989050]

1713989050

[1713989050]

1713989050

[1713989050]

1713989050

[cr1776]

Hi,
A few questions:
1. Is this with Bitcoin Core 0.10.1?
2. Are you only syncing the block chain over your local network to one other machine?  If you can sync from multiple machines, that can speed things up.
3. HDD or SSD on the machines?
4. CPU specs?

:-)

[TierNolan]

However by reviewing the code, I have seen no mistake : if the send buffer of the bitcoin node is not full, I should not experience any 100ms delay.
Once again, trying to change the buffers size of bitcoind had no effect.

It looks like it will only skip the sleep if at least one node received data and it is unprocessed.  I don't why that is connected to the sending buffer.

Have you tried dropping the 100ms sleep to 1ms?

[NeuroticFish]

There are a lot of conditions there. I'd also try to set for a start the sleep to a very low value and if there is a real improvement I'd debug a little.
It's too easy to miss something there.

[Nicolas Dorier]

Is this with Bitcoin Core 0.10.1?

Yes

Are you only syncing the block chain over your local network to one other machine?  If you can sync from multiple machines, that can speed things up.

Yes, but this is not what I want to do : I am currently benchmarking my code on NBitcoin, and noticed that the download was slow and that my receive buffer was never full, which indicate bitcoind is slow.

HDD or SSD on the machines?

4 HDD in RAID5, 10K RPM (the server has his own RAID controller)

CPU spec

I own a Dell Powerdge T620, the CPU is not capped, nor the Disk IO.

Have you tried dropping the 100ms sleep to 1ms?

No, generally speaking, everytimes I tried compiling bitcoind, I gave up after several hours wasted and frustrated, so I just imagine this is caused by the sleep. The 100ms timing seems very consistent.

There are a lot of conditions there. I'd also try to set for a start the sleep to a very low value and if there is a real improvement I'd debug a little.

Thanks, well, I guess I will have to try compiling bitcoin one more time on windows.

I was wondering if I was missing anything obvious, any "magic parameter" somewhere.

[2112]

The 100ms timing seems very consistent.

I was wondering if I was missing anything obvious, any "magic parameter" somewhere.

TCP_NODELAY?

Interestingly John Nagle has an account on this forum:

https://bitcointalk.org/index.php?action=profile;u=28488

Edit: I'm actually posting this from Windows, not my normal machine. It took me a while to locate the place where the OS include files are located. In a search for the correct spelling I found a funny comment in one of the Windows' SDKs:

Code:

// turn off nagling

Obviously making a verb out of somebody's last name is a sign of respect. I'm just wondering whether starting it with lower case 'n' means more respect or less respect?

[Nicolas Dorier]

The 100ms timing seems very consistent.

I was wondering if I was missing anything obvious, any "magic parameter" somewhere.

TCP_NODELAY?

Interestingly John Nagle has an account on this forum:

https://bitcointalk.org/index.php?action=profile;u=28488

Nice try, but did not changed anything.
I think the NoDelay is more for the sender. If I manage to compile bitcoind, I'll try to set the no delay on it, along with the sleep change.

[Nicolas Dorier]

Just to be sure, I'll also try to sync a bitcoind from my dev machine with the one of my server to see if I get the same speed, just to rule out any possibility that something in my code is wrong.

[Nicolas Dorier]

Just tried to connect with a bitcoind on my dev machine. There was some dash to 1MB/s, then it settled around 30kb/s. (I stopped at block 50K)
I guess this is normal since bitcoind checks incoming blocks. Would have loved to try at higher blocks though. (but my dev machine SSD is full )

[hhanh00]

No, generally speaking, everytimes I tried compiling bitcoind, I gave up after several hours wasted and frustrated, so I just imagine this is caused by the sleep. The 100ms timing seems very consistent.

Did you try looking at the github autobuilder log? I think that if Travis can do it, it should be easy to do manually too.
https://travis-ci.org/bitcoin/bitcoin/jobs/62665884

[Nicolas Dorier]

thanks, I'll take a look. I'm not linux wizard but it should help

[TierNolan]

Are you using your own software as receiver?  I think CPU rather than network bandwidth is normally the issue.

[Nicolas Dorier]

Are you using your own software as receiver?  I think CPU rather than network bandwidth is normally the issue.

Yes the client side is my own software. But my CPU is mostly idle, and the rcv buffer of my client does not have time to fill up, meaning that I am processing faster than the speed to which bitcoind is sending me blocks.
What bother me is the consistent delay for filling the rcv buffer once I emptied it by reading my socket... It takes 100ms. (+- 20ms)

[gmaxwell]

There is an intentional sleep in message handling.

It was removed in 0.11 (git master right now) via Patrick Strateman PR 5971.

There are excruciatingly precise building instructions, https://github.com/bitcoin/bitcoin/blob/master/doc/gitian-building.md  sufficient to get someone who isn't terribly technical producing bit identical binaries to the release.

(A simpler build process works too, of course, if you don't care about reproducing the builds-- but those instructions also have the benefit of being very precise.)

[Nicolas Dorier]

There is an intentional sleep in message handling.

It was removed in 0.11 (git master right now) via Patrick Strateman PR 5971.

There are excruciatingly precise building instructions, https://github.com/bitcoin/bitcoin/blob/master/doc/gitian-building.md  sufficient to get someone who isn't terribly technical producing bit identical binaries to the release.

(A simpler build process works too, of course, if you don't care about reproducing the builds-- but those instructions also have the benefit of being very precise.)

Cool I'll try that. If the problem really come from this sleep, it will be more than 30% improvement, this might be a 10x improvement for my case. (Since my code does not verify the blocks, only download them, most of the download time is wasted waiting 100ms)

[Nicolas Dorier]

I tried to follow the build process.

./bin/gbuild --commit bitcoin=master ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml

Stopping target if it is up
Making a new image copy
Starting target
Checking if target is up
Preparing build environment
Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Upgrading system, may take a while
Creating package manifest
Creating build script (var/build-script)
Running build script (log in var/build.log)
Grabbing results
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
./bin/gbuild:21:in `system!': failed to run copy-from-target  out build (RuntimeError)
        from ./bin/gbuild:267:in `block (2 levels) in <main>'
        from ./bin/gbuild:259:in `each'
        from ./bin/gbuild:259:in `block in <main>'
        from ./bin/gbuild:257:in `each'
        from ./bin/gbuild:257:in `<main>'

install.log and build.log are empty.

build-script is

NICO@bitcoin-build:~/gitian-builder$ tail var/build-script
  find . -name "lib*.la" -delete
  find . -name "lib*.a" -delete
  rm -rf ${DISTNAME}/lib/pkgconfig
  find ${DISTNAME} | sort | tar --no-recursion --mode='u+rw,go+r-w,a+X' --owner=0 --group=0 -c -T - | gzip -9n > ${OUTDIR}/${DISTNAME}-${i}.tar.gz
  cd ../../
done
mkdir -p $OUTDIR/src
mv $SOURCEDIST $OUTDIR/src
mv ${OUTDIR}/${DISTNAME}-x86_64-*.tar.gz ${OUTDIR}/${DISTNAME}-linux64.tar.gz
mv ${OUTDIR}/${DISTNAME}-i686-*.tar.gz ${OUTDIR}/${DISTNAME}-linux32.tar.gz

Well, except I forgot something obvious, I guess I will just wait for 0.11 to be out, I guess none of you have time to help me to build successfully, and I don't have time to figure out linux voodoo by myself.
I'll update this post when 0.11 is out to publish the result that 0.11 have on my perfs problems.

[TierNolan]

Cool I'll try that. If the problem really come from this sleep, it will be more than 30% improvement, this might be a 10x improvement for my case. (Since my code does not verify the blocks, only download them, most of the download time is wasted waiting 100ms)

I ran with the lastest on master on Linux, and it is 0-1ms per block for pure downloading.  I made sure to download 1000 blocks with each getdata request.

On Linux, 0.10.1.rc2 is quick as well, but only if I request multiple blocks at once.  Each getdata message costs 100ms, so sending one for every 1000 blocks is much faster.

Weirdly, when I run the java download code on windows with the latest download (0.10.1), it is 20-30ms per block.

"git checkout master"

getdata(1000 blocks): 0-1ms per block
getdata(1 block): 37-42ms per block

"git checkout v0.10.1rc2" [Linux]

getdata(1000 blocks): 1-3ms per block
getdata(1 block): 100-101ms per block

[Nicolas Dorier]

I am already fetching the blocks with getdata 10000 blocks.

The 100ms I have seen is not the same as "100ms per block".
I precisely mean that when the receive buffer of my socket is empty, then it takes consistently 100ms before new data comes inside.

I measure this just before reading the next message.

It makes no differences if those are 1MB blocks or 1kb blocks.

I feel the PR gmaxwell talks about fixed my problem. I'll let you know on 0.11.

[Nicolas Dorier]

I just tested the 0.11rc1.

The headers are downloaded 3 times faster. (Downloading 3MB/s instead of previous 1MB/s)
My block download speed have not changed very much.

I don't notice the 100ms wait time consistently anymore.

[lontivero]

Something is wrong somewhere because I've tested several times and I don't understand what is going on between NBitcoin and bitcoind. Here a share some of the results:

This is one of several run that I did with 0.10.1


And this is the test output: (those numbers are the kb/s)



And the following are the results for the same test against 0.11rc

(run 1)




As we can see, for a while it performed much better however, after a minute more or less it went down.

(run 2)




(run 3)




(run 4)




I tested this in exactly the same way. Bitcoind was connected only with NBitcoin.

Another wierd thing that I saw (i was not able to reproduce it) is that while downloading the blocks, a ping/pong exchange in the middle took 4 seconds.