Bitcoin Forum

I noticed that fetching blocks over the bitcoin protocol is slow in my setting.
My node is in my local network which have over 100MB speed with my dev computer.

I find no way of making it better, and I'd like your advice, here is my finding:

• Average Speed Achieved : 5MB/s
• Max speed Achieved (ponctual) : 15MB/s
• Fetch strategy : getdata of 50 000 blocks, Tip to Genesis
• Client's socket receive buffer is never full (indicate bitcoind does not send fast enough)
• Disk IO and CPU not capped
• When the rcv buffer of the client is empty, it takes 100ms with low variance before new data come
• Using 2 connections instead of one for fetching blocks speed things up a little bit (7.5MB/s average)

I tried to play with receive and send buffer on both the bitcoind and the client code, no effect.

I noticed that the 100ms time it take for my buffer to be filled again must come from ThreadMessageHandler (https://github.com/bitcoin/bitcoin/blob/master/src/net.cpp#L1447).

However by reviewing the code, I have seen no mistake : if the send buffer of the bitcoin node is not full, I should not experience any 100ms delay.
Once again, trying to change the buffers size of bitcoind had no effect.

Any idea ?

My alternative is to parse the blk directory over SMB, which go near 75MB/S.
But this is a fragile solution I'd like to avoid.

Hi,
A few questions:
1. Is this with Bitcoin Core 0.10.1?
2. Are you only syncing the block chain over your local network to one other machine?  If you can sync from multiple machines, that can speed things up.
3. HDD or SSD on the machines?
4. CPU specs?

:-)

It looks like it will only skip the sleep if at least one node received data and it is unprocessed.  I don't why that is connected to the sending buffer.

Have you tried dropping the 100ms sleep to 1ms?

There are a lot of conditions there. I'd also try to set for a start the sleep to a very low value and if there is a real improvement I'd debug a little.
It's too easy to miss something there.

Yes

Yes, but this is not what I want to do : I am currently benchmarking my code on NBitcoin, and noticed that the download was slow and that my receive buffer was never full, which indicate bitcoind is slow.


4 HDD in RAID5, 10K RPM (the server has his own RAID controller)


I own a Dell Powerdge T620 (http://www.zdnet.com/product/dell-poweredge-t620/), the CPU is not capped, nor the Disk IO.

No, generally speaking, everytimes I tried compiling bitcoind, I gave up after several hours wasted and frustrated, so I just imagine this is caused by the sleep. The 100ms timing seems very consistent.

Thanks, well, I guess I will have to try compiling bitcoin one more time on windows.

I was wondering if I was missing anything obvious, any "magic parameter" somewhere.

TCP_NODELAY?

Interestingly John Nagle has an account on this forum:

https://bitcointalk.org/index.php?action=profile;u=28488

Edit: I'm actually posting this from Windows, not my normal machine. It took me a while to locate the place where the OS include files are located. In a search for the correct spelling I found a funny comment in one of the Windows' SDKs:


Obviously making a verb out of somebody's last name is a sign of respect. I'm just wondering whether starting it with lower case 'n' means more respect or less respect?

Nice try, but did not changed anything.
I think the NoDelay is more for the sender. If I manage to compile bitcoind, I'll try to set the no delay on it, along with the sleep change.

Just to be sure, I'll also try to sync a bitcoind from my dev machine with the one of my server to see if I get the same speed, just to rule out any possibility that something in my code is wrong.

Just tried to connect with a bitcoind on my dev machine. There was some dash to 1MB/s, then it settled around 30kb/s. (I stopped at block 50K)
I guess this is normal since bitcoind checks incoming blocks. Would have loved to try at higher blocks though. (but my dev machine SSD is full :()

Did you try looking at the github autobuilder log? I think that if Travis can do it, it should be easy to do manually too.
https://travis-ci.org/bitcoin/bitcoin/jobs/62665884

thanks, I'll take a look. I'm not linux wizard but it should help

Are you using your own software as receiver?  I think CPU rather than network bandwidth is normally the issue.

Yes the client side is my own software. But my CPU is mostly idle, and the rcv buffer of my client does not have time to fill up, meaning that I am processing faster than the speed to which bitcoind is sending me blocks.
What bother me is the consistent delay for filling the rcv buffer once I emptied it by reading my socket... It takes 100ms. (+- 20ms)

There is an intentional sleep in message handling.

It was removed in 0.11 (git master right now) via Patrick Strateman PR 5971.

There are excruciatingly precise building instructions, https://github.com/bitcoin/bitcoin/blob/master/doc/gitian-building.md (https://github.com/bitcoin/bitcoin/blob/master/doc/gitian-building.md)  sufficient to get someone who isn't terribly technical producing bit identical binaries to the release.

(A simpler build process works too, of course, if you don't care about reproducing the builds-- but those instructions also have the benefit of being very precise.)

Cool I'll try that. If the problem really come from this sleep, it will be more than 30% improvement, this might be a 10x improvement for my case. (Since my code does not verify the blocks, only download them, most of the download time is wasted waiting 100ms)

I tried to follow the build process.


install.log and build.log are empty.

build-script is


Well, except I forgot something obvious, I guess I will just wait for 0.11 to be out, I guess none of you have time to help me to build successfully, and I don't have time to figure out linux voodoo by myself.
I'll update this post when 0.11 is out to publish the result that 0.11 have on my perfs problems.

I ran with the lastest on master on Linux, and it is 0-1ms per block for pure downloading.  I made sure to download 1000 blocks with each getdata request.

On Linux, 0.10.1.rc2 is quick as well, but only if I request multiple blocks at once.  Each getdata message costs 100ms, so sending one for every 1000 blocks is much faster.

Weirdly, when I run the java download code on windows with the latest download (0.10.1), it is 20-30ms per block.

"git checkout master"

getdata(1000 blocks): 0-1ms per block
getdata(1 block): 37-42ms per block

"git checkout v0.10.1rc2" [Linux]

getdata(1000 blocks): 1-3ms per block
getdata(1 block): 100-101ms per block

I am already fetching the blocks with getdata 10000 blocks.

The 100ms I have seen is not the same as "100ms per block".
I precisely mean that when the receive buffer of my socket is empty, then it takes consistently 100ms before new data comes inside.

I measure this just before reading the next message.

It makes no differences if those are 1MB blocks or 1kb blocks.

I feel the PR gmaxwell talks about fixed my problem. I'll let you know on 0.11.

I just tested the 0.11rc1.

The headers are downloaded 3 times faster. (Downloading 3MB/s instead of previous 1MB/s)
My block download speed have not changed very much.

I don't notice the 100ms wait time consistently anymore.

Something is wrong somewhere because I've tested several times and I don't understand what is going on between NBitcoin and bitcoind. Here a share some of the results:

This is one of several run that I did with 0.10.1
https://i.imgur.com/AZQNqGs.png

And this is the test output: (those numbers are the kb/s)
https://i.imgur.com/efs0Pc4.png


And the following are the results for the same test against 0.11rc

(run 1)

https://i.imgur.com/FLPT0EU.png
https://i.imgur.com/M8bqKYa.png

As we can see, for a while it performed much better however, after a minute more or less it went down.

(run 2)

https://i.imgur.com/P3YFlTS.png
https://i.imgur.com/RpE576C.png

(run 3)

https://i.imgur.com/3TYIqZO.png
https://i.imgur.com/Dp5aUXF.png

(run 4)

https://i.imgur.com/lRRZtrd.png
https://i.imgur.com/XeCzp5b.png

I tested this in exactly the same way. Bitcoind was connected only with NBitcoin.

Another wierd thing that I saw (i was not able to reproduce it) is that while downloading the blocks, a ping/pong exchange in the middle took 4 seconds.
 

lonteviro, the numbers are in bytes / sec.
I witnessed some peaks at 15 MB/sec like you, but this was more or less random and did not seem to depends on the version of the node.

This is normal, the messages sent from the node to the client are queued in bitcoind.

If client ask 10 000 blocks then does 1 ping, then the pong of the server will be queued at the end of the 10 000 blocks.
This is also why the pong timeout is 20min in bitcoind.

Did you tested on a local node ? your results are very coherent with mine.

I notice no bottleneck of disk IO or CPU. The socket buffer being never full means that NBitcoin is processing blocks faster than they are coming from bitcoind.

Did someone managed (on any library) to fetch block from a local node at a highest speed than 10MB/s ?

Thanks for the explanation about ping/pong messages.

Yes, I was testing with bitcoind and NBitcoin in the same local machine. And yes, CPU and Disk IO are very low usage (CPU is almost idle).