Bitcoin Forum
January 17, 2019, 01:50:31 AM *
News: Latest Bitcoin Core release: 0.17.1 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1] 2 »  All
  Print  
Author Topic: how to maximize block download speed on a single local node  (Read 2241 times)
Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 700
Merit: 501


View Profile
May 13, 2015, 12:42:54 PM
 #1

I noticed that fetching blocks over the bitcoin protocol is slow in my setting.
My node is in my local network which have over 100MB speed with my dev computer.

I find no way of making it better, and I'd like your advice, here is my finding:

  • Average Speed Achieved : 5MB/s
  • Max speed Achieved (ponctual) : 15MB/s
  • Fetch strategy : getdata of 50 000 blocks, Tip to Genesis
  • Client's socket receive buffer is never full (indicate bitcoind does not send fast enough)
  • Disk IO and CPU not capped
  • When the rcv buffer of the client is empty, it takes 100ms with low variance before new data come
  • Using 2 connections instead of one for fetching blocks speed things up a little bit (7.5MB/s average)

I tried to play with receive and send buffer on both the bitcoind and the client code, no effect.

I noticed that the 100ms time it take for my buffer to be filled again must come from ThreadMessageHandler.

However by reviewing the code, I have seen no mistake : if the send buffer of the bitcoin node is not full, I should not experience any 100ms delay.
Once again, trying to change the buffers size of bitcoind had no effect.

Any idea ?

My alternative is to parse the blk directory over SMB, which go near 75MB/S.
But this is a fragile solution I'd like to avoid.

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
1547689831
Hero Member
*
Offline Offline

Posts: 1547689831

View Profile Personal Message (Offline)

Ignore
1547689831
Reply with quote  #2

1547689831
Report to moderator
1547689831
Hero Member
*
Offline Offline

Posts: 1547689831

View Profile Personal Message (Offline)

Ignore
1547689831
Reply with quote  #2

1547689831
Report to moderator
1547689831
Hero Member
*
Offline Offline

Posts: 1547689831

View Profile Personal Message (Offline)

Ignore
1547689831
Reply with quote  #2

1547689831
Report to moderator
PLAY NOW
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1547689831
Hero Member
*
Offline Offline

Posts: 1547689831

View Profile Personal Message (Offline)

Ignore
1547689831
Reply with quote  #2

1547689831
Report to moderator
cr1776
Legendary
*
Offline Offline

Activity: 2128
Merit: 1014


View Profile
May 13, 2015, 01:57:17 PM
 #2

Hi,
A few questions:
1. Is this with Bitcoin Core 0.10.1?
2. Are you only syncing the block chain over your local network to one other machine?  If you can sync from multiple machines, that can speed things up.
3. HDD or SSD on the machines?
4. CPU specs?

:-)
TierNolan
Legendary
*
Offline Offline

Activity: 1218
Merit: 1002


View Profile
May 13, 2015, 03:24:36 PM
 #3

However by reviewing the code, I have seen no mistake : if the send buffer of the bitcoin node is not full, I should not experience any 100ms delay.
Once again, trying to change the buffers size of bitcoind had no effect.

It looks like it will only skip the sleep if at least one node received data and it is unprocessed.  I don't why that is connected to the sending buffer.

Have you tried dropping the 100ms sleep to 1ms?

1LxbG5cKXzTwZg9mjL3gaRE835uNQEteWF
NeuroticFish
Legendary
*
Offline Offline

Activity: 1736
Merit: 1130


There are no mistakes. Only opportunities wasted.


View Profile
May 13, 2015, 03:44:47 PM
 #4

There are a lot of conditions there. I'd also try to set for a start the sleep to a very low value and if there is a real improvement I'd debug a little.
It's too easy to miss something there.

.BITSLER.                 ▄███
               ▄████▀
             ▄████▀
           ▄████▀  ▄██▄
         ▄████▀    ▀████▄
       ▄████▀        ▀████▄
     ▄████▀            ▀████▄
   ▄████▀                ▀████▄
 ▄████▀ ▄████▄      ▄████▄ ▀████▄
█████   ██████      ██████   █████
 ▀████▄ ▀████▀      ▀████▀ ▄████▀
   ▀████▄                ▄████▀
     ▀████▄            ▄████▀
       ▀████▄        ▄████▀
         ▀████▄    ▄████▀
           ▀████▄▄████▀
             ▀██████▀
               ▀▀▀▀
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▄            
▄▄▄▄▀▀▀▀    ▄▄█▄▄ ▀▀▄         
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▄      
█  ▀▄▄  ▀█▀▀ ▄      ▀████   ▀▀▄   
█ █▄  ▀▄   ▀████       ▀▀ ▄██▄ ▀▀▄
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
█  ▀▀       ▀▄▄ ▀████      ▄▄▄▀▀▀  █
█            ▄ ▀▄    ▄▄▄▀▀▀   ▄▄  █
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
█ ▄▄   ███   ▀██  █           ▀▀  █ 
█ ███  ▀██       █        ▄▄      █ 
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀  
▀▄            █        ▀▀      █  
▀▀▄   ███▄  █   ▄▄          █   
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀    
▀▀▄   █   ▀▀▄▄▄▀▀▀         
▄▄▄▄▄▄▄▄▄▄▄█▄▄▀▀▀▀              
              ▄▄▄██████▄▄▄
          ▄▄████████████████▄▄
        ▄██████▀▀▀▀▀▀▀▀▀▀██████▄
▄     ▄█████▀             ▀█████▄
██▄▄ █████▀                ▀█████
 ████████            ▄██      █████
  ████████▄         ███▀       ████▄
  █████████▀▀     ▄███▀        █████
   █▀▀▀          █████         █████
     ▄▄▄         ████          █████
   █████          ▀▀           ████▀
    █████                     █████
     █████▄                 ▄█████
      ▀█████▄             ▄█████▀
        ▀██████▄▄▄▄▄▄▄▄▄▄██████▀
          ▀▀████████████████▀▀
              ▀▀▀██████▀▀▀
            ▄▄▄███████▄▄▄
         ▄█▀▀▀ ▄▄▄▄▄▄▄ ▀▀▀█▄
       █▀▀ ▄█████████████▄ ▀▀█
     █▀▀ ███████████████████ ▀▀█
    █▀ ███████████████████████ ▀█
   █▀ ███████████████▀▀ ███████ ▀█
 ▄█▀ ██████████████▀      ▀█████ ▀█▄
███ ███████████▀▀            ▀▀██ ███
███ ███████▀▀                     ███
███ ▀▀▀▀                          ███
▀██▄                             ▄██▀
  ▀█▄                            ▀▀
    █▄       █▄▄▄▄▄▄▄▄▄█
     █▄      ▀█████████▀
      ▀█▄      ▀▀▀▀▀▀▀
        ▀▀█▄▄  ▄▄▄
            ▀▀█████
[]
Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 700
Merit: 501


View Profile
May 13, 2015, 03:55:09 PM
 #5

Quote
Is this with Bitcoin Core 0.10.1?
Yes

Quote
Are you only syncing the block chain over your local network to one other machine?  If you can sync from multiple machines, that can speed things up.
Yes, but this is not what I want to do : I am currently benchmarking my code on NBitcoin, and noticed that the download was slow and that my receive buffer was never full, which indicate bitcoind is slow.


Quote
HDD or SSD on the machines?
4 HDD in RAID5, 10K RPM (the server has his own RAID controller)

Quote
CPU spec

I own a Dell Powerdge T620, the CPU is not capped, nor the Disk IO.

Quote
Have you tried dropping the 100ms sleep to 1ms?
No, generally speaking, everytimes I tried compiling bitcoind, I gave up after several hours wasted and frustrated, so I just imagine this is caused by the sleep. The 100ms timing seems very consistent.

Quote
There are a lot of conditions there. I'd also try to set for a start the sleep to a very low value and if there is a real improvement I'd debug a little.
Thanks, well, I guess I will have to try compiling bitcoin one more time on windows.

I was wondering if I was missing anything obvious, any "magic parameter" somewhere.

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
2112
Legendary
*
Offline Offline

Activity: 2114
Merit: 1027



View Profile
May 13, 2015, 04:53:11 PM
Last edit: May 13, 2015, 05:18:50 PM by 2112
 #6

The 100ms timing seems very consistent.

I was wondering if I was missing anything obvious, any "magic parameter" somewhere.
TCP_NODELAY?

Interestingly John Nagle has an account on this forum:

https://bitcointalk.org/index.php?action=profile;u=28488

Edit: I'm actually posting this from Windows, not my normal machine. It took me a while to locate the place where the OS include files are located. In a search for the correct spelling I found a funny comment in one of the Windows' SDKs:

Code:
// turn off nagling

Obviously making a verb out of somebody's last name is a sign of respect. I'm just wondering whether starting it with lower case 'n' means more respect or less respect?

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 700
Merit: 501


View Profile
May 13, 2015, 05:15:45 PM
 #7

The 100ms timing seems very consistent.

I was wondering if I was missing anything obvious, any "magic parameter" somewhere.
TCP_NODELAY?

Interestingly John Nagle has an account on this forum:

https://bitcointalk.org/index.php?action=profile;u=28488



Nice try, but did not changed anything.
I think the NoDelay is more for the sender. If I manage to compile bitcoind, I'll try to set the no delay on it, along with the sleep change.

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 700
Merit: 501


View Profile
May 13, 2015, 05:19:13 PM
 #8

Just to be sure, I'll also try to sync a bitcoind from my dev machine with the one of my server to see if I get the same speed, just to rule out any possibility that something in my code is wrong.

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 700
Merit: 501


View Profile
May 13, 2015, 05:35:15 PM
 #9

Just tried to connect with a bitcoind on my dev machine. There was some dash to 1MB/s, then it settled around 30kb/s. (I stopped at block 50K)
I guess this is normal since bitcoind checks incoming blocks. Would have loved to try at higher blocks though. (but my dev machine SSD is full Sad)

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
hhanh00
Sr. Member
****
Offline Offline

Activity: 467
Merit: 261


View Profile
May 15, 2015, 09:08:27 AM
 #10

No, generally speaking, everytimes I tried compiling bitcoind, I gave up after several hours wasted and frustrated, so I just imagine this is caused by the sleep. The 100ms timing seems very consistent.

Did you try looking at the github autobuilder log? I think that if Travis can do it, it should be easy to do manually too.
https://travis-ci.org/bitcoin/bitcoin/jobs/62665884


Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 700
Merit: 501


View Profile
May 15, 2015, 09:46:01 AM
 #11

thanks, I'll take a look. I'm not linux wizard but it should help

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
TierNolan
Legendary
*
Offline Offline

Activity: 1218
Merit: 1002


View Profile
May 15, 2015, 07:32:29 PM
 #12

Are you using your own software as receiver?  I think CPU rather than network bandwidth is normally the issue.

1LxbG5cKXzTwZg9mjL3gaRE835uNQEteWF
Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 700
Merit: 501


View Profile
May 16, 2015, 12:14:31 AM
 #13

Are you using your own software as receiver?  I think CPU rather than network bandwidth is normally the issue.
Yes the client side is my own software. But my CPU is mostly idle, and the rcv buffer of my client does not have time to fill up, meaning that I am processing faster than the speed to which bitcoind is sending me blocks.
What bother me is the consistent delay for filling the rcv buffer once I emptied it by reading my socket... It takes 100ms. (+- 20ms)

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
gmaxwell
Moderator
Legendary
*
qt
Offline Offline

Activity: 2632
Merit: 1846



View Profile
May 16, 2015, 10:05:50 AM
 #14

There is an intentional sleep in message handling.

It was removed in 0.11 (git master right now) via Patrick Strateman PR 5971.

There are excruciatingly precise building instructions, https://github.com/bitcoin/bitcoin/blob/master/doc/gitian-building.md  sufficient to get someone who isn't terribly technical producing bit identical binaries to the release.

(A simpler build process works too, of course, if you don't care about reproducing the builds-- but those instructions also have the benefit of being very precise.)
Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 700
Merit: 501


View Profile
May 16, 2015, 12:48:44 PM
 #15

There is an intentional sleep in message handling.

It was removed in 0.11 (git master right now) via Patrick Strateman PR 5971.

There are excruciatingly precise building instructions, https://github.com/bitcoin/bitcoin/blob/master/doc/gitian-building.md  sufficient to get someone who isn't terribly technical producing bit identical binaries to the release.

(A simpler build process works too, of course, if you don't care about reproducing the builds-- but those instructions also have the benefit of being very precise.)

Cool I'll try that. If the problem really come from this sleep, it will be more than 30% improvement, this might be a 10x improvement for my case. (Since my code does not verify the blocks, only download them, most of the download time is wasted waiting 100ms)

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 700
Merit: 501


View Profile
May 16, 2015, 02:55:09 PM
 #16

I tried to follow the build process.

Quote
./bin/gbuild --commit bitcoin=master ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
Quote
Stopping target if it is up
Making a new image copy
Starting target
Checking if target is up
Preparing build environment
Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Upgrading system, may take a while
Creating package manifest
Creating build script (var/build-script)
Running build script (log in var/build.log)
Grabbing results
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
./bin/gbuild:21:in `system!': failed to run copy-from-target  out build (RuntimeError)
        from ./bin/gbuild:267:in `block (2 levels) in <main>'
        from ./bin/gbuild:259:in `each'
        from ./bin/gbuild:259:in `block in <main>'
        from ./bin/gbuild:257:in `each'
        from ./bin/gbuild:257:in `<main>'

install.log and build.log are empty.

build-script is

Quote
NICO@bitcoin-build:~/gitian-builder$ tail var/build-script
  find . -name "lib*.la" -delete
  find . -name "lib*.a" -delete
  rm -rf ${DISTNAME}/lib/pkgconfig
  find ${DISTNAME} | sort | tar --no-recursion --mode='u+rw,go+r-w,a+X' --owner=0 --group=0 -c -T - | gzip -9n > ${OUTDIR}/${DISTNAME}-${i}.tar.gz
  cd ../../
done
mkdir -p $OUTDIR/src
mv $SOURCEDIST $OUTDIR/src
mv ${OUTDIR}/${DISTNAME}-x86_64-*.tar.gz ${OUTDIR}/${DISTNAME}-linux64.tar.gz
mv ${OUTDIR}/${DISTNAME}-i686-*.tar.gz ${OUTDIR}/${DISTNAME}-linux32.tar.gz

Well, except I forgot something obvious, I guess I will just wait for 0.11 to be out, I guess none of you have time to help me to build successfully, and I don't have time to figure out linux voodoo by myself.
I'll update this post when 0.11 is out to publish the result that 0.11 have on my perfs problems.

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
TierNolan
Legendary
*
Offline Offline

Activity: 1218
Merit: 1002


View Profile
May 16, 2015, 03:51:25 PM
 #17

Cool I'll try that. If the problem really come from this sleep, it will be more than 30% improvement, this might be a 10x improvement for my case. (Since my code does not verify the blocks, only download them, most of the download time is wasted waiting 100ms)

I ran with the lastest on master on Linux, and it is 0-1ms per block for pure downloading.  I made sure to download 1000 blocks with each getdata request.

On Linux, 0.10.1.rc2 is quick as well, but only if I request multiple blocks at once.  Each getdata message costs 100ms, so sending one for every 1000 blocks is much faster.

Weirdly, when I run the java download code on windows with the latest download (0.10.1), it is 20-30ms per block.

"git checkout master"

getdata(1000 blocks): 0-1ms per block
getdata(1 block): 37-42ms per block

"git checkout v0.10.1rc2" [Linux]

getdata(1000 blocks): 1-3ms per block
getdata(1 block): 100-101ms per block

1LxbG5cKXzTwZg9mjL3gaRE835uNQEteWF
Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 700
Merit: 501


View Profile
May 16, 2015, 04:58:20 PM
 #18

I am already fetching the blocks with getdata 10000 blocks.

The 100ms I have seen is not the same as "100ms per block".
I precisely mean that when the receive buffer of my socket is empty, then it takes consistently 100ms before new data comes inside.

I measure this just before reading the next message.

It makes no differences if those are 1MB blocks or 1kb blocks.

I feel the PR gmaxwell talks about fixed my problem. I'll let you know on 0.11.

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 700
Merit: 501


View Profile
June 15, 2015, 12:43:40 AM
 #19

I just tested the 0.11rc1.

The headers are downloaded 3 times faster. (Downloading 3MB/s instead of previous 1MB/s)
My block download speed have not changed very much.

I don't notice the 100ms wait time consistently anymore.

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
lontivero
Full Member
***
Offline Offline

Activity: 160
Merit: 103

Amazing times are coming


View Profile
June 15, 2015, 03:42:11 AM
 #20

Something is wrong somewhere because I've tested several times and I don't understand what is going on between NBitcoin and bitcoind. Here a share some of the results:

This is one of several run that I did with 0.10.1


And this is the test output: (those numbers are the kb/s)



And the following are the results for the same test against 0.11rc

(run 1)




As we can see, for a while it performed much better however, after a minute more or less it went down.

(run 2)




(run 3)




(run 4)




I tested this in exactly the same way. Bitcoind was connected only with NBitcoin.

Another wierd thing that I saw (i was not able to reproduce it) is that while downloading the blocks, a ping/pong exchange in the middle took 4 seconds.
 
Pages: [1] 2 »  All
  Print  
 
Jump to:  

Bitcointalk.org is not available or authorized for sale. Do not believe any fake listings.
Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!