Processor speed and blockchain synchronisation

[Jet Cash]

Now that my new Linux machine is working without any apparent problems, I thought I would do a Bitcoin wallet comparison. I'm using the free WiFi at McDonalds, and both machines are running at the moment. I started the Linux machine first thing this morning, and it was 16 hours behind. I started the Windows machine 2 hours later, and it was 19hours behind. The windows machine finished about 10 minutes ago, but the Linux machine is still 2 hours behind.
Transfer stats -
Windows received 139Mb and sent 3Mb
Linux received 170Mb and sent 8Mb (and has just finished )
Machine specs -
Windows - Core i5 notebook with 2Tb hard drive containing all Bitcoin files
Linux Ubuntu - Celeron with internal SSD for core software, and an external USB SSD for the blockchain files

WiFi runs at just under 30Kb for most of the time, but has short bursts up to 1,000Kb or drops to virtually zero. The transfer histograms for the two machines indicate the same 30Kb transfer for most of the time, but variations are unmatched. The WiFi on the core i5 machine seems to have a greater range.

What do you think is likely to be the reason for the difference in sync' times? My first thought was to blame the SSD, but if this was the case, wouldn't it result in lower activity in the WiFi. My next thought was that the Linux machine has an inferior Wifi processor, and this could be the reason for the larger file transfers if it includes many re-tries.

[Carlton Banks]

More likely the difference between the i5 CPU and the Celeron CPU. The cryptographic verification part of syncing is the main bottleneck since headers-first downloading was introduced in 0.10. How much RAM is in the Celeron machine?

[Jet Cash]

Windows
Intel i5-4210U @ 1.70GHz 2.40GHz 8Gb ram

Ubuntu
Celeron N3060 @ 1.60GHz x 2  1.8 GB

It looks as if you are right then Carlton

It's not really a problem, but it would be nice to speed it up if I can. What do you think about the extra WiFi traffic? I think I'll try a USB WiFi receiver to see if that helps. I don't think I can add extra ram to the Linux box.

[Carlton Banks]

It's not just the clock speed of the processor, there are qualitative differences between Celerons and i5's; your i5 probably has "hyper threading', otherwise known as SIMD, which I think allows one core to do more than one integer instruction per clock cycle etc (and Bitcoin uses mostly integer arithmetic AFAIA). Celeron's are cut down in other ways too, typically the SRAM caches are lower in size (allows chip processors to sell the chips with partially defective SRAM and/or other chips components)

As for the traffic differences... could it just be the relay behaviour of the peers you connected to with each machine?

[Jet Cash]

Thanks for the reply. The main reasons for getting the second machine were longer battery life, better portability, and a cheap machine to experiment with Linux. It seems to be good for all of those, so I guess there is a trade-off involved. I'll just have to live with the slower Bitcoin sync I guess. I wanted to use it to provide a wallet for Bitcoin/cash exchanges, and the sale and purchase of domain names for Bitcoin. I'll try to keep it updated daily, so it should be OK as a background machine.

I looked at the WiFi on the Linux machine, and it does look to be inferior. It only supports one channel, and one standard 802.11n. I may need to update the drivers as well. I think that's a job for Ron though ( I expect you know later-Ron).

[Carlton Banks]

Yes, it's interesting to hear how closely that spec performs compared to an i5 with similar cores and clock speed. I've wondered for a while how much better mobile devices need to perform before full nodes are viable (the price of data is of course also inhibiting, but maybe a bit of public Wi-Fi hopping can mitigate that a little, as you know). You're demonstrating that an 11" low-power low RAM SSD based laptop is very capable.

Maybe something to look into would be getting rid of any unneeded "sideshow" software that Ubuntu is pushing on you (and your available memory). There's probably at least something minor, Linux begins to resemble Windows in this way more as time goes on.

[Jet Cash]

The Ubuntu is straight out of the box. The only things I've added are Bitcoin support, and I think that is going to be included in the next major version. I think I also added some drivers for the external SSD. As you suggested, I need to check for updates and redundant software.

I suspect that it is going to need to be online for four hours or so every day, but I'll review that statement tomorrow. Of course, that doesn't mean you have to sit there watching it for four hours, you can leave it in the car whilst it sync's the blockchain. Core seems to be pretty robust, so that four hours doesn't have to be one burst, you can split it into 4 one hour sessions of course.

The only proviso is that, if you are going to check for a bitcoin payment for a cash exchange, you'll need to have your wallet up to date of course.

[Jet Cash]

Well this is very pleasant - I'm sitting outside in the patio area in the sunshine.

I'll give you guys one tip though. If you are going to run a rig like this, don't forget to connect the SSD drive. Core doesn't get into a sweat about it, but it does suggest that it downloads the blockchain onto your internal SSD, and that may not be quite what you want.

[AlexGR]

It's not just the clock speed of the processor, there are qualitative differences between Celerons and i5's; your i5 probably has "hyper threading', otherwise known as SIMD

SMT (multi threading on same core by utilizing the different pipelines of the same CPU when they are sitting idle)

(SIMD are SSE/AVX instructions where you batch process multiple data with one instruction...)

I'll give you guys one tip though. If you are going to run a rig like this, don't forget to connect the SSD drive. Core doesn't get into a sweat about it, but it does suggest that it downloads the blockchain onto your internal SSD, and that may not be quite what you want.

Another tip is perhaps disabling full journaling of the filesystem for initial sync as that does tend to slow it down, whether SSD or mechanic. Plus on SSD it would double all the writes (reducing the disk's lifetime).

[buwaytress]

Nice information to have and also a slight surprise to me that a lowish-end laptop isn't far behind a more contemporary model, probably agree that the bottleneck really lies with the processor rather than the wireless hardware - but to be sure why not run simple wifi diagnostics on both?

On my older ASUS netbook (eeepc) for example, its wifi range is far greater (stronger signal from distance) than my newer ASUS zenfone, but for some strange reason the mobile is able to download quicker (the trade off is frequent loss of signal). As a result, I can never sync properly on phone.

[IadixDev]

Is there any serious profiling made with coin software to identity performance bottle neck ?

It's probably à thing im going to do in a near future, but it doesn't seem very easy to me cause it involve both i/o on local drive and network, and cpu processing, plus sending nodes dont have necessarily the same through put, and the Connectivity between two nodes is not necessarily the same either, seem hard to do real profiling with this kind of software, but im probably going to get into this soon, and seeing how simd / avx and multi threading can increase performance, but blockchain are not necessarily good target for multi thread cause each block has to be checked after the previous one, so it's more serial than parallele, but im trying to find where threading could really improve performance. Maybe on intra block level as transactions in the same block are not supposed to have dépendance on transactions in the same block, that could be good candidate for threading.

Maybe even different compiler used with different options can make performances difference too.

But as the purenode core code is very simple, it will be much simpler to do good profiling of different parts and see where performances can be improved.

For the moment it seem hard to me to identity clear bottle neck right off the bat like this, there is too much things that can affects node synchronisation speed.

I didn't do much testing with this, just tested with bitmonero cause they have different options to import/export blockchains in different format, but when I tested it didn't even seem to make that big difference to import from local file or download from network, I have optical fiber too, so I guess network speed is not the main issue, but I didn't do much heavy testing yet.

Imo if you want to have meaning result out of this kind of test, you need to do it at least 2/3 times and average the time différence to already eliminate issue coming from the network Connectivity itself and see if you get the same order of result doing it several times, because it's very possible the difference doesn't even come from the computer itself. If one is synchronising from slow node, or have to reject bad blocks or eliminate issue from network Connectivity.

Even WiFi sometime you move the devices 5cm you get x3 signal :p

Would need to make this sort of test on the two node synchronising from the same node who have stable through put and stable connection, like local wire network synchronising from same node, one after each other there you can have more meaning full result on the computer performances itself.

All together it doesn't seem trivial to do good benchmarking / profiling on synchronisation speed.

[Cubic Earth]

Is there any serious profiling made with coin software to identity performance bottle neck ?

Yes. Many thousands of hours....

Bandwidth can be bottleneck in terms of block propagation speed.
Processing power is a bottleneck too, especially for initial chain syncing.

[Jet Cash]

Another tip is perhaps disabling full journaling of the filesystem for initial sync as that does tend to slow it down, whether SSD or mechanic. Plus on SSD it would double all the writes (reducing the disk's lifetime).

That sounds like a good idea, but I can't find any info about it. By initial sync, do you mean the initial downloading of the blockchain? With this computer, I copied the blockchain directory from the i5 computer, and started the new node with the address of the copied file. That seems to have worked without any problems.

[IadixDev]

Is there any serious profiling made with coin software to identity performance bottle neck ?

Yes. Many thousands of hours....

Bandwidth can be bottleneck in terms of block propagation speed.
Processing power is a bottleneck too, especially for initial chain syncing.

Is there any attempt in compressing block data sent over network ? Would that improve block propagation time ?

Is there a place I can find such kind of test and benchmarking ?

Is there any test using asynchronous io too to see if that can make a difference in synch speed or if it's mostly processing power on signature checking that take most of the time ?

[Jet Cash]

Is there any serious profiling made with coin software to identity performance bottle neck ?

Yes. Many thousands of hours....

Bandwidth can be bottleneck in terms of block propagation speed.
Processing power is a bottleneck too, especially for initial chain syncing.

I can't do much about the processor or the ram unfortunately.
The wallet is on the internal SSD, and the blockchain is on the external SSD - it's on a USB3 port, and that is about the fastest connection I can get. Unfortunately this model doesn't have an RJ45 port ( is that the correct name these days? ).
If I get a chance, I'll do a bit of resource analysis to see if there is an apparent bottleneck.
I need to look at the WiFi, and make sure I've got the best drivers. I suspect the aerial on the i5 is twice the width of the Celery, and maybe this cuts down on resends. I'll see if I can find some software to check this.

[Cubic Earth]

Is there any serious profiling made with coin software to identity performance bottle neck ?

Yes. Many thousands of hours....

Bandwidth can be bottleneck in terms of block propagation speed.
Processing power is a bottleneck too, especially for initial chain syncing.

Is there any attempt in compressing block data sent over network ? Would that improve block propagation time ?

Is there a place I can find such kind of test and benchmarking ?

The transaction data in blocks is already highly optimized and compression would yield very minimal gains. However, there are numerous approaches to increasing block propagation... check out compact blocks and x-thin blocks.

[ImHash]

I'm planning to get a lenovo with Celeron N3060 1.6GHz upto 2.48GHz 2MB cache  with 4GB RAM and GeForce 920M 1GB
I will install win 7 what manual configuration do I need to do for best blockchain sync?

[IadixDev]

Is there any serious profiling made with coin software to identity performance bottle neck ?

Yes. Many thousands of hours....

Bandwidth can be bottleneck in terms of block propagation speed.
Processing power is a bottleneck too, especially for initial chain syncing.

Is there any attempt in compressing block data sent over network ? Would that improve block propagation time ?

Is there a place I can find such kind of test and benchmarking ?

The transaction data in blocks is already highly optimized and compression would yield very minimal gains. However, there are numerous approaches to increasing block propagation... check out compact blocks and x-thin blocks.

I would believe there is still potentially repetitive data in blocks that could gain from compression, but not sure how much. But there are still adresses hashes in the script and certain thing that could gain a bit from compression, not sure the gain in transfert speed would over come the loss in compression/decompression time though.

Oki i'll check those

Found this :

"Datastream compression: Testing various compression libraries such as LZO-1x and Zlib have shown it is possible to further reduce block and transaction sizes by 20 to 30% without affecting response times which could also be applied to thinblocks"

Not so marginal

[Jet Cash]

I'm planning to get a lenovo with Celeron N3060 1.6GHz upto 2.48GHz 2MB cache  with 4GB RAM and GeForce 920M 1GB
I will install win 7 what manual configuration do I need to do for best blockchain sync?

Is it this one?
http://www.currys.co.uk/gbuk/computing/laptops/laptops/lenovo-ideapad-110-15-6-laptop-black-10146807-pdt.html

If it is, I'd try to get a copy of the blockchain directory on a dvd, or use the ethernet to connect to a machine that is already running a full node.

It takes forever to download the full blockchain.

[ImHash]

I'm planning to get a lenovo with Celeron N3060 1.6GHz upto 2.48GHz 2MB cache  with 4GB RAM and GeForce 920M 1GB
I will install win 7 what manual configuration do I need to do for best blockchain sync?

Is it this one?
http://www.currys.co.uk/gbuk/computing/laptops/laptops/lenovo-ideapad-110-15-6-laptop-black-10146807-pdt.html

If it is, I'd try to get a copy of the blockchain directory on a dvd, or use the ethernet to connect to a machine that is already running a full node.

It takes forever to download the full blockchain.

No it's ideapad 300-V and from where can I get the full blockchain? I need at least a zip file with resume ability to download with internet download manager.
I have internet speed up to 10MB/s