Quickest method to initialize new bitcoind node

[Soros Shorts]

Can someone tell me what is the best practice for initializing a new bitcoind node quickly? Do I really need to wait 2 days for the blockchain to be downloaded and validated before the node is operational? As far as I can tell most of the time is spent not on network resources but on CPU resources, at least for the hardware I am running (Azure & Amazon EC).

If I already have another fully initalized node that I trust, is there anyway I can just copy the blockchain database from a trusted node over to the new node (bypassing all the time consuming validation)?

[1714045764]

1714045764

[1714045764]

1714045764

[1714045764]

1714045764

[achow101]

Can someone tell me what is the best practice for initializing a new bitcoind node quickly? Do I really need to wait 2 days for the blockchain to be downloaded and validated before the node is operational? As far as I can tell most of the time is spent not on network resources but on CPU resources, at least for the hardware I am running (Azure & Amazon EC).

If I already have another fully initalized node that I trust, is there anyway I can just copy the blockchain database from a trusted node over to the new node (bypassing all the time consuming validation)?

If you have access (can login and copy stuff to and from) to a fully synced node, you could copy over most of the stuff in the data directory and the node would just pick from where that data was left off.

[Soros Shorts]

Can someone tell me what is the best practice for initializing a new bitcoind node quickly? Do I really need to wait 2 days for the blockchain to be downloaded and validated before the node is operational? As far as I can tell most of the time is spent not on network resources but on CPU resources, at least for the hardware I am running (Azure & Amazon EC).

If I already have another fully initalized node that I trust, is there anyway I can just copy the blockchain database from a trusted node over to the new node (bypassing all the time consuming validation)?

If you have access (can login and copy stuff to and from) to a fully synced node, you could copy over most of the stuff in the data directory and the node would just pick from where that data was left off.

Trying this out now. I had to stop the source node for 20 minutes to make a local copy. Now I am copying 60GB worth of data to the new server. Unfortunately this is a single threaded SSH/SCP operation and the servers are 6000 miles apart (high latency), so it remains to be seen if I save any time at all.

[ranochigo]

Can someone tell me what is the best practice for initializing a new bitcoind node quickly? Do I really need to wait 2 days for the blockchain to be downloaded and validated before the node is operational? As far as I can tell most of the time is spent not on network resources but on CPU resources, at least for the hardware I am running (Azure & Amazon EC).

If I already have another fully initalized node that I trust, is there anyway I can just copy the blockchain database from a trusted node over to the new node (bypassing all the time consuming validation)?

If you have access (can login and copy stuff to and from) to a fully synced node, you could copy over most of the stuff in the data directory and the node would just pick from where that data was left off.

Trying this out now. I had to stop the source node for 20 minutes to make a local copy. Now I am copying 60GB worth of data to the new server. Unfortunately this is a single threaded SSH/SCP operation and the servers are 6000 miles apart (high latency), so it remains to be seen if I save any time at all.

Only the download speed at your server to the fully synchronized node and the upload speed from fully synchronized node matters. I wouldn't recommend going with this way, the client would still have to verify every single block. With the CPU as a bottleneck, it would likely take far more time. Bitcoin Core with headers-first synchronization downloads from various nodes simultaneously so it would be much faster.

[BitAurum]

Can someone tell me what is the best practice for initializing a new bitcoind node quickly? Do I really need to wait 2 days for the blockchain to be downloaded and validated before the node is operational? As far as I can tell most of the time is spent not on network resources but on CPU resources, at least for the hardware I am running (Azure & Amazon EC).

If I already have another fully initalized node that I trust, is there anyway I can just copy the blockchain database from a trusted node over to the new node (bypassing all the time consuming validation)?

If you have access (can login and copy stuff to and from) to a fully synced node, you could copy over most of the stuff in the data directory and the node would just pick from where that data was left off.

Trying this out now. I had to stop the source node for 20 minutes to make a local copy. Now I am copying 60GB worth of data to the new server. Unfortunately this is a single threaded SSH/SCP operation and the servers are 6000 miles apart (high latency), so it remains to be seen if I save any time at all.

Only the download speed at your server to the fully synchronized node and the upload speed from fully synchronized node matters. I wouldn't recommend going with this way, the client would still have to verify every single block. With the CPU as a bottleneck, it would likely take far more time. Bitcoin Core with headers-first synchronization downloads from various nodes simultaneously so it would be much faster.

The new client won't have to verify every single block if the entire .bitcoin directory is copied from another fully initialized node.

[Soros Shorts]

Update: So the file copy method works better. Both servers are 100 Mbps down/up, but with a single SSH session I was only able to get 7 Mbps throughput.  I ended up having to write a script to split the files up into 10 groups and run parallel copy operations over 10 different SSH sessions to get close to the max. 100 Mbps throughput and was able to initialize a new node in under 3 hours compared with 2+ days the normal way.

As a test, I also set up a temp web server on the source node and tested copying the files over using the HTTP protocol. It turned out to be a lot faster than SSH for the single threaded copy, but I still needed to run 2 parallel copy operations to max. out my 100 Mbps bandwidth. This is understandable as SSH/SCP is a chattier protocol than HTTP and generates more "dead air" over a WAN.

So yes, copying the files over is a lot quicker than just starting up a blank node from scratch. but you still need to some work on your part to get the most of your available bandwidth.

[mocacinno]

Update: So the file copy method works better. Both servers are 100 Mbps down/up, but with a single SSH session I was only able to get 7 Mbps throughput.  I ended up having to write a script to split the files up into 10 groups and run parallel copy operations over 10 different SSH sessions to get close to the max. 100 Mbps throughput and was able to initialize a new node in under 3 hours compared with 2+ days the normal way.

As a test, I also set up a temp web server on the source node and tested copying the files over using the HTTP protocol. It turned out to be a lot faster than SSH for the single threaded copy, but I still needed to run 2 parallel copy operations to max. out my 100 Mbps bandwidth. This is understandable as SSH/SCP is a chattier protocol than HTTP and generates more "dead air" over a WAN.

So yes, copying the files over is a lot quicker than just starting up a blank node from scratch. but you still need to some work on your part to get the most of your available bandwidth.

I've always found SSH slow, but you can speed it up by using a different algo:

https://www.neuhalfen.name/2009/02/04/scp_performance_gain_by_using_right_algorithm/

you can try it out next time

[BitAurum]

So yes, copying the files over is a lot quicker than just starting up a blank node from scratch. but you still need to some work on your part to get the most of your available bandwidth.

You will need to do some work only if you are in a hurry. At 50 Mbps(Max of a single-threaded HTTP) it will only take ~1-2 hours to copy the entire blockchain. Still better than starting Bitcoin Core from fresh.