poordeveloper (OP)
|
I'm planning to setup around 20 full nodes, and in order to make the process as fast as possible I was looking for a way to speed the "bootstrap" process.
Last time I did this there was a torrent regularly updated I think, as well as a site with a public download.
That was quite a few years back - now I've found a few of them, but I'm unsure if any of them are maintained by the community / a trusted community member, as I'd obviously prefer to use a trusted data source.
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3444
Merit: 17427
Thick-Skinned Gang Leader and Golden Feather 2021
|
From what I've seen, it's best to have Bitcoin Core download everything on it's own. If you run 20 nodes in the same data center, it might be worth using something like rsync to transfer data from one server to all others.
|
|
|
|
poordeveloper (OP)
|
|
March 09, 2019, 01:36:47 PM |
|
From what I've seen, it's best to have Bitcoin Core download everything on it's own. If you run 20 nodes in the same data center, it might be worth using something like rsync to transfer data from one server to all others.
Thanks a lot for your reply. The plan is for them to be located on as many different locations as possible. What is the bottleneck with the current version for the download? I'm asking because if it's the network, I can get a 10G server to download the data and use rsync later as you said (some locations have quite limited ports, and I think rsync will be faster in those cases).
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3444
Merit: 17427
Thick-Skinned Gang Leader and Golden Feather 2021
|
What is the bottleneck with the current version for the download? From my experience: it's either CPU, or disk IO.
|
|
|
|
Jet Cash
Legendary
Offline
Activity: 2786
Merit: 2471
https://JetCash.com
|
My bottleneck was bandwidth and peer activity, but I was using public WiFi. I don't recommend this. Using an SSD rather than an HDD does make a big difference when synchronising.
|
Offgrid campers allow you to enjoy life and preserve your health and wealth. Save old Cars - my project to save old cars from scrapage schemes, and to reduce the sale of new cars. My new Bitcoin transfer address is - bc1q9gtz8e40en6glgxwk4eujuau2fk5wxrprs6fys
|
|
|
ABCbits
Legendary
Offline
Activity: 3010
Merit: 7896
Crypto Swap Exchange
|
There's too much risks downloading blockchain from 3rd party, you better download whole blockchain once on fast PC with fast internet connection then copy it to other 20 devices. But if you like to live dangerously, there are few sites/thread about it : 1. https://bitcointalk.org/index.php?topic=1310261.0 (outdated) 2. https://getbitcoinblockchain.com/ (rarely see people mention this website) 3. https://bitcointalk.org/index.php?topic=2321650 (also outdated) What is the bottleneck with the current version for the download? I'm asking because if it's the network, I can get a 10G server to download the data and use rsync later as you said (some locations have quite limited ports, and I think rsync will be faster in those cases).
Ordered by most-common reason : 1. Disk I/O 2. Sync speed 3. CPU 4. RAM I doubt your 20 devices uses SSD (or have it in big capacity), so you definitely will experience bottleneck on Disk I/O.
|
|
|
|
NeuroticFish
Legendary
Offline
Activity: 3808
Merit: 6523
Looking for campaign manager? Contact icopress!
|
|
March 09, 2019, 08:00:58 PM |
|
I've read somewhere on Bitcointalk at some point, maybe more than one year ago, that downloading the blockchain bootstrap file has become meaningless because the bottleneck is usually not the internet connection (it's the HDD).
The only use would be to save some bandwidth if you have physical access to all those devices. But does it worth the risk?
|
|
|
|
DaCryptoRaccoon
|
|
March 11, 2019, 08:59:57 PM |
|
https://bitcoin.org/bin/block-chain/Found this torrent link on the bitcoin.org website might be useful. Note the block height is 317000 for this. I have no clue how this is maintained or updated someone else might have a better answer for you. -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
Torrent info hash: 36719ba2cecf9f3bd7c5abfb7a88e939611b536c Torrent magnet link: magnet:?xt=urn:btih:36719ba2cecf9f3bd7c5abfb7a88e939611b536c&dn=bootstrap.dat&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Ftracker.publicbt.com%3A80&tr=udp%3A%2F%2Fcoppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=http%3A%2F%2Fbttracker.crunchbanglinux.org%3A6969%2Fannounce
Block height: 317000 Filename: bootstrap.dat Byte size: 22566124235 SHA1: 2e6e5dd20cd9c21960ba94f7265f42c4b6443758 SHA256: d05062052be4dd6fbaeac5f17598e52f7ad545f9f938acefaf7067ecce1d7b5a
-----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: GPGTools - http://gpgtools.org
iQEcBAEBCgAGBQJT+PcrAAoJEL8a4JGi25zKmj4H/i8xKIXW/xnw5i5Pc+ciUuc2 Zo+KxQd1YGRmJSvGsyTgTqK4IJUBD8s8tIGTtznCJ2/Q3wIdZmoqxkST7r4rqJly cOyJ/n19K5beYvVekIgHVXFmhdJhZf8KNnyuV+DpHmmLKYyZtvkoY9b3hEBHVtSM AkjuWyA9IEPeghxBX32JAToXOZGs9vZghNyIMg/fWbGLuv1oX7QCjAAua0NAMbLX diSvVcLXskTRy38W4whz1DX6ruL8mPRYH5Ko6/cmtVqEkr53aEwYf1XJGHHAz3+J nEHecxBUbiRhDwMdCVtqzKudXWVDtkgVQmRxrkKTw63NTVMge4eEziE2+7JQgq8= =p9TY -----END PGP SIGNATURE-----
|
┏━━━━━━━━━━━━━━━━━┓ ┃ 𝔱𝔥𝔬𝔲 𝔰𝔥𝔞𝔩𝔱 𝔴𝔬𝔯ⱪ 𝔣𝔬𝔯 𝔶𝔬𝔲𝔯 𝔟𝔞𝔤𝔰 ┃ ┃ ➤21/M ┃ ┃ ███▓▓ ███▓▓ ███▓▓ ███▓▓┃
|
|
|
achow101
Moderator
Legendary
Offline
Activity: 3500
Merit: 6833
Just writing some code
|
|
March 12, 2019, 04:26:16 AM |
|
https://bitcoin.org/bin/block-chain/Found this torrent link on the bitcoin.org website might be useful. Note the block height is 317000 for this. I have no clue how this is maintained or updated someone else might have a better answer for you. It is not maintained or updated at all anymore. That's the bootstrap.dat file. It will be slower to download that and sync off of it than it will be to let Bitcoin Core do its normal thing to sync.
The bottlenecks you will probably experience will be Disk I/O, CPU, and network speed. As mentioned by others, the fastest way for you would be to sync one node and then copy that to all of the other nodes. You can speed up that initial node's sync time by increase the dbcache if you have a lot of RAM available (>16 GB). Setting dbcache=8000 will yield the best results as the entire database will be able to be held in memory, thus reducing Disk I/O.
|
|
|
|
StackGambler
Full Member
Offline
Activity: 434
Merit: 101
YouTuber, gambler, and scam-buster.
|
|
March 12, 2019, 10:18:08 AM |
|
Sorry, I'm a little new here. I have two questions:
Firstly, to OP. What is the purpose of running 20 nodes? Is it for a bitcoin service, or just to help the network?
Secondly, to everyone. It seems like having a torrent copy of the blockchain (one that's regularly updated) is something people might be interested in. How would I go about creating one for people to use? I have a synced full node on Windows.
Thanks guys.
|
I like gambling. Probably currently trying to figure out how to pay next month's rent.
|
|
|
ranochigo
Legendary
Offline
Activity: 3038
Merit: 4418
Crypto Swap Exchange
|
|
March 12, 2019, 04:24:50 PM |
|
Secondly, to everyone. It seems like having a torrent copy of the blockchain (one that's regularly updated) is something people might be interested in. How would I go about creating one for people to use? I have a synced full node on Windows.
If you want, you can just copy out your whole Bitcoin data directory, then delete your wallet.dat from your copied data directory. Use a torrent program to share the file. You do need quite a number of seeders since the data directory (blockchain) is relatively big and a large number of seeders is required to spread the load evenly. It might be quite hard to get enough people to seed the file especially since it is not to their best interest to keep two blockchains on the same computer.
|
|
|
|
jackg
Copper Member
Legendary
Offline
Activity: 2856
Merit: 3071
https://bit.ly/387FXHi lightning theory
|
|
March 12, 2019, 05:50:28 PM |
|
Secondly, to everyone. It seems like having a torrent copy of the blockchain (one that's regularly updated) is something people might be interested in. How would I go about creating one for people to use? I have a synced full node on Windows.
Thanks guys.
You can use utorrent to seed a torrent directory (or other torrenting software) and transmit it to others.. You'd have to put it into a different folder though and potentially compress it if possible to make the most of your bandwidth. If you advertise it remember to post a hash of it so people can verify it (normally these torrents ignore signatures that are normally required for network transmissions so a MITM attack is possible). The 7zip file manager has some nice hashing features.
|
|
|
|
Subspace_Labs
Newbie
Offline
Activity: 3
Merit: 2
|
It seems like having a torrent copy of the blockchain (one that's regularly updated) is something people might be interested in. How would I go about creating one for people to use? I have a synced full node on Windows.
Thanks guys.
It sounds like a good idea but it's really not for many people. The problem is that you have to trust the creator of the torrent and trustlessness is one of the main features of Bitcoin. As others have said, downloading it via Bitcoin Core isn't much more resource intensive if you're using a desktop. If you're using some sort of low-power device like a Raspberry Pi, then it will take weeks at best to download and index. If you're using a device like that it's best to download it via Bitcoin Core on a desktop and then copy the data directory over to the Pi. In this case the bottleneck would be CPU and memory. With a fast enough desktop the bottleneck becomes bandwidth.
|
|
|
|
HCP
Legendary
Offline
Activity: 2086
Merit: 4318
<insert witty quote here>
|
|
March 12, 2019, 08:18:05 PM |
|
Secondly, to everyone. It seems like having a torrent copy of the blockchain (one that's regularly updated) is something people might be interested in. How would I go about creating one for people to use? I have a synced full node on Windows.
My advice would be "Don't Do it"... you would be simply perpetuating the "myth" that downloading the blockchain via a torrent will decrease the time required to sync... it won't. It is likely to actually be slower than just letting Bitcoin Core do it's thing and sync normally. Remember... syncing the blockchain is NOT just simply downloading blocks. For the record, achow101 is a very knowledgeable Bitcoin Core dev... he knows what he is talking about... It is not maintained or updated at all anymore. That's the bootstrap.dat file. It will be slower to download that and sync off of it than it will be to let Bitcoin Core do its normal thing to sync.
The bottlenecks you will probably experience will be Disk I/O, CPU, and network speed. As mentioned by others, the fastest way for you would be to sync one node and then copy that to all of the other nodes. You can speed up that initial node's sync time by increase the dbcache if you have a lot of RAM available (>16 GB). Setting dbcache=8000 will yield the best results as the entire database will be able to be held in memory, thus reducing Disk I/O.
|
|
|
|
jackg
Copper Member
Legendary
Offline
Activity: 2856
Merit: 3071
https://bit.ly/387FXHi lightning theory
|
|
March 12, 2019, 09:43:40 PM |
|
@hcp the psychology of doing the data download and thenstarting up core is a potential upside though. It's faster to scan the blockchain while you have a physical copy of it and you take out all the downloading to start with. Downloading and syncing via core may be faster but the two individual steps in their own are also faster and more psychologically appealing as it feels like the sync is going faster (from my own experience a few versions ago, it might be out of date entirely).
|
|
|
|
|