|
Title: Trusted blockchain download source for mass installation Post by: poordeveloper on March 09, 2019, 01:05:03 PM I'm planning to setup around 20 full nodes, and in order to make the process as fast as possible I was looking for a way to speed the "bootstrap" process.
Last time I did this there was a torrent regularly updated I think, as well as a site with a public download. That was quite a few years back - now I've found a few of them, but I'm unsure if any of them are maintained by the community / a trusted community member, as I'd obviously prefer to use a trusted data source. Title: Re: Trusted blockchain download source for mass installation Post by: LoyceV on March 09, 2019, 01:25:18 PM From what I've seen, it's best to have Bitcoin Core download everything on it's own.
If you run 20 nodes in the same data center, it might be worth using something like rsync to transfer data from one server to all others. Title: Re: Trusted blockchain download source for mass installation Post by: poordeveloper on March 09, 2019, 01:36:47 PM From what I've seen, it's best to have Bitcoin Core download everything on it's own. Thanks a lot for your reply.If you run 20 nodes in the same data center, it might be worth using something like rsync to transfer data from one server to all others. The plan is for them to be located on as many different locations as possible. What is the bottleneck with the current version for the download? I'm asking because if it's the network, I can get a 10G server to download the data and use rsync later as you said (some locations have quite limited ports, and I think rsync will be faster in those cases). Title: Re: Trusted blockchain download source for mass installation Post by: LoyceV on March 09, 2019, 01:49:27 PM What is the bottleneck with the current version for the download? From my experience: it's either CPU, or disk IO.Title: Re: Trusted blockchain download source for mass installation Post by: Jet Cash on March 09, 2019, 03:21:29 PM My bottleneck was bandwidth and peer activity, but I was using public WiFi. I don't recommend this. Using an SSD rather than an HDD does make a big difference when synchronising.
Title: Re: Trusted blockchain download source for mass installation Post by: ABCbits on March 09, 2019, 06:16:16 PM There's too much risks downloading blockchain from 3rd party, you better download whole blockchain once on fast PC with fast internet connection then copy it to other 20 devices.
But if you like to live dangerously, there are few sites/thread about it : 1. https://bitcointalk.org/index.php?topic=1310261.0 (https://bitcointalk.org/index.php?topic=1310261.0) (outdated) 2. https://getbitcoinblockchain.com/ (https://getbitcoinblockchain.com/) (rarely see people mention this website) 3. https://bitcointalk.org/index.php?topic=2321650 (https://bitcointalk.org/index.php?topic=2321650) (also outdated) What is the bottleneck with the current version for the download? I'm asking because if it's the network, I can get a 10G server to download the data and use rsync later as you said (some locations have quite limited ports, and I think rsync will be faster in those cases). Ordered by most-common reason : 1. Disk I/O 2. Sync speed 3. CPU 4. RAM I doubt your 20 devices uses SSD (or have it in big capacity), so you definitely will experience bottleneck on Disk I/O. Title: Re: Trusted blockchain download source for mass installation Post by: NeuroticFish on March 09, 2019, 08:00:58 PM I've read somewhere on Bitcointalk at some point, maybe more than one year ago, that downloading the blockchain bootstrap file has become meaningless because the bottleneck is usually not the internet connection (it's the HDD).
The only use would be to save some bandwidth if you have physical access to all those devices. But does it worth the risk? Title: Re: Trusted blockchain download source for mass installation Post by: DaCryptoRaccoon on March 11, 2019, 08:59:57 PM https://bitcoin.org/bin/block-chain/
Found this torrent link on the bitcoin.org website might be useful. Note the block height is 317000 for this. I have no clue how this is maintained or updated someone else might have a better answer for you. Code: -----BEGIN PGP SIGNED MESSAGE----- Title: Re: Trusted blockchain download source for mass installation Post by: achow101 on March 12, 2019, 04:26:16 AM https://bitcoin.org/bin/block-chain/ It is not maintained or updated at all anymore. That's the bootstrap.dat file. It will be slower to download that and sync off of it than it will be to let Bitcoin Core do its normal thing to sync.Found this torrent link on the bitcoin.org website might be useful. Note the block height is 317000 for this. I have no clue how this is maintained or updated someone else might have a better answer for you. The bottlenecks you will probably experience will be Disk I/O, CPU, and network speed. As mentioned by others, the fastest way for you would be to sync one node and then copy that to all of the other nodes. You can speed up that initial node's sync time by increase the dbcache if you have a lot of RAM available (>16 GB). Setting dbcache=8000 will yield the best results as the entire database will be able to be held in memory, thus reducing Disk I/O. Title: Re: Trusted blockchain download source for mass installation Post by: StackGambler on March 12, 2019, 10:18:08 AM Sorry, I'm a little new here. I have two questions:
Firstly, to OP. What is the purpose of running 20 nodes? Is it for a bitcoin service, or just to help the network? Secondly, to everyone. It seems like having a torrent copy of the blockchain (one that's regularly updated) is something people might be interested in. How would I go about creating one for people to use? I have a synced full node on Windows. Thanks guys. Title: Re: Trusted blockchain download source for mass installation Post by: ranochigo on March 12, 2019, 04:24:50 PM Secondly, to everyone. It seems like having a torrent copy of the blockchain (one that's regularly updated) is something people might be interested in. How would I go about creating one for people to use? I have a synced full node on Windows. If you want, you can just copy out your whole Bitcoin data directory, then delete your wallet.dat from your copied data directory. Use a torrent program to share the file. You do need quite a number of seeders since the data directory (blockchain) is relatively big and a large number of seeders is required to spread the load evenly. It might be quite hard to get enough people to seed the file especially since it is not to their best interest to keep two blockchains on the same computer.Title: Re: Trusted blockchain download source for mass installation Post by: jackg on March 12, 2019, 05:50:28 PM Secondly, to everyone. It seems like having a torrent copy of the blockchain (one that's regularly updated) is something people might be interested in. How would I go about creating one for people to use? I have a synced full node on Windows. Thanks guys. You can use utorrent to seed a torrent directory (or other torrenting software) and transmit it to others.. You'd have to put it into a different folder though and potentially compress it if possible to make the most of your bandwidth. If you advertise it remember to post a hash of it so people can verify it (normally these torrents ignore signatures that are normally required for network transmissions so a MITM attack is possible). The 7zip file manager has some nice hashing features. Title: Re: Trusted blockchain download source for mass installation Post by: Subspace_Labs on March 12, 2019, 08:14:48 PM It seems like having a torrent copy of the blockchain (one that's regularly updated) is something people might be interested in. How would I go about creating one for people to use? I have a synced full node on Windows. Thanks guys. It sounds like a good idea but it's really not for many people. The problem is that you have to trust the creator of the torrent and trustlessness is one of the main features of Bitcoin. As others have said, downloading it via Bitcoin Core isn't much more resource intensive if you're using a desktop. If you're using some sort of low-power device like a Raspberry Pi, then it will take weeks at best to download and index. If you're using a device like that it's best to download it via Bitcoin Core on a desktop and then copy the data directory over to the Pi. In this case the bottleneck would be CPU and memory. With a fast enough desktop the bottleneck becomes bandwidth. Title: Re: Trusted blockchain download source for mass installation Post by: HCP on March 12, 2019, 08:18:05 PM Secondly, to everyone. It seems like having a torrent copy of the blockchain (one that's regularly updated) is something people might be interested in. How would I go about creating one for people to use? I have a synced full node on Windows. My advice would be "Don't Do it"... you would be simply perpetuating the "myth" that downloading the blockchain via a torrent will decrease the time required to sync... it won't. It is likely to actually be slower than just letting Bitcoin Core do it's thing and sync normally.Remember... syncing the blockchain is NOT just simply downloading blocks. For the record, achow101 is a very knowledgeable Bitcoin Core dev... he knows what he is talking about... It is not maintained or updated at all anymore. That's the bootstrap.dat file. It will be slower to download that and sync off of it than it will be to let Bitcoin Core do its normal thing to sync. The bottlenecks you will probably experience will be Disk I/O, CPU, and network speed. As mentioned by others, the fastest way for you would be to sync one node and then copy that to all of the other nodes. You can speed up that initial node's sync time by increase the dbcache if you have a lot of RAM available (>16 GB). Setting dbcache=8000 will yield the best results as the entire database will be able to be held in memory, thus reducing Disk I/O. Title: Re: Trusted blockchain download source for mass installation Post by: jackg on March 12, 2019, 09:43:40 PM @hcp the psychology of doing the data download and thenstarting up core is a potential upside though. It's faster to scan the blockchain while you have a physical copy of it and you take out all the downloading to start with. Downloading and syncing via core may be faster but the two individual steps in their own are also faster and more psychologically appealing as it feels like the sync is going faster (from my own experience a few versions ago, it might be out of date entirely).
|