Bitcoin node cluster

[samuel-sd]

I would like to implement this architecture:
- the nfs storage with bitcoin blockchain
- the main instance that keeps blockchain updated
- A few instances that allow other software to interact with node.

Using a few instances let me scale in and out, gradually update nodes and etc.

Is it possible to set up using bitcoind?

[1715120288]

1715120288

[1715120288]

1715120288

[jackg]

This might not be the best place to get help with this as it's a little more operating system based and people who deal with that os might be best to advise you...

A few things we should get out of the way is what operating system do you plan to use? Linux would be much easier for achieving this but it's not the only option. Are you comfortable with servers and virtual server management (I don't think it's a high learning curve if not).

Are you wanting isolated instances of bitcoind for each task or do you want a central machine that takes requests from the rest of the nodes on the network?

[samuel-sd]

This might not be the best place to get help with this as it's a little more operating system based and people who deal with that os might be best to advise you...

My questions are not about OS settings, they are about bitcoind service and its properties.

A few things we should get out of the way is what operating system do you plan to use?

It is going to be linux, there is no doubt about it.

Are you comfortable with servers and virtual server management

Yes, I am a devops engineer, that's what I do for a living. I am pretty comfortable with Linuxes and cloud infrastructure.

Are you wanting isolated instances of bitcoind for each task or do you want a central machine that takes requests from the rest of the nodes on the network?

I would like to keep them isolated but they all have to be connected to the shared NFS storage. The main idea is to make a scalable robust solution that allows to serve high workload.
The problem is I don't know if a few bitcoind services can work with one shared (NFS) database and what special settings I need to implement.

[HeRetiK]

I would like to keep them isolated but they all have to be connected to the shared NFS storage. The main idea is to make a scalable robust solution that allows to serve high workload.
The problem is I don't know if a few bitcoind services can work with one shared (NFS) database and what special settings I need to implement.

It seems to be theoretically possible, but may lead to data corruption as it's not a use case that has been accounted for.

Here's a related discussion from 2016.
https://bitcointalk.org/index.php?topic=1340309.0

Here's a guy on reddit discussing how they approached the matter:
https://www.reddit.com/r/btc/comments/4r48oa/running_a_node_with_the_database_on_a/d4yolwm/

[NotATether]

Running copies of bitcoind on different nodes is equivalent to running multiple full nodes, all independent from each other. But you can import your private keys from one bitcoind node to another and a bunch of bitcoind nodes with the same private keys are for all purposes equivalent and you can select one at random to run an RPC call on.

It's easy to put the blockchain on an NFS mount, just create and mount the NFS filesystem and run bitcoind with the --data-dir switch and point it at your NFS mount path (--data-dir /mount/your/nfs/path). For integrity purposes, I don't think you should share the blockchain with each bitcoind node because each bitcoind instance assumes it is the sole controller of the blockchain data. So you can make a folder in your mount for each bitcoind node so they have their own copy of the blockchain.

Your main bottleneck will be disk read/write speed, especially read.

It’s common for full nodes on high-speed connections to use 200 gigabytes upload or more a month. Download usage is around 20 gigabytes a month, plus around an additional 340 gigabytes the first time you start your node.

So your required reading and writing throughput is multiplied by the number of nodes you are running, since all of them are writing to the same disk. So not only will the disk be very slow, it will fail quicker, especially if it's an SSD. Get a bunch of HDD drives on RAID 0 and put NFS on that. RAID 0 will make performance bearable with several nodes writing to it. It won't solve the disk failure problem but that's bearable because the blockchain can just be downloaded again with no loss of bitcoins.

[DaveF]

Yes it can be done. I tried to do it and succeeded. But, no really your should not.
You can have multiple bicoind talk to the same network location for data, it's just a line in a config file. And you can allow multiple places to connect to it.
BUT if any of the nodes other then the main attempt to write to it. It's going to lead to data corruption.

Seriously, if you are working on a project that really needs it a pair of mirrored Samsung PM883 480GB drives is less then $250. And that should hold you for at least another couple of years for blockchain storage. So even if you need 4 sets it's still under $1k USD. Not a totally insignificant amount but it still should not be enough to kill a project.

Not "may lead", but "will lead". There's a reason Bitcoin Core shows warning "Error: Cannot obtain a lock on ...." if you attempt to run 2 bitcoin core process.

That is also is (was? have not done it in a while) quick config change to work around, but yeah, really should not be done.

-Dave

[samuel-sd]

Thanks for the answers. As far as I understood, we have two problems:
1. bitcoind service wants to read and write (fully control) the blockchain. The bitcoind service is not designed to share its database with other replicas.
2. NFS storage speed will not allow me to scale my architecture properly.

Let's discus a little bit different architecture. I would like to have redundancy in case if one node goes down or the datacenter looses connectivity. Can I put a few nodes behind a loadbalancer?
I am curious how do people implement it for serious projects, where the service cannot be stopped because the only bitcoind server went down.

[HeRetiK]

Thanks for the answers. As far as I understood, we have two problems:
1. bitcoind service wants to read and write (fully control) the blockchain. The bitcoind service is not designed to share its database with other replicas.
2. NFS storage speed will not allow me to scale my architecture properly.

Let's discus a little bit different architecture. I would like to have redundancy in case if one node goes down or the datacenter looses connectivity. Can I put a few nodes behind a loadbalancer?
I am curious how do people implement it for serious projects, where the service cannot be stopped because the only bitcoind server went down.

I'm not sure about exchanges or crypto casinos, but Trezor / SatoshiLabs for example has multiple full nodes running (for both Bitcoin and the various alts they are supporting):

https://status.trezor.io/

Visiting the servers in your browser (btc1.trezor.io, etc) will lead you to each respective blockchain explorer, where you can see that each node has a slightly different state. Even the last block update timestamp may vary by a second or two, so you see that they are using multiple full nodes rather than multiple clients sharing the same blockchain data.

[NotATether]

Let's discus a little bit different architecture. I would like to have redundancy in case if one node goes down or the datacenter looses connectivity. Can I put a few nodes behind a loadbalancer?
I am curious how do people implement it for serious projects, where the service cannot be stopped because the only bitcoind server went down.

Yes you can. Like I said before, simply importing the same private keys into each bitcoind node will make them all have the same state of the keys. So you can design a load balancer that makes an RPC call to sign or broadcast a transaction from one of the non-network-congested nodes, and the rest of the nodes will automatically mirror the new state by virtue of the blockchain. You can then query wallet state from one of the other nodes and you will get up-to-date information from it.

You still have to update each bitcoind node separately though, but that should not be a problem. You're using Linux for this, you can automate the updating process by running automation software such as Puppet or Ansible that downloads the latest bitcoind version from the package manager.

[DaveF]

Thanks for the answers. As far as I understood, we have two problems:
1. bitcoind service wants to read and write (fully control) the blockchain. The bitcoind service is not designed to share its database with other replicas.
2. NFS storage speed will not allow me to scale my architecture properly.

Let's discus a little bit different architecture. I would like to have redundancy in case if one node goes down or the datacenter looses connectivity. Can I put a few nodes behind a loadbalancer?
I am curious how do people implement it for serious projects, where the service cannot be stopped because the only bitcoind server went down.

The issue with a loadbalancer is that unless you have 2 or more with failover it's just another single point of failure.
Using the trezor method that HeRetiK posted is a good solution so long as you have a no connect timeout set in your app. #1 does not respond in "X" seconds go to #2 and so on.

And then you need an alert system to let you know that #1 did not respond.

And then some inter-connectivity between data centers so you have geographic redundancy.

And then some app that checks the status of the nodes to make sure that they are all in sync.

-Dave

[samuel-sd]

The issue with a loadbalancer is that unless you have 2 or more with failover it's just another single point of failure.

Cloud providers take care about maintaining infrastructure, it's not a problem anymore.

Trying to build it up yourself will cost more and will be less reliable.