Electrum, high availability and horizontal scalability

[ayadev]

Hey,

I've been reading some topics on the message board stating that Electrum wasn't design to handle wallets with a large addresses/transactions history, although there are no hard-coded limitations.

In a merchant situation where a lot of addresses and transactions occur on a daily basis (e.g naivly 100 addresses for 100 transactions a day), does it make sense to "load balance" payments using multiple wallets (e.g naivly 1 wallet for 10 transactions a day = 10 instances of wallets for 100 tansactions a day) ? Does anybody experienced such architecture ?

Thanks

[1713884118]

1713884118

[1713884118]

1713884118

[1713884118]

1713884118

[1713884118]

1713884118

[ranochigo]

In a merchant situation where a lot of addresses and transactions occur on a daily basis (e.g 100 addresses for 100 transactions a day), does it make sense to "load balance" payments using multiple wallets ? Does anybody experienced such architecture ?

The way I've seen and recommend people to implement if they're using Electrum is to only include the master public key on the production server and store the seeds and the main wallet on a separate server. The payment is usually done by checking against some block explorer APIs or through an Electrum running in watch-only mode on the server.

Okay that aside, the main problem doesn't arise until you need to reload the wallet and/or spend the huge number of UTXOs. Spending such transactions or loading them will cause Electrum to lag as it will involves large amount of data being transferred from the Electrum servers and IIRC, some has restrictions on it as well.

I don't recommend people using Electrum as a merchant because it has loads of limitations combined with the lackluster RPC interface (perhaps I'm asking for too much) that I've encountered. Bitcoin Core is a much better choice in comparison and you won't face much problems either.

[ayadev]

Hi @ranochigo,

Thanks for your answer.

Maybe I should have started by looking at Bitcoin Core to have a better overview. I will take a look at it asap.

To be more precise, the idea behind this would be to automatically create a new Electrum wallet and run it as a read-only wallet (in a new containerized instance) each time a limitation is reached (e.g gap limit), avoiding as much as possible the limitations you mentioned. This is theory.

Could you give me some details about what is missing (in your opinion) in Electrum RPC interface ?



  




  

[ranochigo]

To be more precise, the idea behind this would be to automatically create a new Electrum wallet and run it as a read-only wallet (in a new containerized instance) each time a limitation is reached (e.g gap limit), avoiding as much as possible the limitations you mentioned. This is theory.

I imagine that it'll be a hassle to keep track of the wallets. You'll probably have to change out for a new wallet everyday. Gap limit is not a limitation, it's just how far your wallet will read ahead so no issues there. You would have to account for those who would generate an address for payment but not pay in the end as well.

Could you give me some details about what is missing (in your opinion) in Electrum RPC interface

It's more of the few developers-centric RPCs (getrawmempool,etc) which wouldn't really affect normal users.

[pooya87]

The problem comes from the fact that Electrum writes the wallet file in simple JSON format which it looks up later so a better approach would be to use everything in Electrum except parts of the wallet layer.
It would require more work but theoretically with some modification to the code you could replace the wallet file (the way it is written to disk and read) with a database instead so that looking up inside millions of inputs becomes super fast and use that with the rest of the Electrum code for signing, downloading txs and blocks from nodes, ...

[ayadev]

I imagine that it'll be a hassle to keep track of the wallets. You'll probably have to change out for a new wallet everyday. Gap limit is not a limitation, it's just how far your wallet will read ahead so no issues there.

Yes I was wrong, gab limit is not a limitation nor an issue (except when you have to restore a wallet using a seed if I understood well). I assumed gab limit could be used to delimit how many address should be used per wallet instance, but this is not relevant at all. I still need to do some homework)

You would have to account for those who would generate an address for payment but not pay in the end as well.

In my situation I guess I can mitigate this by expiration times and limitating how many pending requests a user can generate at the same time.

On architecture side, one solution would be to look for expired requests and reuse addresses, assuming that reusing an address which has never been funded is an acceptable privacy risk. But again, I may be missing something.

It's more of the few developers-centric RPCs (getrawmempool,etc) which wouldn't really affect normal users.

I didn't know about solutions such as getrawmempool thanks!

I'm mainly using Electrum because I guess this wallet provides a "lightweight" additionnal layer as well as some valuable features (secret seed, client-side stored private keys, socks5 proxy support, etc.). At this point, I don't have enough knowledge nor experience to deal with a full node directly.

[ranochigo]

In my situation I guess I can mitigate this by expiration times and limitating how many pending requests a user can generate at the same time.

On architecture side, one solution would be to look for expired requests and reuse addresses, assuming that reusing an address which has never been funded is an acceptable privacy risk. But again, I may be missing something.

It's more of a hassle if the User A accidentally sends funds to the address after it's expiration but it has already been assigned to User B. Such a scenario can be tricky to handle.

That being said, I think you might be able to do with having addresses that doesn't have any transactions since those wouldn't incur any resource penalty when querying them with an Electrum server. I missed that.

I didn't know about solutions such as getrawmempool thanks!

I'm mainly using Electrum because I guess this wallet provides a "lightweight" additionnal layer as well as some valuable features (secret seed, client-side stored private keys, socks5 proxy support, etc.). At this point, I don't have enough knowledge nor experience to deal with a full node directly.

I wouldn't say Electrum is the best candidate because it isn't designed for heavy workload in the first place and having to rely on someone else's Bitcoin node to tell you the correct information doesn't sit too well with me. You do have a point with the issue about xpub, since Bitcoin Core won't derive any xpub. IMO, it would be complicated to manage and swap wallets every now and then. If that's something that you're able to accommodate then I don't think there's too much of a problem. If you'd like, there are services like Bitpay which helps to streamline the process, at the expense of it being a centralised service.

[ayadev]

The problem comes from the fact that Electrum writes the wallet file in simple JSON format which it looks up later so a better approach would be to use everything in Electrum except parts of the wallet layer.
It would require more work but theoretically with some modification to the code you could replace the wallet file (the way it is written to disk and read) with a database instead so that looking up inside millions of inputs becomes super fast and use that with the rest of the Electrum code for signing, downloading txs and blocks from nodes, ...

That's an interesting point of view. I guess JSON is a core design choice made by the developer(s) to avoid dependencies and because they assume that most users won't have to deal with a large wallet. However a database backend would be a merchant requirement for sure. It should be studied more closely.
 

[ayadev]

It's more of a hassle if the User A accidentally sends funds to the address after it's expiration but it has already been assigned to User B. Such a scenario can be tricky to handle.

That being said, I think you might be able to do with having addresses that doesn't have any transactions since those wouldn't incur any resource penalty when querying them with an Electrum server. I missed that.

In the other hand, if I run a new wallet instance every time "10" addresses have been assigned to a payment request (funded or not), it could become a substantial waste of resource on my side...not to mention managing hundreds or thousands of wallets is probably a funny nightmare.

Another solution would be to not reuse an address for a "very raisonnable" period of time in order to mitigate this risk.

I wouldn't say Electrum is the best candidate because it isn't designed for heavy workload in the first place and having to rely on someone else's Bitcoin node to tell you the correct information doesn't sit too well with me.

I'm actually trying to estimate that design weakness and finding a creative way to bypass it... even though I don't really know where I'm going at right now but it's getting better)

As far as I know you can run your own Electrum server and make all your wallets connect to that particular server only. However, it could become a single point of failure. Of course you can also run your own Bitcoin node if this is a critical business requirement. But in the end (IMO) we always have to rely on a third-party provider at some point.

You do have a point with the issue about xpub, since Bitcoin Core won't derive any xpub. IMO, it would be complicated to manage and swap wallets every now and then. If that's something that you're able to accommodate then I don't think there's too much of a problem.

I guess this architecture could survive as long as it follows very simple rules and avoid complexity as much as possible (I mean, in terms of wallet management/swap policy)

If you'd like, there are services like Bitpay which helps to streamline the process, at the expense of it being a centralised service.

Yes I already took a (short) look at Bitpay, as well as BTCPay server which is API compliant, as a possible alternative. However at this point I actually need to avoid centralized services as well as third-party payment providers. This is a technical/business requirement.

Thanks again for your answers and point of view!

[Abdussamad]

Hey,

I've been reading some topics on the message board stating that Electrum wasn't design to handle wallets with a large addresses/transactions history, although there are no hard-coded limitations.

In a merchant situation where a lot of addresses and transactions occur on a daily basis (e.g naivly 100 addresses for 100 transactions a day), does it make sense to "load balance" payments using multiple wallets (e.g naivly 1 wallet for 10 transactions a day = 10 instances of wallets for 100 tansactions a day) ? Does anybody experienced such architecture ?

Thanks

no just run a full node

Yes I already took a (short) look at Bitpay, as well as BTCPay server which is API compliant, as a possible alternative. However at this point I actually need to avoid centralized services as well as third-party payment providers. This is a technical/business requirement.

Thanks again for your answers and point of view!

btcpayserver is not centralized. it needs bitcoin core running. with bitcoin core i.e. a full node you have the blockchain on disk so you can handle very large wallets.

[ayadev]

no just run a full node

Like I said, currently I can't build a payment process on top of a full node (and going live with it)... i need more experience in Bitcoin/Blockchain architecture to achieve this efficiently.

Although Electrum seems to have issues with large wallets, its interface comply wth most of my actual business requirements.

Except large wallets, could you tell me what, in your opinion, what could possibly go wrong with Electrum ?

btcpayserver is not centralized. it needs bitcoin core running. with bitcoin core i.e. a full node you have the blockchain on disk so you can handle very large wallets.

Yes BTCPayserver is obviously not centralized. I was basically mentioning that BTCPay is compatible with Bitpay API. Sorry for that confusion.

I tried to run a self-hosted BTCPay instance some weeks ago to test. I gave up because there was a lot of dependencies, options and a complexity which is not required in my case.

Thanks for your answer

[HCP]

Honestly, the best solution is a full node... attempting to "workaround" Electrum's limitations with regards to large volumes of transactions/addresses etc is only going to end in tears

You'll either create a solution that doesn't scale well or becomes a nightmare to manage.

Note that the limitations of "large volume" aren't necessarily a "client" issue, so much as a "server" bandwidth imposed limit. The most common server (ElectrumX) has some "default" limits that will be triggered by a client requesting large amounts of data (ie. when trying to sync a wallet with 1000s of addresses/transactions etc). When that happens, the client is throttled/timed out and it appears as though "syncing" is stuck or broken.

If you run your own Electrum server, these limits can easily be modified to workaround this limitation... However this leads to this:

As far as I know you can run your own Electrum server and make all your wallets connect to that particular server only. However, it could become a single point of failure. Of course you can also run your own Bitcoin node if this is a critical business requirement. But in the end (IMO) we always have to rely on a third-party provider at some point.

You'll find that all Electrum servers require access to a Bitcoin Node anyway... so we're back to the solution of: "just run a Bitcoin Node"


So... in summary, I can see 3 options:

1. Find a friendly Electrum Server admin who would be willing to setup a server for you with modified limits
or
2. Setup your own Bitcoin Node + Electrum Server setup with modified limits
or
3. Setup your own Bitcoin Node + BTCPayServer

Personally, I'd probably go with #3... it's the least "hacky" solution using tools designed specifically for the job you appear to be trying to accomplish.

[ayadev]

Honestly, the best solution is a full node... attempting to "workaround" Electrum's limitations with regards to large volumes of transactions/addresses etc is only going to end in tears

You'll either create a solution that doesn't scale well or becomes a nightmare to manage.

Note that the limitations of "large volume" aren't necessarily a "client" issue, so much as a "server" bandwidth imposed limit. The most common server (ElectrumX) has some "default" limits that will be triggered by a client requesting large amounts of data (ie. when trying to sync a wallet with 1000s of addresses/transactions etc). When that happens, the client is throttled/timed out and it appears as though "syncing" is stuck or broken.

If you run your own Electrum server, these limits can easily be modified to workaround this limitation... However this leads to this:

As far as I know you can run your own Electrum server and make all your wallets connect to that particular server only. However, it could become a single point of failure. Of course you can also run your own Bitcoin node if this is a critical business requirement. But in the end (IMO) we always have to rely on a third-party provider at some point.

You'll find that all Electrum servers require access to a Bitcoin Node anyway... so we're back to the solution of: "just run a Bitcoin Node"


So... in summary, I can see 3 options:

1. Find a friendly Electrum Server admin who would be willing to setup a server for you with modified limits
or
2. Setup your own Bitcoin Node + Electrum Server setup with modified limits
or
3. Setup your own Bitcoin Node + BTCPayServer

Personally, I'd probably go with #3... it's the least "hacky" solution using tools designed specifically for the job you appear to be trying to accomplish.

Thanks for your answer, especially the points you mentioned about Electrum server-side limitations.

You all agree on the fact that Electrum is not the best solution in that case so I'm going to stop swimming against the tide from now )

I'm gonna give BTCPayserver a second chance and alongside with that, trying to run a full node!

 

[Abdussamad]

running a full node is not some complex endeavor. you just run bitcoin core software and it does the sync automatically. it just takes time to download the blockchain. you can enable pruning to save disk space.