Bitcoin Forum
May 26, 2024, 03:28:17 AM *
News: Latest Bitcoin Core release: 27.0 [Torrent]
 
  Home Help Search Login Register More  
  Show Posts
Pages: [1]
1  Bitcoin / Electrum / Re: Electrum, high availability and horizontal scalability on: December 22, 2020, 12:35:09 AM
Honestly, the best solution is a full node... attempting to "workaround" Electrum's limitations with regards to large volumes of transactions/addresses etc is only going to end in tears Undecided

You'll either create a solution that doesn't scale well or becomes a nightmare to manage.

Note that the limitations of "large volume" aren't necessarily a "client" issue, so much as a "server" bandwidth imposed limit. The most common server (ElectrumX) has some "default" limits that will be triggered by a client requesting large amounts of data (ie. when trying to sync a wallet with 1000s of addresses/transactions etc). When that happens, the client is throttled/timed out and it appears as though "syncing" is stuck or broken.

If you run your own Electrum server, these limits can easily be modified to workaround this limitation... However this leads to this:
As far as I know you can run your own Electrum server and make all your wallets connect to that particular server only. However, it could become a single point of failure. Of course you can also run your own Bitcoin node if this is a critical business requirement. But in the end (IMO) we always have to rely on a third-party provider at some point.
You'll find that all Electrum servers require access to a Bitcoin Node anyway... so we're back to the solution of: "just run a Bitcoin Node" Tongue Wink


So... in summary, I can see 3 options:

1. Find a friendly Electrum Server admin who would be willing to setup a server for you with modified limits
or
2. Setup your own Bitcoin Node + Electrum Server setup with modified limits
or
3. Setup your own Bitcoin Node + BTCPayServer

Personally, I'd probably go with #3... it's the least "hacky" solution using tools designed specifically for the job you appear to be trying to accomplish.


Thanks for your answer, especially the points you mentioned about Electrum server-side limitations.

You all agree on the fact that Electrum is not the best solution in that case so I'm going to stop swimming against the tide from now )

I'm gonna give BTCPayserver a second chance and alongside with that, trying to run a full node!

 
2  Bitcoin / Electrum / Re: Electrum, high availability and horizontal scalability on: December 21, 2020, 06:10:04 PM
no just run a full node

Like I said, currently I can't build a payment process on top of a full node (and going live with it)... i need more experience in Bitcoin/Blockchain architecture to achieve this efficiently.

Although Electrum seems to have issues with large wallets, its interface comply wth most of my actual business requirements.

Except large wallets, could you tell me what, in your opinion, what could possibly go wrong with Electrum ?

btcpayserver is not centralized. it needs bitcoin core running. with bitcoin core i.e. a full node you have the blockchain on disk so you can handle very large wallets.

Yes BTCPayserver is obviously not centralized. I was basically mentioning that BTCPay is compatible with Bitpay API. Sorry for that confusion.

I tried to run a self-hosted BTCPay instance some weeks ago to test. I gave up because there was a lot of dependencies, options and a complexity which is not required in my case.

Thanks for your answer
3  Bitcoin / Electrum / Re: Electrum, high availability and horizontal scalability on: December 21, 2020, 01:02:39 AM
It's more of a hassle if the User A accidentally sends funds to the address after it's expiration but it has already been assigned to User B. Such a scenario can be tricky to handle.

That being said, I think you might be able to do with having addresses that doesn't have any transactions since those wouldn't incur any resource penalty when querying them with an Electrum server. I missed that.

In the other hand, if I run a new wallet instance every time "10" addresses have been assigned to a payment request (funded or not), it could become a substantial waste of resource on my side...not to mention managing hundreds or thousands of wallets is probably a funny nightmare.

Another solution would be to not reuse an address for a "very raisonnable" period of time in order to mitigate this risk.

I wouldn't say Electrum is the best candidate because it isn't designed for heavy workload in the first place and having to rely on someone else's Bitcoin node to tell you the correct information doesn't sit too well with me.

I'm actually trying to estimate that design weakness and finding a creative way to bypass it... even though I don't really know where I'm going at right now but it's getting better)

As far as I know you can run your own Electrum server and make all your wallets connect to that particular server only. However, it could become a single point of failure. Of course you can also run your own Bitcoin node if this is a critical business requirement. But in the end (IMO) we always have to rely on a third-party provider at some point.

You do have a point with the issue about xpub, since Bitcoin Core won't derive any xpub. IMO, it would be complicated to manage and swap wallets every now and then. If that's something that you're able to accommodate then I don't think there's too much of a problem.

I guess this architecture could survive as long as it follows very simple rules and avoid complexity as much as possible (I mean, in terms of wallet management/swap policy)

If you'd like, there are services like Bitpay which helps to streamline the process, at the expense of it being a centralised service.

Yes I already took a (short) look at Bitpay, as well as BTCPay server which is API compliant, as a possible alternative. However at this point I actually need to avoid centralized services as well as third-party payment providers. This is a technical/business requirement.

Thanks again for your answers and point of view!
4  Bitcoin / Electrum / Re: Electrum, high availability and horizontal scalability on: December 20, 2020, 05:17:28 PM
The problem comes from the fact that Electrum writes the wallet file in simple JSON format which it looks up later so a better approach would be to use everything in Electrum except parts of the wallet layer.
It would require more work but theoretically with some modification to the code you could replace the wallet file (the way it is written to disk and read) with a database instead so that looking up inside millions of inputs becomes super fast and use that with the rest of the Electrum code for signing, downloading txs and blocks from nodes, ...

That's an interesting point of view. I guess JSON is a core design choice made by the developer(s) to avoid dependencies and because they assume that most users won't have to deal with a large wallet. However a database backend would be a merchant requirement for sure. It should be studied more closely.
 
5  Bitcoin / Electrum / Re: Electrum, high availability and horizontal scalability on: December 20, 2020, 04:50:04 PM
I imagine that it'll be a hassle to keep track of the wallets. You'll probably have to change out for a new wallet everyday. Gap limit is not a limitation, it's just how far your wallet will read ahead so no issues there.

Yes I was wrong, gab limit is not a limitation nor an issue (except when you have to restore a wallet using a seed if I understood well). I assumed gab limit could be used to delimit how many address should be used per wallet instance, but this is not relevant at all. I still need to do some homework)

You would have to account for those who would generate an address for payment but not pay in the end as well.
In my situation I guess I can mitigate this by expiration times and limitating how many pending requests a user can generate at the same time.

On architecture side, one solution would be to look for expired requests and reuse addresses, assuming that reusing an address which has never been funded is an acceptable privacy risk. But again, I may be missing something.


It's more of the few developers-centric RPCs (getrawmempool,etc) which wouldn't really affect normal users.

I didn't know about solutions such as getrawmempool thanks!

I'm mainly using Electrum because I guess this wallet provides a "lightweight" additionnal layer as well as some valuable features (secret seed, client-side stored private keys, socks5 proxy support, etc.). At this point, I don't have enough knowledge nor experience to deal with a full node directly.
6  Bitcoin / Electrum / Re: Electrum, high availability and horizontal scalability on: December 19, 2020, 06:19:50 PM
Hi @ranochigo,

Thanks for your answer.

Maybe I should have started by looking at Bitcoin Core to have a better overview. I will take a look at it asap.

To be more precise, the idea behind this would be to automatically create a new Electrum wallet and run it as a read-only wallet (in a new containerized instance) each time a limitation is reached (e.g gap limit), avoiding as much as possible the limitations you mentioned. This is theory.

Could you give me some details about what is missing (in your opinion) in Electrum RPC interface ?



  




  
7  Bitcoin / Electrum / Electrum, high availability and horizontal scalability on: December 19, 2020, 05:33:40 PM
Hey,

I've been reading some topics on the message board stating that Electrum wasn't design to handle wallets with a large addresses/transactions history, although there are no hard-coded limitations.

In a merchant situation where a lot of addresses and transactions occur on a daily basis (e.g naivly 100 addresses for 100 transactions a day), does it make sense to "load balance" payments using multiple wallets (e.g naivly 1 wallet for 10 transactions a day = 10 instances of wallets for 100 tansactions a day) ? Does anybody experienced such architecture ?

Thanks
8  Bitcoin / Electrum / Re: Electrum to notify on address status change (CLI) on: December 19, 2020, 04:47:50 PM
Hey,

I guess I started to read Electrum Protocol section but content was marked as outdated (redirecting to ElectrumX doc) so I may have missed it. Anyways.

Thank you so much for your help !
9  Bitcoin / Electrum / Electrum to notify on address status change (CLI) on: December 19, 2020, 03:52:25 PM
Hello,

As a developer, I've been testing Electrum command line interface (as well as JSON RPC) for some days to receive payments through my web apps.

I've been trying to use notify (address, [url]) command to get notified every time a wallet's address status changes.

It works quite well, I receive JSON responses but I don't know how to process the status string, which appears to be encoded. I don't know whether this is a binary or encrypted string actually.

Typical JSON response:

Code:
{"address": "tb1poly23484gvhb9atztrvaydzh0t9hcadncrmba1", "status": "34a4e9286fa705980e42edbc95db515593d46f30164f8d45d4b2387e0541b211"}

I may be missing something obvious but I haven't found specific documentation on this.

I tested this with Electrum 4.0.7 and a read-only Testnet wallet.

Any further information ?

Thanks





Pages: [1]
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!