Electrum createnewaddress gap limit question (receiving lots of payments)

[bere3kin]

Hi again folks!
Tried to search the answer on forum, but not really clear about it. To clarify I trying to make a receiving addresses using createnewaddress and I need a lot of it with receiving callback for each (as I suppose by using notify method). I want to know, how much addresses can be generated - 20, 100 or unlimited, that can be safety used i.e. receive payments, get notifications and spend it? Documentation says that method "Create a new receiving address, beyond the gap limit of the wallet". I know what does it mean, but is it ok to use such address (that beyond gap) for receiving payment in current version of wallet? Maybe you can advise already ready-made solution to receive payments for large client base using Elecreum.

[1715017942]

1715017942

[jackg]

I not sure if there is a limit somewhere,

You might do what I did and run out of space on a drive you’re testing it with before you go past the gap limit (it was either 1000000 or 1000000000 and I think the process ran out of ram )...

[Abdussamad]

electrum tends to slow down after a few thousand addresses so you'll have to rotate wallets every so often. also use addrequest instead with the --force switch.  see `electrum help addrequest` . you can specify any amount to addrequest. it doesn't have any effect on how much people can send you.

[HCP]

Documentation says that method "Create a new receiving address, beyond the gap limit of the wallet". I know what does it mean, but is it ok to use such address (that beyond gap) for receiving payment in current version of wallet?

The short answer is Yes.

The complete answer is: It should be OK to use such addresses to receive payment. Just with the caveat that if you ever need to restore the wallet from seed at some point in the future, your funds may not show up automagically when you first restore as, by default, Electrum only has a gap limit of 20 for receive addresses (and 5 for change addresses).

So, if you end up creating addresses out past the gap limit and they receive funds... but you never actually receive any coins to the addresses contained within the gap limit... if you restore the wallet, by default it'll generate the 20 empty addresses and then stop looking.

However, this doesn't mean the funds will be lost... It is easy enough to simply manually increase the gap limit and the wallet will then "find" the funds stored out past the default limits.

[jackg]

Documentation says that method "Create a new receiving address, beyond the gap limit of the wallet". I know what does it mean, but is it ok to use such address (that beyond gap) for receiving payment in current version of wallet?

The short answer is Yes.

The complete answer is: It should be OK to use such addresses to receive payment. Just with the caveat that if you ever need to restore the wallet from seed at some point in the future, your funds may not show up automagically when you first restore as, by default, Electrum only has a gap limit of 20 for receive addresses (and 5 for change addresses).

So, if you end up creating addresses out past the gap limit and they receive funds... but you never actually receive any coins to the addresses contained within the gap limit... if you restore the wallet, by default it'll generate the 20 empty addresses and then stop looking.

However, this doesn't mean the funds will be lost... It is easy enough to simply manually increase the gap limit and the wallet will then "find" the funds stored out past the default limits.

I’d say it’s probably best to backup an encrypted wallet file...

It then also means that when you need to resend your wallets then you should be able to restart your payment processor without much of a delay in resyncing, changing gap limits etc...

[bere3kin]

Documentation says that method "Create a new receiving address, beyond the gap limit of the wallet". I know what does it mean, but is it ok to use such address (that beyond gap) for receiving payment in current version of wallet?

The short answer is Yes.

The complete answer is: It should be OK to use such addresses to receive payment. Just with the caveat that if you ever need to restore the wallet from seed at some point in the future, your funds may not show up automagically when you first restore as, by default, Electrum only has a gap limit of 20 for receive addresses (and 5 for change addresses).

So, if you end up creating addresses out past the gap limit and they receive funds... but you never actually receive any coins to the addresses contained within the gap limit... if you restore the wallet, by default it'll generate the 20 empty addresses and then stop looking.

However, this doesn't mean the funds will be lost... It is easy enough to simply manually increase the gap limit and the wallet will then "find" the funds stored out past the default limits.

I a little bit disappointed about it. It's almost 2018' ending, but still no any human-like merchant script that can cover all that problems? Is there really no simple logic to clean up own addresses for e.g.?

[jackg]

Documentation says that method "Create a new receiving address, beyond the gap limit of the wallet". I know what does it mean, but is it ok to use such address (that beyond gap) for receiving payment in current version of wallet?

The short answer is Yes.

The complete answer is: It should be OK to use such addresses to receive payment. Just with the caveat that if you ever need to restore the wallet from seed at some point in the future, your funds may not show up automagically when you first restore as, by default, Electrum only has a gap limit of 20 for receive addresses (and 5 for change addresses).

So, if you end up creating addresses out past the gap limit and they receive funds... but you never actually receive any coins to the addresses contained within the gap limit... if you restore the wallet, by default it'll generate the 20 empty addresses and then stop looking.

However, this doesn't mean the funds will be lost... It is easy enough to simply manually increase the gap limit and the wallet will then "find" the funds stored out past the default limits.

I a little bit disappointed about it. It's almost 2018' ending, but still no any human-like merchant script that can cover all that problems? Is there really no simple logic to clean up own addresses for e.g.?

It’s literally 2 lines in a console, I’d say it hardly requires a script. Sure, you can reduce it to 1 line but that’s not saving much.

[bob123]

I a little bit disappointed about it. It's almost 2018' ending, but still no any human-like merchant script that can cover all that problems? Is there really no simple logic to clean up own addresses for e.g.?

What kind of problems ? This isn't a problem at all.

That's a design-decision. It is a trade-off between convenience and efficiency. There is no reason to check thousands of addresses if that's unnecessary for 95% of all electrum user. That's just a slow down.

If one has created 1000 addresses which received transactions, he would simply need to generate 1000 addresses again after recovering to see the correct balance.
Private-/public- keypairs are generated successively, starting with the index 0. There is no risk in losing coins or anything.

[bere3kin]

I a little bit disappointed about it. It's almost 2018' ending, but still no any human-like merchant script that can cover all that problems? Is there really no simple logic to clean up own addresses for e.g.?

What kind of problems ? This isn't a problem at all.

That's a design-decision. It is a trade-off between convenience and efficiency. There is no reason to check thousands of addresses if that's unnecessary for 95% of all electrum user. That's just a slow down.

If one has created 1000 addresses which received transactions, he would simply need to generate 1000 addresses again after recovering to see the correct balance.
Private-/public- keypairs are generated successively, starting with the index 0. There is no risk in losing coins or anything.

Ok, I talking about the way to receive bitcoins from users with ability to notice server about payment through callback function. This simple scheme we can see into any big exchange services, like bitfinex and so on. Ppl just go to their deposit tab, see their deposit address and they can send bitcoins to it, at any time. And any other can do that. I'm not sure, that such services working this way. So maybe Electrum not suitable tool for developing it or I'm wrong?

electrum tends to slow down after a few thousand addresses so you'll have to rotate wallets every so often. also use addrequest instead with the --force switch.  see `electrum help addrequest` . you can specify any amount to addrequest. it doesn't have any effect on how much people can send you.

It's kinda deceision, but as for me, it's not cool. As I described, the purpose is give access to any user ability to deposit their own address.

[HCP]

It looks like you're wanting the convenience of an SPV wallet (like Electrum)... with the flexibility and power of a "Full Node" wallet (like Bitcoin Core).

I'd say that your options come down to:

1. Use a payment processor (cost = $)
2. Create your own payment processing system (cost = time and possibly also $ if you have to contract out)
3. Use an open source payment processing system (do these even exist? )

[bob123]

Ok, I talking about the way to receive bitcoins from users with ability to notice server about payment through callback function. This simple scheme we can see into any big exchange services, like bitfinex and so on.

Services which handle a big amount of deposits (e.g. exchange) shouldn't rely on a (standard) wallet at all.

IMO, the correct way for such a (big) business which receives multiple transactions per minute would be to build the whole system from scratch.
That's the only way one can somehow 'guarantee' that it works as it is supposed to.

For small online shops with a few payments per day, using core or electrum is perfectly fine. But with several hundred / thousand transaction per day, that would be negligent.
Using a software which has not been written explicitly for this one service means that there is noone responsible for the correctness of the code, eventual vulnerabilities, bugs, etc... An absolute no-go.

[pooya87]

Ok, I talking about the way to receive bitcoins from users with ability to notice server about payment through callback function. This simple scheme we can see into any big exchange services, like bitfinex and so on.

Services which handle a big amount of deposits (e.g. exchange) shouldn't rely on a (standard) wallet at all.

IMO, the correct way for such a (big) business which receives multiple transactions per minute would be to build the whole system from scratch.
That's the only way one can somehow 'guarantee' that it works as it is supposed to.

For small online shops with a few payments per day, using core or electrum is perfectly fine. But with several hundred / thousand transaction per day, that would be negligent.
Using a software which has not been written explicitly for this one service means that there is noone responsible for the correctness of the code, eventual vulnerabilities, bugs, etc... An absolute no-go.

although that sounds good in theory but it is not practical. making something from scratch is hard, time consuming and is susceptible to having a lot of bugs. you should never do that specially when everything is open source and in the past 10 years they have been tested and battle hardened.

what businesses should do and are doing is that they build on top of the already existing systems. for example as an exchange you build a system on top of a full verification node such as bitcoin core. that way you only have to focus on the payment processing part, securing your database and things like that instead of focusing on things that a node is responsible for.

[bob123]

although that sounds good in theory but it is not practical. making something from scratch is hard, time consuming and is susceptible to having a lot of bugs. you should never do that specially when everything is open source and in the past 10 years they have been tested and battle hardened.

what businesses should do and are doing is that they build on top of the already existing systems. for example as an exchange you build a system on top of a full verification node such as bitcoin core. that way you only have to focus on the payment processing part, securing your database and things like that instead of focusing on things that a node is responsible for.

I guess my phrase 'build the whole system from scratch' has been missunderstood.
I never intended to say that building a full node is included. I should have said it in a more clearer way.

What i wanted to say is that a big service shouldn't use open-source payment processor (e.g. btcpayserver) or rely on a software like electrum.

IMO, using core is perfectly fine (and required). But anything on-top should be self-coded (and audited of course). Companies need people responsible for the running software.
And when using other peoples software (which is always harder to read/understand than own code) the chances are higher that a few bugs/vulnerabilities stay unseen.


Even software running for multiple years can have severe vulnerabilities (e.g. electrum RPC vulnerability). Therefore it's quite important to build the majority yourself if you are heavily relying on it to work fully functional.

[jackg]

although that sounds good in theory but it is not practical. making something from scratch is hard, time consuming and is susceptible to having a lot of bugs. you should never do that specially when everything is open source and in the past 10 years they have been tested and battle hardened.

what businesses should do and are doing is that they build on top of the already existing systems. for example as an exchange you build a system on top of a full verification node such as bitcoin core. that way you only have to focus on the payment processing part, securing your database and things like that instead of focusing on things that a node is responsible for.

I guess my phrase 'build the whole system from scratch' has been missunderstood.
I never intended to say that building a full node is included. I should have said it in a more clearer way.

What i wanted to say is that a big service shouldn't use open-source payment processor (e.g. btcpayserver) or rely on a software like electrum.

IMO, using core is perfectly fine (and required). But anything on-top should be self-coded (and audited of course). Companies need people responsible for the running software.
And when using other peoples software (which is always harder to read/understand than own code) the chances are higher that a few bugs/vulnerabilities stay unseen.


Even software running for multiple years can have severe vulnerabilities (e.g. electrum RPC vulnerability). Therefore it's quite important to build the majority yourself if you are heavily relying on it to work fully functional.

There’s the added thing to that if the source code is published, a hacker trying to find the vulnerability has a similar copy to the software you have and if the vulnerability is obvious to them and has been cloned onto your service, you can say bye to the bitcoins you have ... (Or at the very least, a “hello” from all the angry emailers who want to know why they didn’t get something from your site).

[tarball]

3. Use an open source payment processing system (do these even exist? )

I might have misread, but isn't there 'electrum merchant' (https://github.com/spesmilo/electrum-merchant)?

[bere3kin]

3. Use an open source payment processing system (do these even exist? )

I might have misread, but isn't there 'electrum merchant' (https://github.com/spesmilo/electrum-merchant)?

It is, but once again, this includes concept of expired requests, that's not what we're looking for. It's easy but rough for clients. The idea to create payment processor for accepting payments at any time, to fixed address with ability notify server of confirmation. That's it.

2. Create your own payment processing system (cost = time and possibly also $ if you have to contract out)

Can you give an advice from where we should start? What is taken as a basis for developing it?

[bob123]

2. Create your own payment processing system (cost = time and possibly also $ if you have to contract out)

Can you give an advice from where we should start? What is taken as a basis for developing it?

You take core and build everything else on-top.

You can use whatever language you prefer. Core's functions walletnotify and blocknotify will help you to achieve what you are looking for.
They call a script specified by you after a transaction has been received (with the tx id as parameter).