My electrum wallet contains over 17000 addresses (& growing) - am I in trouble?

[arsenische]

I created my Electrum wallet in 2013 and I liked how it worked. But now it contains over 17000 addresses and takes a lot of time to synchronize. Even if it is synchronized, it becomes sluggish after I send a transaction.

Can it handle the ever-increasing amount of bitcoin addresses?

Public Electrum servers have limits of 10000 addresses, does it mean I should create a new wallet each time my old one approaches this limit?

Thanks!

[Quickseller]

Electrum is theoretically able to handle an unlimited number of addresses. If you have 17,000 addresses in your wallet and your computer/electrum is running slow then you can create a new electrum wallet, send a few small transactions to addresses associated with that wallet and then try spending some of those transactions. If you can do all this without it being sluggish, then the root cause likely has something to do with having such a large number of addresses.

If you do create a new wallet then you should still keep the old wallet seed (and backup file) in case there is some reason you need access to one of the private keys (for example to sign a message and/or if someone sends bitcoin to an address in your old wallet.

[arsenische]

Thanks for your answer!

Actually I do use several Electrum wallets.

A small one with 421 addresses takes less than a minute to load, connect and synchronize. When I start it with -v option (verbose) then it displays a bunch of WalletSynchronizer messages in console without any any errors.

But for the big wallet it writes "SSLError: The write operation timed out" in console all the time and I am spending half a day to get it into synchronized state (restarting it time to time if it doesn't output anything meaningful a while).

Connections settings are the same.

After I broadcast a transaction I can see that Electrum fetches the history for each of its inputs. It takes at least 5 seconds and a bunch of SSLErrors to fetch the history for 1 input. And my transactions from the big wallet often contain hundreds of inputs, so it takes a while.

[Abdussamad]

This has come up before. Electrum isn't designed to handle so many addresses. You should create a new wallet (file > new/restore) and send your bitcoins to an address in it. You can leave the old wallet file on your hard drive in case someone sends bitcoins to an address in it in the future.

To open a specific wallet file use file> open or the command line switch -w.

[ThomasV]

many Electrum servers have a DoS-protection limit of 10000 addresses.

I suggest you archive your wallet, and create a new one.

If you cannot find any server and have issues sending your funds to the new wallet, you may import only the funded addresses in a temporary wallet.

Here is how you can see the list of funded addresses, in version 2.3:

Code:

./electrum listaddresses --funded

same list with balance:

Code:

./electrum listaddresses --funded --balance

then get the private keys with that command:

Code:

./electrum getprivatekeys <address>

[arsenische]

Thanks for your answers, very helpful!

Not sure what are the bottlenecks there. Do you think Electrum could support unlimited amount of bitcoin addresses without performance issues (assuming that only a few thousands of the addresses are being used at a time)?

E. g. would it be technically feasible to:
1) automatically delete information about the used addresses with 0 balance that haven't been used for a while
2) partially recover this information when those addresses receive new funds
?

[arsenische]

Satoshi quoted once:  

Oct. 3, 2010: Sigh… why delete a wallet instead of moving it aside and keeping the old copy just in case? You should never delete a wallet.

I agree with Satoshi on this, but please keep in mind that Electrum wallet is deterministic. I don't suggest to delete the wallet, I suggest to delete (or just to ignore) the information that can be easily recovered from the seed and the blockchain if needed.

[arsenische]

The problem with my wallet is that any of its 17K+ addresses may receive money but it is already too large to get synchronized in a timely fashion.

I wish Electrum had parameters "start key index" and "stop key index" that could be used to create multiple wallets from the same seed with different sets of the keys, e. g. wallet1.dat for keys 0..9999, wallet2.dat for keys 10000..19999, etc. Then each individual wallet wouldn't be too large and I could synchronize them time to time separately

[arsenische]

I tried to synchronize my wallet for 2 days but couldn't do it. It looked like public Electrum servers refused to serve me due to the huge amount of bitcoin addresses I have. Here is a dirty hack I came up with to overcome this issue:

In /usr/local/lib/python2.7/dist-packages/electrum/synchronizer.py

I changed  

Code:

        self.subscribe_to_addresses(set(self.wallet.addresses(True)))

to

Code:

        self.subscribe_to_addresses(set(self.wallet.addresses(True)[self.wallet.storage.get('start_index'):self.wallet.storage.get('stop_index')]))

What it does is it allows the wallet to synchronize partially from start_index to stop_index (excluding stop_index). If start_index is not set then it will start from 0. If stop_index is not set then it will go up to the end. That's what I thought, but when I started Electrum with start_index=0 and stop_index=100, for some reason it didn't stop at 100 and synchronized it up to the end. Perhaps some other logic comes into play there, but at least Electrum servers didn't reject me and my wallet got synchronized from the first attempt! I hope I didn't break anything.


PS: I used Electrum's console to specify start_index and stop_index parameters, e. g.: wallet.storage.put('start_index', 10000) sets start_index to 10000 and wallet.storage.put('start_index', None) resets it.

[arsenische]

Despite the fact that the "hacked" version of Electrum displays the balance as if it were synchronized, unfortunately this hack didn't solve my problem.

Perhaps during synchronization phase Electrum removes unnecessary information from the wallet so that it is not possible to synchronize it partially.

When I try to spend funds, it returns error 25. When I start the wallet with "-v" option, I see a bunch of errors "'No information available about transaction', u'code': -5}" from the WalletSynchronizer and sometimes "Error: history not synchronized" or "KeyError".

So I probably need to either find some non-deterministic SPV wallet and export the keys there, or look deeper into Electrum's sources and understand how it works before making modifications. Or maybe just restore the wallet from the seed and try to synchronize from scratch? It is going to take days...

[ThomasV]

are most of your addresses used and empty?
I think an option to not subscribe to used addresses would make sense.

[arsenische]

are most of your addresses used and empty?
I think an option to not subscribe to used addresses would make sense.

yes, they are, but any of them may receive transactions in future.

Eventually I managed to send out the money from the old wallet.

I tried to partially synchronize the wallet form 0 to 10000, then from 10000 to 11000, from 11000 to 12000, etc, and noticed that somewhere between 16000 and 16500 addresses an exception happened:

Code:

Notifying GUI
Exception in thread Thread-12:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/electrum/synchronizer.py", line 53, in run
    self.run_interface()
  File "/usr/local/lib/python2.7/dist-packages/electrum/synchronizer.py", line 160, in run_interface
    self.wallet.receive_history_callback(addr, hist)
  File "/usr/local/lib/python2.7/dist-packages/electrum/wallet.py", line 755, in receive_history_callback
    self.remove_transaction(tx_hash, height)
  File "/usr/local/lib/python2.7/dist-packages/electrum/wallet.py", line 739, in remove_transaction
    self.txi.pop(tx_hash)

(the line numbers might not match yours since I was experimenting with the sources locally)

So I added a "return" to the beginning of the remove_transaction function in wallet.py (so that this function doesn't do anything and doesn't raise an exception) and successfully synchronized the wallet and sent out funds from it. I think this problem is solved for me until I upgrade to the next version of Electrum