What's the best way to watch the balance of hundreds of thousands of addresses?

[arsenische]

I have an ever-growing list of cold wallets with thousands of bitcoin addresses in each of them. Most of them are probably empty and won't ever be used, but still I'd like to check the sum of bitcoins they hold. What's the best (cheapest and easiest) way of doing it?

My first thought was to import them as watch-only addresses to the bitcoin core instance that is running on my server. But I am afraid it may affect the server performance and I don't want to risk it.

So I should probably set up a separate server for this purpose and download the blockchain again.

Or is there a way to check the total balance of hundreds of thousands of bitcoin addresses with some free online service or a light wallet?

[BurtW]

So you are creating cold wallets with the hope of someday stealing someone else's coins?  

All you need is a way to monitor all the addresses you are creating in order to swipe the coins if there is ever an address collision?

That is what it sounds like.

[arsenische]

So you are creating cold wallets with the hope of someday stealing someone else's coins? 

All you need is a way to monitor all the addresses you are creating in order to swipe the coins if there is ever an address collision?

That is what it sounds like.

No, I think that's a pretty common use case (and the recommended practice) for bitcoin businesses: to generate a new bitcoin address each time your customer wants to deposit something.

But if you do so, you end up with an ever-growing amount of bitcoin addresses that might be a problem for bitcoin wallet software (e. g. Electrum becomes pretty much unusable when you have over 10K addresses in your wallet).

So you create a new wallet every 10K addresses to avoid this problem and thus end up with many wallets. Most of the customers usually deposit to the most recent wallet, but there is still a chance that somebody deposits to the old one. Of course your server-side software listens to transactions and processes the deposits made to the old wallets too.

But sometimes you need to audit your bitcoin holdings in order to make sure that everything is all right.

I currently have only 1 idea how to accomplish this: to set up a new bitcoin core instance, import the bitcoin addresses there and let it synchronize. Perhaps it is not a problem at all, but it requires to rent a server and to download the whole block chain, moreover I am not sure how many addresses bitcoin core instance can handle on practice (obviously more than Electrum).

I wonder if there is an easer way to determine the exact amount of bitcoins in all those old wallets.

[shorena]

I just set keypoolrefill to 100000 on a core testnet wallet. When the log showed ~20k addresses generated the UI stopped working.

Code:

2015-12-20 22:37:42 Pre-allocating up to position 0x8000000 in blk00022.dat
2015-12-20 22:37:43 UpdateTip: new best=0000000000000020542d296e1cf56f991f838814583b8ba1473687c2cc2f0d67  height=605658  log2_work=66.751446  tx=7191458  date=2015-11-17 01:19:20 progress=0.998588  cache=5.8MiB(21617tx)
2015-12-20 22:37:43 keypool added key 118, size=101
2015-12-20 22:37:43 keypool added key 119, size=102
2015-12-20 22:37:43 keypool added key 120, size=103
2015-12-20 22:37:43 keypool added key 121, size=104
2015-12-20 22:37:43 keypool added key 122, size=105
2015-12-20 22:37:43 keypool added key 123, size=106
...
2015-12-20 22:48:48 keypool added key 100017, size=100000
2015-12-20 22:48:48 keypool added key 100018, size=100001
2015-12-20 22:48:48 keypool reserve 18
2015-12-20 22:48:48 keypool return 18

UI is working fine now. Wallet.dat is 64552 KB in size. Its still synchronizing, as I didnt use the testnet wallet for a while. Start up is slightly slower than my regular wallet with < 300 keys.

Code:

{
"walletversion" : 60000,
"balance" : 0.00000000,
"unconfirmed_balance" : 0.00000000,
"immature_balance" : 0.00000000,
"txcount" : 36,
"keypoololdest" : 1447094366,
"keypoolsize" : 100001
}

Let me test one more thing here. A wallet file that is >421000 kb in size. It contains ~100 private keys that were disclosed during one of the earlier spam attacks on the network.

Code:

2015-12-20 23:08:59 init message: Loading wallet...
2015-12-20 23:09:05 nFileVersion = 110000
2015-12-20 23:09:05 Keys: 203 plaintext, 0 encrypted, 203 w/ metadata, 203 total
2015-12-20 23:09:05  wallet                 5991ms
2015-12-20 23:09:05 init message: Rescanning...
2015-12-20 23:09:05 Rescanning last 22988 blocks (from block 366433)...
2015-12-20 23:10:05 Still rescanning. At block 371174. Progress=0.878292
2015-12-20 23:11:05 Still rescanning. At block 376108. Progress=0.915589
2015-12-20 23:12:05 Still rescanning. At block 381165. Progress=0.951302
2015-12-20 23:13:05 Still rescanning. At block 385897. Progress=0.980783
2015-12-20 23:13:54  rescan               289616ms
2015-12-20 23:13:54 init message: Activating best chain...

Once loaded the UI is unusable. It constantly uses 100% of one of the cores and eats >1.2 GB memory (compared to regular ~200-300MB). Last log entry was 8 minutes ago. I have to assume it crashed now.

To sum it up. A high load of transactions puts more straing on core than a high load of addresses. I would assume this is the same for watch only wallets.

[arsenische]

Thanks for sharing!

The ever-increasing amount of bitcoin addresses imply an ever-increasing amount of transactions too.

So it could be a problem even for reference client.

I wonder how block explorers (such as blockexplorer.com, blocktrail.com, blockchain.info etc) watch all the bitcoin addresses, how much resources they take.

[Tavos]

You could try using blockchain.info's web wallet feature that lets you add a watch only address.

[arsenische]

You could try using blockchain.info's web wallet feature that lets you add a watch only address.

AFAIK they have a limitation of only 1000 addresses per wallet. Or does this limit not apply to watch only addresses? If so then do they have an API to import watch-only addresses?

[Dabs]

You'd need a custom solution. I think there was a program some time ago that just looked up all the top thousand addresses by going through the blockchain.

One way to do this is to brute force searching the blockchain by keeping it ALL in memory, as doing it on a HDD or even SSD would be too slow. There are used servers that have 100+ GBs of RAM you can buy on ebay. If you are a business, then you must have server hardware. And then there is the rent-a-cloud solution.

I'm pretty sure there is a more elegant way though.

[shorena]

You'd need a custom solution. I think there was a program some time ago that just looked up all the top thousand addresses by going through the blockchain.

One way to do this is to brute force searching the blockchain by keeping it ALL in memory, as doing it on a HDD or even SSD would be too slow. There are used servers that have 100+ GBs of RAM you can buy on ebay. If you are a business, then you must have server hardware. And then there is the rent-a-cloud solution.

I'm pretty sure there is a more elegant way though.

A blockchain explorer of your own, like ABE[1] could be a solution. Alternativly if time is not an issue you could check them through an API. If you have 100k addresses and assume 2 requests per second a cycle would take ~14 hours. Should be a reasonable enough response time. This is easily within free limits of e.g. blocktrail[2] and would only need a few lines of code.

[1] https://en.bitcoin.it/wiki/Abe
[2] https://www.blocktrail.com/api/docs#rate_limits

[morantis]

Use an API through a third party.  There is no easy way to do what you are doing other than using a third party or writing the code yourself.

[arsenische]

Thanks for your responses! I had these ideas in mind, but hoped that there is something better

Abe could be an option, but I didn't find hardware requirements for it, moreover it might be incompatible with the newer node software:

Abe works with files created by the original (Satoshi) Bitcoin client. You will need a copy of the block files (blk0001.dat, blk0002.dat, etc. in your Bitcoin directory or its blocks/ subdirectory). You may let Abe read the block files while Bitcoin runs, assuming Bitcoin only appends to the file. Prior to Bitcoin v0.8, this assumption seemed safe. Abe may need some fixes to avoid skipping blocks while current and future Bitcoin versions run.

It seems like the Blocktrail API would be a good-enough solution for my purposes.