hd-wallet-addrs: a new tool for HD Wallet address discovery. Please test!

[danda]

hd-wallet-addrs is a command-line tool that performs Bip32 wallet address discovery.  It finds all addresses in an HD-wallet that have ever been used (received funds).  There is also a simple web frontend available.

I appreciate any testing, feedback, suggestions you can provide.  So far I have tested with Copay and mycelium android.  Reports with other wallets would be great!

GitHub: https://github.com/dan-da/hd-wallet-addrs
Web Frontend: https://mybitprices.info/hd-wallet-addrs.html

I initially wrote this utility to find wallet addresses for an auditing tool I was working on.  But I realized it could be useful for other people and purposes as well.

Many (most?) HD Wallets do not provide a complete list of addresses that have been used, but do provide the master XPub key. So this tool empowers users by providing information kept hidden by their own wallet software.

This can be useful for bitcoin developers or anyone wishing to audit their wallet software in some fashion.  Or just for record keeping purposes.

Both regular HD wallets (single address) and multi-sig wallets (eg Copay) are supported.  

Reports are available in json, plaintext, and html. Columns can be changed or re-ordered via command-line.

Here's an example:

Code:

$ ./hd-wallet-addrs.php -g --xpub=xpub6BfKpqjTwvH21wJGWEfxLppb8sU7C6FJge2kWb9315oP4ZVqCXG29cdUtkyu7YQhHyfA5nt63nzcNZHYmqXYHDxYo8mm1Xq1dAC7YtodwUR --logfile=/tmp/log.txt

 --- Wallet Discovery Report --- 


Found 3 Receive addresses and 2 Change addresses.
  Receive --  Used: 3   Unused: 0
  Change  --  Used: 2   Unused: 0

+------------------------------------+---------+----------------+------------+------------+---------+
| addr                               | type    | total_received | total_sent | balance    | relpath |
+------------------------------------+---------+----------------+------------+------------+---------+
| 1Ge6rDuyCdYVGhXZjcK4251q67GXMKx6xK | Receive |     0.00120000 | 0.00100000 | 0.00020000 | 0/0     |
| 1NVsB73WmDGXSxv77sh9PZENH2x3RRnkDY | Receive |     0.00130000 | 0.00100000 | 0.00030000 | 0/1     |
| 1BkgqiHcvfnQ2wrPN5D2ycrvZas3nibMjC | Receive |     0.00040000 | 0.00000000 | 0.00040000 | 0/2     |
| 12SisoiXLUEbkytL5Pzia1jBY8gJP5XN8D | Change  |     0.00184874 | 0.00000000 | 0.00184874 | 1/0     |
| 1CkvACVpFwkPnMG13w9kXXE9YcsiyL4pcY | Change  |     0.00194876 | 0.00000000 | 0.00194876 | 1/1     |
+------------------------------------+---------+----------------+------------+------------+---------+

or to generate a simple list for copy/paste:

Code:

$ ./hd-wallet-addrs.php -g --format=addrlist --xpub=xpub6BfKpqjTwvH21wJGWEfxLppb8sU7C6FJge2kWb9315oP4ZVqCXG29cdUtkyu7YQhHyfA5nt63nzcNZHYmqXYHDxYo8mm1Xq1dAC7YtodwUR --logfile=/tmp/log.txt

 --- Wallet Discovery Report --- 

Found 3 Receive addresses and 2 Change addresses.
  Receive --  Used: 3   Unused: 0
  Change  --  Used: 2   Unused: 0

1Ge6rDuyCdYVGhXZjcK4251q67GXMKx6xK
1NVsB73WmDGXSxv77sh9PZENH2x3RRnkDY
1BkgqiHcvfnQ2wrPN5D2ycrvZas3nibMjC
12SisoiXLUEbkytL5Pzia1jBY8gJP5XN8D
1CkvACVpFwkPnMG13w9kXXE9YcsiyL4pcY

I plan to add a couple simple features in the next few days:

 + a flag to find all addresses up to <limit>, regardless if used or not.  ( no need to query blockchain for this. )
 + a flag to find only change addresses or only receive addresses.
 + a flag to enable round-robin of blockchain API providers for improved privacy.

[btc_enigma]

Nice tool. Good work !

Does it send multiple address at once to server ? I know it will be slow, but we can have option to send only one address at a time to random API provider.

[siameze]

Looks like a very nifty tool, and I love useful cli items for my toolbox. added! 

[danda]

Interesting idea.  I like it.

Presently it can make use of 3 blockchain providers:  insight, toshi, and blockchain.info.

Insight and toshi accept only one address at a time.  blockchain.info accepts multiple.   So blockchain.info is the default.

The application code does the queries in batches, and the insight and toshi classes internally serialize the batches into single requests.

I just noticed that blockonomics also accepts multiple and does not require an API key.  I'll see about adding support for it.

insight and toshi can be run locally, and that removes any privacy risk.   But of course it is a giant hassle so most people will not do that.

Therefore, I like your idea as a way to regain some privacy, just at the expense of speed.  a convenient middle ground.

I will add it to the TODO list.

Probably I will do something like "--round-robin=yes --batch-size=2"

This way, the user can tell the app to use round robin, which will default to a batch size of 1, but user can still override batch-size for a performance boost.

Does it send multiple address at once to server ? I know it will be slow, but we can have option to send only one address at a time to random API provider.

[btc_enigma]

Probably I will do something like "--round-robin=yes --batch-size=2"

Yes, that sounds awesome ! Thanks . Let me know after the changes, would try to test your script

[danda]

@btc_enigma -- Are you associated with blockonomics?

I ask because I started working on integrating blockonomics API and ran into a roadblock. 

It appears there is not a suitable API available after all.   The /api/balance method is close because it accepts multiple addresses and returns a balance.  However, the balance can be 0 if an address has received funds and then sent them all, so what I actually require is total_received.

Additionally for full integration, I really need total_sent.  The other API's I'm using from toshi, insight, and blockchain.info do provide this info, and I make total_received and total_sent available as columns in the reports.

Any chance this data could be added to the balance API results, or add a new API such as /api/addresses?

thanks.

Probably I will do something like "--round-robin=yes --batch-size=2"

Yes, that sounds awesome ! Thanks . Let me know after the changes, would try to test your script

[danda]

This release adds the following features:

• added --api=roundrobin option which will cycle through blockchain providers one address at a time to improve privacy.
• added --batch-size option for direct control over how large each address batch is.
• added --gen-only option to generate addresses without checking the blockchain if they have been used.
• added --type option to control which type of addresses are generated: receive, change, or both.

https://github.com/dan-da/hd-wallet-addrs/releases/tag/hd-wallet-addrs-v0.1.2

[siameze]

This release adds the following features:

• added --api=roundrobin option which will cycle through blockchain providers one address at a time to improve privacy.
• added --batch-size option for direct control over how large each address batch is.
• added --gen-only option to generate addresses without checking the blockchain if they have been used.
• added --type option to control which type of addresses are generated: receive, change, or both.

https://github.com/dan-da/hd-wallet-addrs/releases/tag/hd-wallet-addrs-v0.1.2

Thanks for the update. Will test later this evening and report back. Nice work!

[danda]

cool, let me know how it goes.

Thanks for the update. Will test later this evening and report back. Nice work!

[btc_enigma]

@btc_enigma -- Are you associated with blockonomics?

I ask because I started working on integrating blockonomics API and ran into a roadblock. 

It appears there is not a suitable API available after all.   The /api/balance method is close because it accepts multiple addresses and returns a balance.  However, the balance can be 0 if an address has received funds and then sent them all, so what I actually require is total_received.

Additionally for full integration, I really need total_sent.  The other API's I'm using from toshi, insight, and blockchain.info do provide this info, and I make total_received and total_sent available as columns in the reports.

Any chance this data could be added to the balance API results, or add a new API such as /api/addresses?

thanks.

Probably I will do something like "--round-robin=yes --batch-size=2"

Yes, that sounds awesome ! Thanks . Let me know after the changes, would try to test your script

Got it  ! Yes, the API blockonomics is offering doesn't look suitable. Can you mail this to webmaster@blockonomics.co , they would be happy to help

[danda]

email sent.

[Lorenzo0]

Very interesting tool, thank you OP.

I'm left curious about the details of the auditing tool you mention.
Feel free to share any details about that, if it's ready.

[danda]

I just announced it.  Let me know your thoughts...

I'm left curious about the details of the auditing tool you mention.
Feel free to share any details about that, if it's ready.

[danda]

I just made a minor release to the CLI tool and also the web tool.

Web Changes:

Added an option "Generate Only" to generate a fixed (user-defined) number of addresses without checking if each is used or not.  This speeds up processing and is useful for pre-generating addresses and other use cases.

CLI Changes:

This version adds support for blockr.io as a backend API provider and updates documentation.

https://github.com/dan-da/hd-wallet-addrs/releases/tag/hd-wallet-addrs-v0.1.3

64c2549 bump version to hd-wallet-addrs-v0.1.3
7f22536 Add legacy Copay gap limit warning
7e35d87 update README with blockr.io info
3eec473 add a blockr test case
6c8e2e9 added support for blockr.io API
d43d7b8 document --api=roundrobin feature
141c8fc update README with latest usage

[danda]

update: I've made a couple minor releases since the last time I posted here.  Present version is 0.1.5.

Changes since 0.1.3:

• Adds support for btcd as an API provider for finding used wallet addresses.  So one need not rely on a 3rd party API provider to find all wallet addresses that have ever been used.
  note: btcd does not provide received/sent/balance info for each address, so those fields are empty.
• removes toshi from roundrobin list because toshi.io is no longer available.
• now runs under php7 (as well as php5.5+)
• Fixed bitwasp dependency version problem with php7.

[danda]

The major new feature in 0.1.8 is support for segwit addresses (p2sh and bech32) via ypub and zpub extended keys used by some 3rd party wallets.

Also added/changed since 0.1.5:

• blockcypher and btc.com can be used as api providers for funds lookup queries.
• blockr.io api provider removed -- no longer exists.
• auto-retry api providers 3 times in case of connection glitch
• minor fixes and improvements

[danda]

No new features.  It now runs with php8+.

https://github.com/dan-da/hd-wallet-addrs/releases/tag/v0.3.0