Bitcoin Forum
September 24, 2018, 04:46:16 PM *
News: ♦♦ New info! Bitcoin Core users absolutely must upgrade to previously-announced 0.16.3 [Torrent]. All Bitcoin users should temporarily trust confirmations slightly less. More info.
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: hd-wallet-addrs: a new tool for HD Wallet address discovery. Please test!  (Read 1458 times)
danda
Full Member
***
Offline Offline

Activity: 184
Merit: 126


View Profile WWW
February 05, 2016, 04:36:55 PM
 #1

hd-wallet-addrs is a command-line tool that performs Bip32 wallet address discovery.  It finds all addresses in an HD-wallet that have ever been used (received funds).  There is also a simple web frontend available.

I appreciate any testing, feedback, suggestions you can provide.  So far I have tested with Copay and mycelium android.  Reports with other wallets would be great!

GitHub: https://github.com/dan-da/hd-wallet-addrs
Web Frontend: https://mybitprices.info/hd-wallet-addrs.html

I initially wrote this utility to find wallet addresses for an auditing tool I was working on.  But I realized it could be useful for other people and purposes as well.

Many (most?) HD Wallets do not provide a complete list of addresses that have been used, but do provide the master XPub key. So this tool empowers users by providing information kept hidden by their own wallet software.

This can be useful for bitcoin developers or anyone wishing to audit their wallet software in some fashion.  Or just for record keeping purposes.

Both regular HD wallets (single address) and multi-sig wallets (eg Copay) are supported.  

Reports are available in json, plaintext, and html. Columns can be changed or re-ordered via command-line.

Here's an example:

Code:
$ ./hd-wallet-addrs.php -g --xpub=xpub6BfKpqjTwvH21wJGWEfxLppb8sU7C6FJge2kWb9315oP4ZVqCXG29cdUtkyu7YQhHyfA5nt63nzcNZHYmqXYHDxYo8mm1Xq1dAC7YtodwUR --logfile=/tmp/log.txt

 --- Wallet Discovery Report ---


Found 3 Receive addresses and 2 Change addresses.
  Receive --  Used: 3   Unused: 0
  Change  --  Used: 2   Unused: 0

+------------------------------------+---------+----------------+------------+------------+---------+
| addr                               | type    | total_received | total_sent | balance    | relpath |
+------------------------------------+---------+----------------+------------+------------+---------+
| 1Ge6rDuyCdYVGhXZjcK4251q67GXMKx6xK | Receive |     0.00120000 | 0.00100000 | 0.00020000 | 0/0     |
| 1NVsB73WmDGXSxv77sh9PZENH2x3RRnkDY | Receive |     0.00130000 | 0.00100000 | 0.00030000 | 0/1     |
| 1BkgqiHcvfnQ2wrPN5D2ycrvZas3nibMjC | Receive |     0.00040000 | 0.00000000 | 0.00040000 | 0/2     |
| 12SisoiXLUEbkytL5Pzia1jBY8gJP5XN8D | Change  |     0.00184874 | 0.00000000 | 0.00184874 | 1/0     |
| 1CkvACVpFwkPnMG13w9kXXE9YcsiyL4pcY | Change  |     0.00194876 | 0.00000000 | 0.00194876 | 1/1     |
+------------------------------------+---------+----------------+------------+------------+---------+

or to generate a simple list for copy/paste:

Code:
$ ./hd-wallet-addrs.php -g --format=addrlist --xpub=xpub6BfKpqjTwvH21wJGWEfxLppb8sU7C6FJge2kWb9315oP4ZVqCXG29cdUtkyu7YQhHyfA5nt63nzcNZHYmqXYHDxYo8mm1Xq1dAC7YtodwUR --logfile=/tmp/log.txt

 --- Wallet Discovery Report ---

Found 3 Receive addresses and 2 Change addresses.
  Receive --  Used: 3   Unused: 0
  Change  --  Used: 2   Unused: 0

1Ge6rDuyCdYVGhXZjcK4251q67GXMKx6xK
1NVsB73WmDGXSxv77sh9PZENH2x3RRnkDY
1BkgqiHcvfnQ2wrPN5D2ycrvZas3nibMjC
12SisoiXLUEbkytL5Pzia1jBY8gJP5XN8D
1CkvACVpFwkPnMG13w9kXXE9YcsiyL4pcY


I plan to add a couple simple features in the next few days:

 + a flag to find all addresses up to <limit>, regardless if used or not.  ( no need to query blockchain for this. )
 + a flag to find only change addresses or only receive addresses.
 + a flag to enable round-robin of blockchain API providers for improved privacy.


mybitprices.info - wallet auditing tools
1537807576
Hero Member
*
Offline Offline

Posts: 1537807576

View Profile Personal Message (Offline)

Ignore
1537807576
Reply with quote  #2

1537807576
Report to moderator
1537807576
Hero Member
*
Offline Offline

Posts: 1537807576

View Profile Personal Message (Offline)

Ignore
1537807576
Reply with quote  #2

1537807576
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1537807576
Hero Member
*
Offline Offline

Posts: 1537807576

View Profile Personal Message (Offline)

Ignore
1537807576
Reply with quote  #2

1537807576
Report to moderator
1537807576
Hero Member
*
Offline Offline

Posts: 1537807576

View Profile Personal Message (Offline)

Ignore
1537807576
Reply with quote  #2

1537807576
Report to moderator
btc_enigma
Hero Member
*****
Offline Offline

Activity: 630
Merit: 501


View Profile
February 05, 2016, 05:06:03 PM
 #2

Nice tool. Good work !

Does it send multiple address at once to server ? I know it will be slow, but we can have option to send only one address at a time to random API provider.

siameze
Legendary
*
Offline Offline

Activity: 1064
Merit: 1000



View Profile
February 05, 2016, 05:13:50 PM
 #3

Looks like a very nifty tool, and I love useful cli items for my toolbox. added!  Cheesy


                     ▀▀█████████▀████████████████▄
                        ████▄      ▄████████████████
                     ▄██████▀  ▄  ███████████████████
                  ▄█████████▄████▄███████████████████
                ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀████████
                                               ▀▀███▀
    ▄█▀█       ▄▀  ▄▀▀█  ▄▀   █████████████████▄ ██▀         ▄▀█
   ▄█ ▄▀      ▀█▀ █▀ █▀ ▀█▀  ███████████████████ █▀ ▀▀      ▄▀▄▀
  ▄█    ▄███  █     █   █   ████████████████████  ▄█     ▄▀▀██▀ ▄███
███▄▄▄  █▄▄▄ █▄▄ ▄▄▀   █▄▄ ██████████████████▀▀   █▄▄ ▄▄ █▄▄█▄▄▄█▄▄▄
                           ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
                            ▀▀█████████████▄
                                █████████████▄
                                  █████████████▄
                                    ▀███████▀▀▀▀▀
                                      ▀████▀
                                        ▀█▀
LetItRideINNOVATIVE ▬▬▬
DICE GAME
                        ▄███████████▄
                       ██  ██████████▄
                     ▄█████████████  ██▄
            ▄▄▀█▄▄▄▄▄████████████████████▄
        ▄▄█▀   ███████████  █████  ████  █
    ▄██████ ▄▄███████████████████████████▀
 ▄▀▀ ██████████████████████████  ████  █
█  ▄███████████▀▀▀█████████████████████
██████████████    ████████▀▀██████  █▀
██████████████▄▄▄██████████   ▀▀▀▀▀▀▀
███▀ ▀██████████████████████
██    ███████████████████████
██▄▄██████████████████████████
██████████████▀   ██████████
  █████████████   ▄██████▀▀
     ▀▀██████████████▀▀
         ▀▀██████▀▀
PROVABLY
F A I R
▄█████████████▀ ▄█
██            ▄█▀
██          ▄██ ▄█
██ ▄█▄    ▄███  ██
██ ▀███▄ ▄███   ██
██  ▀███████    ██
██    █████     ██
██     ███      ██
██      ▀       ██
██              ██
▀████████████████▀
BUY  BACK
PLANS
[BTC]
danda
Full Member
***
Offline Offline

Activity: 184
Merit: 126


View Profile WWW
February 05, 2016, 05:47:34 PM
 #4

Interesting idea.  I like it.

Presently it can make use of 3 blockchain providers:  insight, toshi, and blockchain.info.

Insight and toshi accept only one address at a time.  blockchain.info accepts multiple.   So blockchain.info is the default.

The application code does the queries in batches, and the insight and toshi classes internally serialize the batches into single requests.

I just noticed that blockonomics also accepts multiple and does not require an API key.  I'll see about adding support for it.

insight and toshi can be run locally, and that removes any privacy risk.   But of course it is a giant hassle so most people will not do that.

Therefore, I like your idea as a way to regain some privacy, just at the expense of speed.  a convenient middle ground.

I will add it to the TODO list.

Probably I will do something like "--round-robin=yes --batch-size=2"

This way, the user can tell the app to use round robin, which will default to a batch size of 1, but user can still override batch-size for a performance boost.

Does it send multiple address at once to server ? I know it will be slow, but we can have option to send only one address at a time to random API provider.

mybitprices.info - wallet auditing tools
btc_enigma
Hero Member
*****
Offline Offline

Activity: 630
Merit: 501


View Profile
February 06, 2016, 05:13:41 AM
 #5

Quote
Probably I will do something like "--round-robin=yes --batch-size=2"

Yes, that sounds awesome ! Thanks . Let me know after the changes, would try to test your script

danda
Full Member
***
Offline Offline

Activity: 184
Merit: 126


View Profile WWW
February 07, 2016, 01:12:05 AM
 #6

@btc_enigma -- Are you associated with blockonomics?

I ask because I started working on integrating blockonomics API and ran into a roadblock. 

It appears there is not a suitable API available after all.   The /api/balance method is close because it accepts multiple addresses and returns a balance.  However, the balance can be 0 if an address has received funds and then sent them all, so what I actually require is total_received.

Additionally for full integration, I really need total_sent.  The other API's I'm using from toshi, insight, and blockchain.info do provide this info, and I make total_received and total_sent available as columns in the reports.

Any chance this data could be added to the balance API results, or add a new API such as /api/addresses?

thanks.

Quote
Probably I will do something like "--round-robin=yes --batch-size=2"

Yes, that sounds awesome ! Thanks . Let me know after the changes, would try to test your script

mybitprices.info - wallet auditing tools
danda
Full Member
***
Offline Offline

Activity: 184
Merit: 126


View Profile WWW
February 07, 2016, 08:21:46 PM
 #7

This release adds the following features:

  • added --api=roundrobin option which will cycle through blockchain providers one address at a time to improve privacy.
  • added --batch-size option for direct control over how large each address batch is.
  • added --gen-only option to generate addresses without checking the blockchain if they have been used.
  • added --type option to control which type of addresses are generated: receive, change, or both.

https://github.com/dan-da/hd-wallet-addrs/releases/tag/hd-wallet-addrs-v0.1.2


mybitprices.info - wallet auditing tools
siameze
Legendary
*
Offline Offline

Activity: 1064
Merit: 1000



View Profile
February 07, 2016, 08:38:05 PM
 #8

This release adds the following features:

  • added --api=roundrobin option which will cycle through blockchain providers one address at a time to improve privacy.
  • added --batch-size option for direct control over how large each address batch is.
  • added --gen-only option to generate addresses without checking the blockchain if they have been used.
  • added --type option to control which type of addresses are generated: receive, change, or both.

https://github.com/dan-da/hd-wallet-addrs/releases/tag/hd-wallet-addrs-v0.1.2



Thanks for the update. Will test later this evening and report back. Nice work!


                     ▀▀█████████▀████████████████▄
                        ████▄      ▄████████████████
                     ▄██████▀  ▄  ███████████████████
                  ▄█████████▄████▄███████████████████
                ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀████████
                                               ▀▀███▀
    ▄█▀█       ▄▀  ▄▀▀█  ▄▀   █████████████████▄ ██▀         ▄▀█
   ▄█ ▄▀      ▀█▀ █▀ █▀ ▀█▀  ███████████████████ █▀ ▀▀      ▄▀▄▀
  ▄█    ▄███  █     █   █   ████████████████████  ▄█     ▄▀▀██▀ ▄███
███▄▄▄  █▄▄▄ █▄▄ ▄▄▀   █▄▄ ██████████████████▀▀   █▄▄ ▄▄ █▄▄█▄▄▄█▄▄▄
                           ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
                            ▀▀█████████████▄
                                █████████████▄
                                  █████████████▄
                                    ▀███████▀▀▀▀▀
                                      ▀████▀
                                        ▀█▀
LetItRideINNOVATIVE ▬▬▬
DICE GAME
                        ▄███████████▄
                       ██  ██████████▄
                     ▄█████████████  ██▄
            ▄▄▀█▄▄▄▄▄████████████████████▄
        ▄▄█▀   ███████████  █████  ████  █
    ▄██████ ▄▄███████████████████████████▀
 ▄▀▀ ██████████████████████████  ████  █
█  ▄███████████▀▀▀█████████████████████
██████████████    ████████▀▀██████  █▀
██████████████▄▄▄██████████   ▀▀▀▀▀▀▀
███▀ ▀██████████████████████
██    ███████████████████████
██▄▄██████████████████████████
██████████████▀   ██████████
  █████████████   ▄██████▀▀
     ▀▀██████████████▀▀
         ▀▀██████▀▀
PROVABLY
F A I R
▄█████████████▀ ▄█
██            ▄█▀
██          ▄██ ▄█
██ ▄█▄    ▄███  ██
██ ▀███▄ ▄███   ██
██  ▀███████    ██
██    █████     ██
██     ███      ██
██      ▀       ██
██              ██
▀████████████████▀
BUY  BACK
PLANS
[BTC]
danda
Full Member
***
Offline Offline

Activity: 184
Merit: 126


View Profile WWW
February 08, 2016, 07:11:28 PM
 #9

cool, let me know how it goes.

Thanks for the update. Will test later this evening and report back. Nice work!

mybitprices.info - wallet auditing tools
btc_enigma
Hero Member
*****
Offline Offline

Activity: 630
Merit: 501


View Profile
February 09, 2016, 12:48:54 PM
 #10

@btc_enigma -- Are you associated with blockonomics?

I ask because I started working on integrating blockonomics API and ran into a roadblock. 

It appears there is not a suitable API available after all.   The /api/balance method is close because it accepts multiple addresses and returns a balance.  However, the balance can be 0 if an address has received funds and then sent them all, so what I actually require is total_received.

Additionally for full integration, I really need total_sent.  The other API's I'm using from toshi, insight, and blockchain.info do provide this info, and I make total_received and total_sent available as columns in the reports.

Any chance this data could be added to the balance API results, or add a new API such as /api/addresses?

thanks.

Quote
Probably I will do something like "--round-robin=yes --batch-size=2"

Yes, that sounds awesome ! Thanks . Let me know after the changes, would try to test your script

Got it  ! Yes, the API blockonomics is offering doesn't look suitable. Can you mail this to webmaster@blockonomics.co , they would be happy to help

danda
Full Member
***
Offline Offline

Activity: 184
Merit: 126


View Profile WWW
February 09, 2016, 09:06:10 PM
 #11

email sent.

mybitprices.info - wallet auditing tools
Lorenzo0
Newbie
*
Offline Offline

Activity: 6
Merit: 0


View Profile
February 10, 2016, 02:42:57 PM
 #12

Very interesting tool, thank you OP.

I'm left curious about the details of the auditing tool you mention.
Feel free to share any details about that, if it's ready.
danda
Full Member
***
Offline Offline

Activity: 184
Merit: 126


View Profile WWW
February 10, 2016, 08:01:10 PM
 #13

I just announced it.  Let me know your thoughts...

I'm left curious about the details of the auditing tool you mention.
Feel free to share any details about that, if it's ready.

mybitprices.info - wallet auditing tools
danda
Full Member
***
Offline Offline

Activity: 184
Merit: 126


View Profile WWW
April 08, 2016, 08:45:04 PM
 #14

I just made a minor release to the CLI tool and also the web tool.

Web Changes:

Added an option "Generate Only" to generate a fixed (user-defined) number of addresses without checking if each is used or not.  This speeds up processing and is useful for pre-generating addresses and other use cases.

CLI Changes:

This version adds support for blockr.io as a backend API provider and updates documentation.

https://github.com/dan-da/hd-wallet-addrs/releases/tag/hd-wallet-addrs-v0.1.3

64c2549 bump version to hd-wallet-addrs-v0.1.3
7f22536 Add legacy Copay gap limit warning
7e35d87 update README with blockr.io info
3eec473 add a blockr test case
6c8e2e9 added support for blockr.io API
d43d7b8 document --api=roundrobin feature
141c8fc update README with latest usage

mybitprices.info - wallet auditing tools
danda
Full Member
***
Offline Offline

Activity: 184
Merit: 126


View Profile WWW
August 07, 2017, 08:22:03 PM
 #15

update: I've made a couple minor releases since the last time I posted here.  Present version is 0.1.5.

Changes since 0.1.3:

  • Adds support for btcd as an API provider for finding used wallet addresses.  So one need not rely on a 3rd party API provider to find all wallet addresses that have ever been used.
    note: btcd does not provide received/sent/balance info for each address, so those fields are empty.
  • removes toshi from roundrobin list because toshi.io is no longer available.
  • now runs under php7 (as well as php5.5+)
  • Fixed bitwasp dependency version problem with php7.


mybitprices.info - wallet auditing tools
danda
Full Member
***
Offline Offline

Activity: 184
Merit: 126


View Profile WWW
August 02, 2018, 12:08:47 PM
 #16

The major new feature in 0.1.8 is support for segwit addresses (p2sh and bech32) via ypub and zpub extended keys used by some 3rd party wallets.

Also added/changed since 0.1.5:

  • blockcypher and btc.com can be used as api providers for funds lookup queries.
  • blockr.io api provider removed -- no longer exists.
  • auto-retry api providers 3 times in case of connection glitch
  • minor fixes and improvements

mybitprices.info - wallet auditing tools
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!