Best way to find batches of Base58 addresses

[mahurovihamilo]

Hi all,

what would you recommend to web scrap for batches of Base58 (legacy) addresses? what is the best way to find loads of them and inquire into their TXs and balance?

Thanks !

[pooya87]

Run a full node with modifications so that it can index the blockchain based on addresses like what the block explorers do so that there is an up to date list of addresses with all UTXOs.

Otherwise you can only use third party services like the existing block explorers with their limitations to get a historical snapshot of the blockchain, addresses or UTXOs. Like this one to get addresses (base58 and others) with balance alongside maybe this one.

[SilverCryptoBullet]

what would you recommend to web scrap for batches of Base58 (legacy) addresses? what is the best way to find loads of them and inquire into their TXs and balance?

Blockchair.com has data dumps.
https://gz.blockchair.com/bitcoin/addresses/

The file for Bitcoin addresses on 22-May-2024 01:23 is 1 GB. You can download the file (.tsv) and do your filter to check Legacy addresses and balances.

[mahurovihamilo]

Thank you Gentlemen, excellent suggestions!

[ABCbits]

Run a full node with modifications so that it can index the blockchain based on addresses like what the block explorers do so that there is an up to date list of addresses with all UTXOs.

OP also could run either block explorer/Electrum server, where OP could directly access database created by those.

Otherwise you can only use third party services like the existing block explorers with their limitations to get a historical snapshot of the blockchain, addresses or UTXOs. Like this one to get addresses (base58 and others) with balance alongside maybe this one.

It's great suggestion. Although LoyceV website seems to be down where my browser shows ERR_CONNECTION_RESET or PR_IO_TIMEOUT_ERROR.

what would you recommend to web scrap for batches of Base58 (legacy) addresses? what is the best way to find loads of them and inquire into their TXs and balance?

Blockchair.com has data dumps.
https://gz.blockchair.com/bitcoin/addresses/

The file for Bitcoin addresses on 22-May-2024 01:23 is 1 GB. You can download the file (.tsv) and do your filter to check Legacy addresses and balances.

It's not practical due to default 10 kB/s limit though.

You can download 1 file at a time and the speed is limited to 10 kB/s. Our API keys are not applicable for increasing the download speed of data dumps. If you are a business or an academia/non-profit organisation that requires a higher speed, please contact us at <info@blockchair.com>.

[pooya87]

It's great suggestion. Although LoyceV website seems to be down where my browser shows ERR_CONNECTION_RESET or PR_IO_TIMEOUT_ERROR.

It's not down since I can access it using my Tor browser. However I can't access it normally! The DNS lookup fails which is weird.

Code:

>nslookup -q=A http://addresses.loyce.club/
Server:  dns.google
Address:  8.8.8.8

*** dns.google can't find http://addresses.loyce.club/: Non-existent domain

[LoyceV]

It's great suggestion. Although LoyceV website seems to be down where my browser shows ERR_CONNECTION_RESET or PR_IO_TIMEOUT_ERROR.

It works for me:

Code:

wget  -O- http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz
~
100%[==============================================================================================>]   1.47G  27.6MB/s    in 55s

Just in case, I restarted Apache2. That took a lot longer than I expected, I don't know why. Maybe it needed a good kick

The DNS lookup fails which is weird.

Try without "http://", that made it work here.

what is the best way to find loads of them and inquire into their TXs and balance?

For "loads" of addresses, I recommend this:

How to use
The most likely use is to check a long list of Bitcoin addresses for a remaining balance.
On Linux, use this to find matching addresses (after extrating the compressed .gz file of course):

Code:

comm -12 Bitcoin_addresses_LATEST.txt <(cat mylist.txt | sort | uniq)

• Bitcoin_addresses_LATEST.txt: the extracted latest version downloaded from addresses.loyce.club.
• mylist.txt: your own list of addresses, one address per line.

This takes only seconds to check millions of addresses. If your text file has Microsoft formatting, you may need to use this instead:

Code:

comm -12 Bitcoin_addresses_LATEST.txt <(cat mylist.txt | fromdos | sort | uniq)

If you want the balances of many addresses:

Code:

grep -f mylist.txt blockchair_bitcoin_addresses_and_balance_LATEST.tsv

This is a bit slower and eats RAM: 500,000 input addresses uses 1.5 GB RAM and took 50 seconds. You could mix it with the earlier command to only search the balance for addresses that aren't empty.

Click the quote to download the files.

[mahurovihamilo]

Thank you LoyceV, this is very useful.

Is there a way or command you could recommend to sort or isolate Base58 (legacy) addresses from the TSV sheet > to a data.txt file?

Thanks.

[LoyceV]

Is there a way or command you could recommend to sort or isolate Base58 (legacy) addresses from the TSV sheet > to a data.txt file?

Use grep:

Code:

grep "^1" blockchair_bitcoin_addresses_and_balance_May_22_2024.tsv > legacy_and_balance.tsv

I did it for you: legacy_and_balance.tsv (file will be deleted in 24h).

[mahurovihamilo]

Thank you LoyceV !!

this is awesome ! Ill be able to do it from now on.