List of all Bitcoin addresses with a balance

[LoyceV]

@LoyceV (OP)
Hi, I need Bitcoin Public Key Database with balance

As albert0bsd said, it's complicated. Read the last few pages of this topic, read LoyceV's small Linux commands for handling big data and most importantly read this post on the problems I ran into at my attempt to get all known pubkeys.
I haven't found a solution, which is why I abandoned the pubkey project. If you can tell me how to get pubkeys out of the data I've published, I'll may it another try.

[pbies]

Is it possible to gain RIPEMD160 version of these addresses?

Eventually how to convert these addresses to RIPEMD160?

[LoyceV]

Is it possible to gain RIPEMD160 version of these addresses?

Eventually how to convert these addresses to RIPEMD160?

I'm not sure what RIPEMD160 is, and how it relates to pubkeys, but I think this applies:

The very act of hashing destroys any relation there could have been with inputs, no matter what the hashing algorithm is.

[wordbustingmistake]

Is it possible to gain RIPEMD160 version of these addresses?

Eventually how to convert these addresses to RIPEMD160?

P2PKH (26-35 characters long addresses starting 1, base58check encoded) and P2WPKH (42 characters long addresses starting bc1q, bech32 encoded) addresses are basically Ripemd160 hashes of public keys just encoded and with a little overhead (e.g. checksums). From loyce set of addresses you need to select only addresses matching proper length and starting characters, then use a decoder to verify checksums, decode addresses to hex and remove whatever overhead is leftover. Bit library for Python has base58 and base32 classes with methods to do that, so it's going to be something like:

Code:

print(base58.b58decode_check(P2PKH_address)[1:].hex())
print(bytes(base32.decode(P2WPKH_address)[1]).hex())

[pbies]

...

Hi, I need Bitcoin Public Key Database with balance
Please upload Bitcoin Public Key Database with balance your site http://addresses.loyce.club/

...

I can share such database. It is 5 MB as .rar.

Inside are: transaction ids, amounts, public keys.

You can obtain such file by go program with name "bitcoin-utxo-dump".

[seoincorporation]

...

Hi, I need Bitcoin Public Key Database with balance
Please upload Bitcoin Public Key Database with balance your site http://addresses.loyce.club/

...

I can share such database. It is 5 MB as .rar.

Inside are: transaction ids, amounts, public keys.

You can obtain such file by go program with name "bitcoin-utxo-dump".

I was reading about bitcoin-utxo-dump and looks like the right tool for this project, the files are not 5 MB, the file should be around 7GB and it takes close to 20 minutes to get all the unspent coins. After that we should group the unspent transactions from the same address and that way we can know the balance from each address...

The thing here is, you should run the dump each hour, then group the address and make a new csv file with the data, then update the database from the csv file, sounds as a busy cron job, but is possible.

Source: https://github.com/in3rsha/bitcoin-utxo-dump

[Mek]

Hello
if I run my own full node, how can I generate the same TSV file as is available on loyce.club site? (all addresses with balance)
Thanks.

[ABCbits]

Hello
if I run my own full node, how can I generate the same TSV file as is available on loyce.club site? (all addresses with balance)
Thanks.

It depends on which software you use to run full node. Some software (including Bitcoin Core) doesn't have address index, which force you to either perform workaround (e.g. load and parse UTXO) or install additional software (such as Electrum server or self-hosted mempool.space) which perform address index.

[LoyceV]

if I run my own full node, how can I generate the same TSV file as is available on loyce.club site? (all addresses with balance)

As far as I know, all you need for this is the data in the "chainstate" directory, which isn't that much. There are a few parsers out there, just try to find which one you like.
I don't need more frequent updates, so I never setup my own parser.

[stas87]

Hi, thanks you for the ability to download all funded bitcoin addresses. Do you have such option for USDT (Tron) addresses? Or its blockchain is a way too heavy? Thanks

[LoyceV]

Hi, thanks you for the ability to download all funded bitcoin addresses. Do you have such option for USDT (Tron) addresses? Or its blockchain is a way too heavy? Thanks

Nope, only Bitcoin here.

[Mek]

if I run my own full node, how can I generate the same TSV file as is available on loyce.club site? (all addresses with balance)

As far as I know, all you need for this is the data in the "chainstate" directory, which isn't that much. There are a few parsers out there, just try to find which one you like.
I don't need more frequent updates, so I never setup my own parser.

OK, can you please publish the way you generate the TSV file? The latest one on the website is already a week old. Maybe someone needs more frequent updates, or to have a backup plan should your website go down or stop being updated (these things happen...)

[LoyceV]

OK, can you please publish the way you generate the TSV file?

I did publish that, read the OP.

The latest one on the website is already a week old.

Thanks for this, I kinda forgot about this data project when I started another (big) download from Blockchair. Since I'm only allowed to download one connection at a time, it blocked other updates.
I'm running an update now, it should be available in about 3 hours.

Maybe someone needs more frequent updates, or to have a backup plan should your website go down or stop being updated (these things happen...)

If that happens, they'll have to figure it out on their own. If you're interested in discussing different parsers, it's probably best to open another topic about it.

[pbies]

For all of you, a script which downloads and converts LoyceV addresses to brainflayer bloom filter.
Taken are all >1 mBTC addresses.

Code:

#!/usr/bin/env bash
set -e
echo "### Public addresses downloader and converter to bloom filter ###"

echo [01/12] Download the addresses
wget http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz

echo [02/12] Unpack the archive
pv blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz | gunzip > blockchair_bitcoin_addresses_and_balance_LATEST.tsv

echo [03/12] Check lowest balance line
line=$(grep -m 1 -bne $'\t99999$' blockchair_bitcoin_addresses_and_balance_LATEST.tsv | cut -d ":" -f 2)

echo [04/12] Get only with proper balance
head -c $line blockchair_bitcoin_addresses_and_balance_LATEST.tsv > int1.txt

echo [05/12] Get rid of first line
tail -n +2 int1.txt > int2.txt

echo [06/12] Get only addresses, sort and write to file
pv int2.txt | cut -f 1 | sort > gte1mBTC.txt

echo [07/12] Search for incompatible addresses
line=$(grep -m 1 -bne $'^bc1p' "gte1mBTC.txt" | cut -d ":" -f 2)

echo [08/12] Get only compatible addresses
head -c $line "gte1mBTC.txt" > int3.txt

echo "[09/12] Convert addresses to hash160 ($(wc -l < int3.txt) lines)"
pv int3.txt | ~/brainflayer/brainflayer -o int4.txt

echo [10/12] Sort and uniq hashes
pv -c int4.txt | sort -u | pv -c > hash160.txt

echo [11/12] Convert hex to bloom filter
~/brainflayer/hex2blf "hash160.txt" bloom.blf

echo [12/12] Finished!
>&2 echo -ne "\x7"

[LoyceV]

For all of you, a script which downloads and converts LoyceV addresses to brainflayer bloom filter.

I've added it to the OP:

If you want to create a brainflayer bloom filter: pbies wrote a guide.

Note that I haven't tested it, so as always, do your own research. You shouldn't be messing with bloom filters if you don't know what you're doing anyway

[LoyceV]

Since HASH160 keeps coming up, I'll link World's fastest and simplest block parser for those who (only) need all HASH160 here. This is for all addresses ever used though.

[LoyceV]

I've moved this project to another (donated) server. Everything should be back online again, let me know if there are any issues.

[Ermitage]

Hi,
Why do I see only the address bc1qpcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t on blockchair.com but nowhere else?
BR

[citb0in]

Hi,
Why do I see only the address bc1qpcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t on blockchair.com but nowhere else?
BR

simply said, its an invalid bitcoin address. Just because blockchair displays it, doesn't mean it exists. Blockchair seems to accept any address via URL parameter, even invalid ones like the one you mentioned.

Valid characters in in bech32 addresses are: 023456789acdefghjklmnpqrstuvwxyz
So invalid characters are: 1bio

Try enter this URL and you will get a similar result of what you asked for:
https://blockchair.com/bitcoin/address/bc1q123testbio123testbio123

However, your question is off-topic. Please make sure that in future you select a suitable subforum in which your question fits. If in doubt, create a new thread.

[Ermitage]

Sorry, I didn't mention that my question is related to the thread, but indeed.
The LoyceVs list contains the line. So, the question was "Why does the list contain the line if the address doesn't exist in the real blockchain?".
The same for these addresses (for convenience, the first one is also included):
bc1pqcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t
bc1sw50qgdz25j
bc1pewum2h6sap5ch5lf9qns4t0ufe9f6wf3z5s225
bc1zw508d6qejxtdg4y5r3zarvaryvaxxpcs
bc1pvw7t698cn379acgstnzwtnn8r6qxernnsrul3z
bc1pgptwf4gcrwy3qvq26p3cwapagaxyvsd8syuwhq
bc1putjagjerw0qw9qmy0wxu2pfk0v5t8gu7ranaxg
bc1pj7gr7lpxdnf6fm82jyy602t7xs9jjf3cp30yc7
bc1prkqmm25t4fn84k3j0dvlhkut4yeyuwurekel3c
bc1pxcsyvgrxyp8jqmeqtqs8sgpkyq7zq0sxpf58k
bc1p042rxsz3865566ykcd956pymkn0fxkn8lq96jx
bc1px5sy2gr9yp8zqm3q2us8wgp4yq4jq0gf5cpy9
bc1pxusywgr8ypgzqupqtys8jgphyq4zqgcmufa07
bc1p23jk6urvv96x2gp3yqszqgpqyqszqgqgxs8es
bc1p8ysyjgrfypfzqu3q9usrqgpeyqnzqfgv63qmh
bc1pxvsyxgrrypxzqmpq25s82gpnypajqlg7ux9e3
bc1pxqsrzgpjyqejqdpqx5srvgphyquzqwgc3wgd8
bc1pdv6mhmhayxp7xjxut0zhdpexskf7te5wfa53pv
bc1pjw7qg604staq70s86sfdq80zy5df2ljdl9n370
bc1p8qsysgrgypgjqufqtgs85gpcyqjzqsqul7r68
bc1pxysyzgrpyp9zq63q2vs8xgp3ypdjqhgfsx3d3
bc1pchjs0jhpga0lrqwwn26k6a9v5hj9pr8e522vq8
bc1pxgsyygrzyp9jq6eq2ss8ggpjyq5zq2g4sz4gk
bc1pxssyggryypxjqmfq2cs8vgp5yqsjq0ctxl0l2
bc1zqyqs3juw9m
bc1pm9jzmujvdqjj6y28hptk859zs3yyv78hlz84pm