Bitcoin Forum

Bitcoin => Project Development => Topic started by: LoyceV on June 11, 2020, 03:30:34 PM



Title: List of all Bitcoin addresses with a balance
Post by: LoyceV on June 11, 2020, 03:30:34 PM
Background
There's another very useful data dump on Blockchair (https://gz.blockchair.com/bitcoin/addresses/) that deserves attention: blockchair_bitcoin_addresses_latest.tsv.gz (https://gz.blockchair.com/bitcoin/addresses/blockchair_bitcoin_addresses_latest.tsv.gz)!

On many occasions I found myself searching for a complete snapshot of all funded addresses, for instance to search for a balance when you only have a partial Bitcoin address. This file takes almost a day to download at 10 kB/s.

The data
See addresses.loyce.club (http://addresses.loyce.club/?C=M;O=D). I keep 18 snapshots of Blockchair's daily data.
Sample: blockchair_bitcoin_addresses_and_balance_DATE.tsv.gz (sorted by balance, highest first (in satoshis)):
Code:
address balance
35hK24tcLEWcgNA4JxpvbkNkoAcDGqQPsP      25550215765875
3KZ526NxCVXbKwwP66RgM3pte6zW4gY1tD      10185724750535
37XuVSEpWW4trkfmvWzegTHQt7BdktSKUs      9450577254951
.......
m-3165957a315e3d9d2de76eccb1140cb8      1
127TnYq7APW8WfKewd7EdxA8gMUXEtr623      1
1E6NkSVsBewyz8Z8wJBYVTKgWmdqcUSWkS      1
This file is in TSV (https://en.wikipedia.org/wiki/Tab-separated_values) format. It's probably too large (30 million rows) to import into a spreadsheet.

I create a new file with all funded addresses, without balances, and without "weird" addresses (the ones with "-" in it).
Sample: Bitcoin_addresses_DATE.txt.gz (sorted in alphabetical order):
Code:
1111111111111111111114oLvT2
111111111111111111112BEH2ro
111111111111111111112xT3273
.......
bc1qzzzz6hthvpjgtl9pgqyepwwzfrky2ntmm5ccpc
bc1qzzzzp3khtxe32q03qfm6epm5ytyytq7lfcakpn
bc1zqyqsywvzqe
This file is in TEXT format. The first addresses are burn addresses, nobody can access those funds. The (incomplete) last address somehow really got funded (https://blockchair.com/bitcoin/address/bc1zqyqsywvzqe).

Download speed should be around 1000 times faster than Blockchair (https://gz.blockchair.com/bitcoin/addresses/). I don't offer uncompressed downloads.

How to use
The most likely use is to check a long list of Bitcoin addresses for a remaining balance.
On Linux, use this to find matching addresses (after extrating the compressed .gz file of course):
Code:
comm -12 Bitcoin_addresses_LATEST.txt <(cat mylist.txt | sort | uniq)
  • Bitcoin_addresses_LATEST.txt: the extracted latest version downloaded from addresses.loyce.club (http://addresses.loyce.club/).
  • mylist.txt: your own list of addresses, one address per line.
This takes only seconds to check millions of addresses. If your text file has Microsoft formatting, you may need to use this instead:
Code:
comm -12 Bitcoin_addresses_LATEST.txt <(cat mylist.txt | fromdos | sort | uniq)

If you want the balances of many addresses:
Code:
grep -f mylist.txt blockchair_bitcoin_addresses_and_balance_LATEST.tsv
This is a bit slower and eats RAM: 500,000 input addresses uses 1.5 GB RAM and took 50 seconds. You could mix it with the earlier command to only search the balance for addresses that aren't empty.

If you want to know the balance of a certain address, but you don't want any block explorer or SPV wallet on the planet to know you're looking for that address, you can use this:
Code:
wget -qO- http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz | gunzip | grep -m1 1NXYoJ5xU91Jp83XfVMHwwTUyZFK64BoAD; sleep 5m; kill -9 $$
This doesn't save the file on your local drive, stops downloading after the first match, and kills your Bash shell 5 minutes later (so no .bash_history gets saved). Note (January 12, 2021): I now realize my webserver could know when you stop downloading. If you don't want this, remove "-m1" from grep.
If you want to search more than one address, please download the file first.

If you want to create a brainflayer bloom filter: pbies wrote a guide (https://bitcointalk.org/index.php?topic=5254914.msg62774192#msg62774192).

Data retention and updates
I'll provide daily updates. I keep the latest 6 daily snapshots, and the latest 12 monthly snapshots. One a day (or once a month) I delete the oldest files.

New: Direct link to LATEST versions
blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz (http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz) (currently 862MB)
Bitcoin_addresses_LATEST.txt.gz (http://addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz) (currently 750 MB)

New: Keeping track of the total number of funded addresses
See total_number_of_funded_addresses.txt (http://addresses.loyce.club/total_number_of_funded_addresses.txt). Starting December 2020, I'll add a daily total address count to this list. Long-term, this might prove useful (or at least produce nice graphs).

Credits
Blockchair Database Dumps (https://blockchair.com/dumps) has a staggering amount of data, easily accessible (at 10 kB/s (https://gz.blockchair.com/README.html)) with daily updates. All data presented in this topic comes from Blockchair.

No spam please.
Self-moderated against spam. Discussion and questions are welcome.



Related topics
Bitcoin block data available in CSV format (https://bitcointalk.org/index.php?topic=5246271.0)
List of all Bitcoin addresses with a balance (https://bitcointalk.org/index.php?topic=5254914.0)
List of all Bitcoin addresses ever used (https://bitcointalk.org/index.php?topic=5265993.0)
[~500 GB] Bitcoin block data: inputs, outputs and transactions (https://bitcointalk.org/index.php?topic=5307550.0)
[800 GB] Ethereum data (https://bitcointalk.org/index.php?topic=5307550.msg56043463#msg56043463)


Title: Re: List of all Bitcoin addresses with a balance
Post by: BrewMaster on June 12, 2020, 04:18:32 AM
how did you download these files? i've tried a couple of times using my cloud downloader but each time it failed near the end. i suspected the file might have been removed or changed since it took nearly a day to download.

I'm not sure what the weird "address" format at the end is.
for some reason blockchair.com assigns these weird format addresses to any outscript that they see. this includes OP_RETURN outputs too which can be found in almost all of the coinbase transactions.
the last one in your list for example is a simple P2PKH script that starts with a garbage being pushed to the stack followed by an OP_DROP which makes the pushto virtually not-exist.
https://blockchair.com/bitcoin/transaction/48e7694b3de213e2ab36cfcfaaa3013a22e1a5dc28168772cb01be99f91a12fa


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on June 12, 2020, 09:15:33 AM
how did you download these files?
wget (https://www.gnu.org/software/wget/)

Quote
i've tried a couple of times using my cloud downloader but each time it failed near the end. i suspected the file might have been removed or changed since it took nearly a day to download.
As long as your download doesn't disconnect, I assume the file you're downloading doesn't change. However, one of my downloads got corrupted, so I now test each new .gz-file before sharing it, and before deleting the oldest one.

Quote
for some reason blockchair.com assigns these weird format addresses to any outscript that they see. this includes OP_RETURN outputs too which can be found in almost all of the coinbase transactions.
the last one in your list for example is a simple P2PKH script that starts with a garbage being pushed to the stack followed by an OP_DROP which makes the pushto virtually not-exist.
https://blockchair.com/bitcoin/transaction/48e7694b3de213e2ab36cfcfaaa3013a22e1a5dc28168772cb01be99f91a12fa
I thought it must have been OP_RETURN indeed, but didn't look into it.
As an easy "solution", I'll remove the 397,175 "addresses" that contain a "-" from Bitcoin_addresses_DATE.txt.gz. I won't change existing files for this, so updates start in about 2 days.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on June 15, 2020, 09:25:06 AM
In Ratimov's Russian translation (https://bitcointalk.org/index.php?topic=5255422.0), FontSeli asked (https://bitcointalk.org/index.php?topic=5255422.msg54617683#msg54617683) how this can be useful for a normal person (thanks Google Translate). I'll answer this here.

I can think of several other uses:
  • Say you have a list of your old Bitcoin addresses (https://bitcointalk.org/index.php?topic=5254427.msg54589402#msg54589402) and want to know if they hold any balance. But you don't want to let any block explorer or Electrum server know which address you're looking at. By downloading all addresses, you can easily do a local search without anyone else knowing which address you're looking for.
  • I've also seen cases where someone has an incomplete Bitcoin address, and wants to know if it's worth to pursue recovering the private key. Block explorers don't let you search for incomplete addresses. As a rediculous example, here are all funded addresses that contain "Loyce":
Code:
1LoyceVSbrv3CP3jT12qVYrNSbrqz4FL5q      1499000
1PKDzdoRNoP1LoyceZLT1VwNsvP2GBuxk6      100000
1F9Q7dzMyFBkrNQvdo4ECLL5DvLoycewSw      71917
17YTjpKVtRLAVP6igaztot5Loycegd5igR      70701
1MakvXVkoAGJXhbbTEAM3Loyce6LibBaEQ      31590


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on June 21, 2020, 11:57:57 AM
still the choosing of any block explorer to pursue the  addresses of your interest would lead to  swifter result. All out good VPN would  help to hide from the curiosity expressed by those who are behind the  block explorer.
For many users that will be true. I just offer one more option :)


Title: Re: List of all Bitcoin addresses with a balance
Post by: bitmover on June 30, 2020, 02:41:59 AM
Nice job LoyceV.

You are good with these kind of data.

I am searching for historical volume data from exchanges, BTC pairs. I tried in coinmarketcap but I couldn't find there. Do you have any idea where I can find it?


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on June 30, 2020, 10:05:53 AM
I am searching for historical volume data from exchanges, BTC pairs. I tried in coinmarketcap but I couldn't find there. Do you have any idea where I can find it?
I haven't seen this data yet. I've seen sites that sell historic data, but I'm not sure if they include exchange volumes.
And even if you do find it, the data must have come from the exchanges themselves, and some of them are known to provide fake data.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on July 08, 2020, 11:33:57 AM
I can think of several other uses:
I thought of another use: what if you wanted to know how many addresses might be ChipMixer chips? They have specific sizes of 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 or 8192 mBTC. A quick search (using yesterday's address list) shows:
1 mBTC: 266408 addresses
2 mBTC: 66301 addresses
4 mBTC: 30343 addresses
8 mBTC: 15695 addresses
16 mBTC: 8290 addresses
32 mBTC: 3894 addresses
64 mBTC: 1797 addresses
128 mBTC: 1167 addresses
256 mBTC: 595 addresses
512 mBTC: 259 addresses
1024 mBTC: 442 addresses
2048 mBTC: 120 addresses
4096 mBTC: 139 addresses
8192 mBTC: 59 addresses



Title: Re: List of all Bitcoin addresses with a balance
Post by: bitwise on September 17, 2020, 03:45:08 PM
AFAIK you don't need to download this if you run a node
on any bitcoind node you can run dumptxoutset command and then just parse it

and when https://github.com/bitcoin/bitcoin/pull/18689/ gets merged, you won't even need to parse it


Title: Re: List of all Bitcoin addresses with a balance
Post by: btc_enigma on September 24, 2020, 09:43:56 AM
AFAIK you don't need to download this if you run a node
on any bitcoind node you can run dumptxoutset command and then just parse it

and when https://github.com/bitcoin/bitcoin/pull/18689/ gets merged, you won't even need to parse it


This feature would be really nice. Thanks for highlighting it


Title: Re: List of all Bitcoin addresses with a balance
Post by: alevlaslo on October 19, 2020, 06:40:49 AM
Hello, please divide the addresses into three groups, we can not import different types into the Vanitysearch program at the same time


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on October 19, 2020, 10:11:37 AM
Hello, please divide the addresses into three groups
I won't add different file formats for specific uses (because of restrictions on disk space and server load).

You're free to do that yourself though, it's quite easy:
For all Bech32 addresses:
Code:
wget http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_October_19_2020.tsv.gz -O - | gunzip | grep "^b" > Bech32_bitcoin_addresses_and_balance_October_19_2020.tsv
Replace the "b" by "1" or "3" for other addresses, and adjust the file date for the latest version (http://addresses.loyce.club/?C=M;O=D).



I'm glad to see this addresses subdomain consumes more data than the rest of loyce.club combined, that means it's being used :)


Title: Re: List of all Bitcoin addresses with a balance
Post by: alevlaslo on October 19, 2020, 11:37:06 AM
thank you, but I have Windows, where do I need to set this command and what program to install?


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on October 19, 2020, 11:54:21 AM
I have Windows, where do I need to set this command and what program to install?
I have no idea :P

So I got you this instead: http://alladdresses.loyce.club:20319/tmp/
It won't be updated, and it's scheduled to be deleted in 7 days.


Title: Re: List of all Bitcoin addresses with a balance
Post by: alevlaslo on October 19, 2020, 12:13:45 PM
thank you very much!


Title: Re: List of all Bitcoin addresses with a balance
Post by: alevlaslo on October 19, 2020, 01:03:12 PM
how do I remove extra characters? The program does not accept with them, each line only must has an address only, there should be nothing else


Title: Re: List of all Bitcoin addresses with a balance
Post by: bitmover on October 19, 2020, 02:02:29 PM
how do I remove extra characters? The program does not accept with them, each line only must has an address only, there should be nothing else

Which program are you using?
You will struggle a lot to manipulate this kind of data in apps such as excel.

personally, I could only open that 1GB files in my python environment. All other programs just froze

Edit: I was able to open the larger file in Notepad++.


Title: Re: List of all Bitcoin addresses with a balance
Post by: alevlaslo on October 19, 2020, 03:37:14 PM
I have Notepad++, but I don't have enough RAM, it freezes after selecting all the text and ALT+SHIFT+>>>


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on October 19, 2020, 04:05:01 PM
how do I remove extra characters?
You mean remove the balance? I just use cut (https://man7.org/linux/man-pages/man1/cut.1.html). I got you 3 new files (http://alladdresses.loyce.club:20319/tmp/) with only addresses.


Title: Re: List of all Bitcoin addresses with a balance
Post by: alevlaslo on October 19, 2020, 06:35:31 PM
thank you, I already figured it out, I have an old hard drive, I had to wait 1 hour.


Title: Re: List of all Bitcoin addresses with a balance
Post by: alevlaslo on October 21, 2020, 07:01:23 AM
for 130000000000000000000000000000000000/24000000 years I will find privkey of one address :)

but if I shall lucky, than I can find it in one second




https://c.radikal.ru/c17/2010/59/e7cccf4aefc1.png (https://radikal.ru)



Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on October 27, 2020, 12:41:54 PM
This subdomain is currently offline:
The data
See addresses.loyce.club (http://addresses.loyce.club/?C=M;O=D).
I think I ran out of bandwidth (250 GB) this month. I haven't hit this limit before, so I don't know if it requires manual intervention to be online again in November.
Update: it should be online again November 1st. I'll upgrade the hosting when transaction fees drop (https://bitcointalk.org/index.php?topic=2848987.0). I need to wait for the current hosting to be online again to copy data anyway.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on November 01, 2020, 09:07:10 AM
Update: It works again! I'll see what Black Friday brings for hosting offers :)


Title: Re: List of all Bitcoin addresses with a balance
Post by: alevlaslo on November 03, 2020, 07:06:14 AM
well alphabetically, thank you, it is convenient to delete unnecessary things with this command




https://a.radikal.ru/a22/2011/b0/fd1e21921d6at.jpg (https://radikal.ru/big/xkhzgrut1xcft)




Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on November 09, 2020, 11:46:28 AM
Interesting: I noticed Bitcoin_addresses_November_09._2020.txt.gz is 2 MB smaller than Bitcoin_addresses_November_08_2020.txt.gz. I checked if the newer file might be incomplete, but it doesn't look like that.
It turns out there were 45,303 funded addresses less on the 9th than on the 8th, which means someone (or many people) have been consolidating inputs (https://bitcointalk.org/index.php?topic=2848987.0) to take advantage of the lower fees during the weekend.
Once I have a different host for these files, I can keep daily snapshots a bit longer to see if there's a pattern.



A day later, there are 149,295 less funded Bitcoin addresses. I didn't expect such a large daily difference.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on November 14, 2020, 06:00:02 PM
I'm moving this to a new webhost. As a temporary location, the data will be available on http://old.loyce.club/?C=M;O=D until I'm done.
Done! This project now gets 1 TB per month bandwidth (4 times more), and a few more snapshots.
See addresses.loyce.club (http://addresses.loyce.club/?C=M;O=D).

If all goes well, it'll keep 6 snapshots. When I have a few, I'll see how I can keep some older ones too instead of only the most recent days.


Title: Re: List of all Bitcoin addresses with a balance
Post by: Vod on November 14, 2020, 06:26:55 PM
I have Notepad++, but I don't have enough RAM, it freezes after selecting all the text and ALT+SHIFT+>>>

Try the free Libre suite.  https://www.libreoffice.org/discover/calc/

Also, Visual Studio code can open large files/


Title: Re: List of all Bitcoin addresses with a balance
Post by: alevlaslo on November 15, 2020, 10:19:45 AM
thanks, but I no longer work on the address list, I use the public key to one address


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on November 19, 2020, 09:08:22 PM
If all goes well, it'll keep 6 snapshots. When I have a few, I'll see how I can keep some older ones too instead of only the most recent days.
Update: I changed my script to keep 9 snapshots. I'll keep 4 from the last 3-4 days, and 5 snapshots from the last 4-5 months. The older snapshots will be created in the coming months, I don't have them yet.

I couldn't test this yet, I'll see next month if it works as expected.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on December 06, 2020, 12:16:48 PM
Update:
If you want to know the balance of a certain address, but you don't want any block explorer or SPV wallet on the planet to know you're looking for that address, you can use this:
Code:
wget -qO- http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz | gunzip | grep -m1 1NXYoJ5xU91Jp83XfVMHwwTUyZFK64BoAD; sleep 5m; kill -9 $$
This doesn't save the file on your local drive, stops downloading after the first match, and kills your Bash shell 5 minutes later (so no .bash_history gets saved).
If you want to search more than one address, please download the file first.

New: Direct link to LATEST versions
blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz (http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz) (currently 862MB)
Bitcoin_addresses_LATEST.txt.gz (http://addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz) (currently 750 MB)

New: Keeping track of the total number of funded addresses
See total_number_of_funded_addresses.txt (http://addresses.loyce.club/total_number_of_funded_addresses.txt). Starting December 2020, I'll add a daily total address count to this list. Long-term, this might prove useful (or at least produce nice graphs).
Sample:
Code:
31,115,394 funded Bitcoin addresses on September 10, 2020
  23,536,754 of those start with 1
  6,025,854 of those start with 3
  1,552,770 of those start with bc1q

32,467,107 funded Bitcoin addresses on November 12, 2020
  24,223,336 of those start with 1
  6,402,910 of those start with 3
  1,840,844 of those start with bc1q

32,478,246 funded Bitcoin addresses on November 13, 2020
  24,230,032 of those start with 1
  6,400,301 of those start with 3
  1,847,896 of those start with bc1q
If you add up the addresses starting with 1, 3 and bc1q, you'll notice 16 addresses are missing. Those are:
Code:
bc1p23jk6urvv96x2gp3yqszqgpqyqszqgqa6qtuj
bc1p8qsysgrgypgjqufqtgs85gpcyqjzqsqfrw0l9
bc1p8ysyjgrfypfzqu3q9usrqgpeyqnzqfgexpv74
bc1pmfr3p9j00pfxjh0zmgp99y8zftmd3s5pmedqhyptwy6lm87hf5ss52r5n8
bc1pq2kqvpm76ewe20lcacq740p054at9sv7vxs0jn2u0r90af0k633322m7s8v
bc1pqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs3wf0qm
bc1pv22mcnt30gwvk8g72szz700n4tkkx2qur2adj6pt8hl37hcf9dascxyf42
bc1px5sy2gr9yp8zqm3q2us8wgp4yq4jq0guggdp8
bc1pxcsyvgrxyp8jqmeqtqs8sgpkyq7zq0snaecz5
bc1pxgsyygrzyp9jq6eq2ss8ggpjyq5zq2gqvjed5
bc1pxqsrzgpjyqejqdpqx5srvgphyquzqwgdd7yg9
bc1pxssyggryypxjqmfq2cs8vgp5yqsjq0c760r6g
bc1pxusywgr8ypgzqupqtys8jgphyq4zqgcwqe32u
bc1pxvsyxgrrypxzqmpq25s82gpnypajqlgtqkfun
bc1pxysyzgrpyp9zq63q2vs8xgp3ypdjqhguvkagn
bc1zqyqsywvzqe
I don't know the story behind them, someone has been creating non-standard outputs. See txid 8bb2ce18914cfcb68e21686362b879396c2c27b51f1ec4be25c064f48f848f2d (https://blockchair.com/bitcoin/transaction/8bb2ce18914cfcb68e21686362b879396c2c27b51f1ec4be25c064f48f848f2d) for most of them.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on December 18, 2020, 09:34:31 AM
I'm happily surprised this data has quite a few daily users:
https://loyce.club/other/traffic.png


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on December 26, 2020, 11:37:46 AM
I've counted some dust addresses:
573,966 addresses hold 546 satoshi.
168,663 addresses hold 547 satoshi.
74,835 addresses hold 548 satoshi.
2212 addresses hold 549 satoshi.
12,745 addresses hold 550 satoshi.
1,301 addresses hold 500 satoshi.
310,361 addresses hold 1000 satoshi.
6,010 addresses hold 9 satoshi.
4,209 addresses hold 1001 satoshi.

2,821,479 addresses hold 1000 sat or less. Total value: 1,403,085,278 satoshi (~14 BTC).
30,649,056 addresses hold more than 1000 sat. Total value: 1,858,148,094,500,905 satoshi (~18.58 million BTC).


Title: Re: List of all Bitcoin addresses with a balance
Post by: bitmover on January 06, 2021, 09:28:16 PM
I don't know the story behind them, someone has been creating non-standard outputs. See txid 8bb2ce18914cfcb68e21686362b879396c2c27b51f1ec4be25c064f48f848f2d (https://blockchair.com/bitcoin/transaction/8bb2ce18914cfcb68e21686362b879396c2c27b51f1ec4be25c064f48f848f2d) for most of them.

That's interesting.
All of them are unspent, so they aren't spendable probably.

Nodes shouldn't validate transactions to invalid addresses..


Title: Re: List of all Bitcoin addresses with a balance
Post by: cryptothoughts42 on January 20, 2021, 04:14:01 AM
@LoyceV do you know how to find out which Bitcoin block Blockchair's database dumps are taking a snapshot of?


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on January 20, 2021, 09:07:42 AM
@LoyceV do you know how to find out which Bitcoin block Blockchair's database dumps are taking a snapshot of?
My guess would have been midnight ("Blockchair time"), but it's not because this transaction mined 2021-01-20 00:10 (https://blockchair.com/bitcoin/transaction/f29e3164ea1dc6603721ad9943ff09ab5abac665c6537a69f440cc9922a60fee) is already included in the LATEST data dump (http://addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz).
If you want a more exact snapshot time: just compare current balances for a few recent transactions and see which one is included in the latest data dump.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on January 24, 2021, 09:44:19 AM
I'm happily surprised this data has quite a few daily users:
https://loyce.club/other/traffic.png
Joke's on me, at the rate this is going this VPS runs out of bandwidth (1 TB/month) 3 days before the end of this month:
https://loyce.club/other/traffic3.png


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on January 27, 2021, 06:15:14 PM
I've just doubled my allowed bandwidth (paid by Bitcoin Lightning Network) to 2 TB/month. However, the webhost hasn't processed the payment yet. If it processes it on time, you guys can continue downloading tomorrow. If not, it will reach it's 1 TB bandwidth limit tomorrow and get be suspended until the end of the month.

To be continued ........


Update: upgrade completed! I'm curious though: I'd love to know what this data is being used for. There must be more than a thousand downloads per month now.


Title: Re: List of all Bitcoin addresses with a balance
Post by: blauervogel on January 28, 2021, 01:17:11 PM
Hi LoyceV, thank you so much for providing the address list.

I wanted to let you know that I am building a collider which is similar to the "Large Bitcoin Collider" project. I know that the chances of a collision are fu**ing small, but I am interested in the design of bitcoin and thought that is a great way to learn about it.

Anyways, I downloaded it only once. So there must be many more interested in the list :D
Cheers


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on February 06, 2021, 08:50:31 AM
I wanted to let you know that I am building a collider which is similar to the "Large Bitcoin Collider" project. I know that the chances of a collision are fu**ing small
That's a great way to prove to yourself how secure Bitcoin really is ;)
Thanks for posting what you use this for.


Title: Re: List of all Bitcoin addresses with a balance
Post by: starmyc on February 07, 2021, 02:36:00 PM
I'm curious though: I'd love to know what this data is being used for. There must be more than a thousand downloads per month now.

Yes, me too I'm genuinely curious about this.


Title: Re: List of all Bitcoin addresses with a balance
Post by: Evilish on February 13, 2021, 10:56:29 PM
I've just doubled my allowed bandwidth (paid by Bitcoin Lightning Network) to 2 TB/month. However, the webhost hasn't processed the payment yet. If it processes it on time, you guys can continue downloading tomorrow. If not, it will reach it's 1 TB bandwidth limit tomorrow and get be suspended until the end of the month.

To be continued ........


Update: upgrade completed! I'm curious though: I'd love to know what this data is being used for. There must be more than a thousand downloads per month now.

Wow, more than a thousand downloads per month. That's bound to eat up a lot of bandwidth seeing as each download must be ~1 GB.

My guess would be some of those are people utilizing your list to locate their full addresses using first bits. I collect physical Bitcoin collectibles and currently use smartbit.com.au to find full addresses associated with physical coins using the provided first bits.

I feel it would be really neat to do that using shell and not having to go through a third party. But for my use case I would probably need to download the file only once and use the same copy over and over seeing as most older collectibles should already be under your latest list.

Thanks for your work on this. Count on one more download and an additional 1 GB increase in bandwidth in the next few days because I am going to download a copy and spin up a small script for my own use soon. ;)


Title: Re: List of all Bitcoin addresses with a balance
Post by: DougM on February 15, 2021, 03:22:17 PM
I've counted some dust addresses:
573,966 addresses hold 546 satoshi.
168,663 addresses hold 547 satoshi.
74,835 addresses hold 548 satoshi.
2212 addresses hold 549 satoshi.
12,745 addresses hold 550 satoshi.
1,301 addresses hold 500 satoshi.
310,361 addresses hold 1000 satoshi.
6,010 addresses hold 9 satoshi.
4,209 addresses hold 1001 satoshi.

2,821,479 addresses hold 1000 sat or less. Total value: 1,403,085,278 satoshi (~14 BTC).
fascinating...so your theory is these are small amount accounts are a result of dust attacks? Could some be 'left over change' during a transaction that is lost/forgotten about?  Thanks for sharing!


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on February 15, 2021, 03:37:31 PM
so your theory is these are small amount accounts are a result of dust attacks? Could some be 'left over change' during a transaction that is lost/forgotten about?
Dust attacks typically use the minimum amount (546 sat), so that's quite likely. From this data, I can't count the addresses that hold a dust input and also a larger amount.

Feel free to check a few:
Code:
1EM72MRxX5teiLDZDkp6vpB7Fs4gVcsBWF      546
197uWhobsZ4boSR8LjZTCpKkHuCQL15y4N      546
1DcjMsvCj1AATH6kQubsikK3pUeyFhSngp      546
1P2o8eACQFFDHHq4zZHSwPaTr347raGNNM      546
1ECYdDEmzRKm53i7H4PhCEar2SdtTemfgC      546
1J9xAa2fFabDZmPmapiczcRoeeSqtXLSC9      546
12wj7j4RSduvEt6TqoKTgH6W92Zdh1MUmL      546
1F1qvANdwZkSFosjxVY9VHD7phnZyhnXhH      546
1FN4QEPDT6K2QT6SNETmgPJdL4Qajx5x2K      546
1JzAdzFR89jT8rTCi8LzV6kj57NeXaBXd1      546
1Mik1tJvn6aunTgSbN2UzitzdZBoxeqV5E      546
3LGrNL6toFoeJb9DGYRKvvaheyVg9jCJq6      546
19gXeh5v5T2UTLJrPPWok424EuyTAtr3vK      546
3Pimkc1JiSiMVvMF4fATmWW1Z76TzVQQex      546
1EB6PXMKDE8fkN9DZPPuzH1qoN6nzuqjN6      546
1DVzTh1zA8azcjWvcHnwKvKc8ACYhSLRwd      546
3BSK1KX3WzdXcQDSoXL39g3X7HH1CKsUqy      546
17c6ii1kBTBDhEhB5DVjY5DLFe9pyYtf4y      546
1AKSbLMPjwErAsijTPUrNF87XfjMzEF7Jt      546
1C7tsDp4SGGsWzySycobW5SKZNDtGdzYd6      546
3BPMryLEyp6eCSgdbayrSroF1abU52H5Ti      546
3JbUHH3ZdfjMar6PDEFVxZVJcL4g3wxFZ1      546
1Km9HaKxEntE8NPaBp1ataTMVKeaMiZe4x      546
3Be4gfZBe2zfWNAR3zjqXpr2aCSwgJ9CkH      546
18kgQniviHyMxTEmhojASfvB39cuwayv19      546
1Nzzx5jFFv8NWoNSAMVxKK5EuVEnr91x2r      546
13CHpohNridm8L6VMSsZ4NqvVdbwqdsCwN      546
1N99f4eC6TbzF7iHCDzMhhaC2HW9qZGc1R      546
18ESDDtFFqFSbHfpe1VfNWg7A2Ym17QCFr      546
14aPgqsQAxrHktVDe4yHwHdmzNAb5pFiBB      546
39ZjTnjbFbXBHmPPm3SxHw8eBV34ggDzL7      546
19AhE5cFabYu1k7EZf1rCNMWz2nmZ4agx1      546
16owPsPAATsEz8mMC2SJEZkcnRC5n9Djb9      546
I only checked the last one (https://blockchair.com/bitcoin/address/16owPsPAATsEz8mMC2SJEZkcnRC5n9Djb9), and it's an Omli Layer transaction. Great, this crap spams the blockchain too :(

As for dust attacks: I'm now kinda curious how much was paid in fees (and how much data got added to the blockchain) because of people who included the dust in their transactions. It's a lot of data (https://bitcointalk.org/index.php?topic=5307550.0) so I can't quickly get a number for this.

If wallets could start ignoring dust inputs by default, that would be great!


Title: Re: List of all Bitcoin addresses with a balance
Post by: DougM on February 15, 2021, 06:00:07 PM
so your theory is these are small amount accounts are a result of dust attacks? Could some be 'left over change' during a transaction that is lost/forgotten about?
Dust attacks typically use the minimum amount (546 sat), so that's quite likely. From this data, I can't count the addresses that hold a dust input and also a larger amount.
..
I only checked the last one (https://blockchair.com/bitcoin/address/16owPsPAATsEz8mMC2SJEZkcnRC5n9Djb9), and it's an Omli Layer transaction. Great, this crap spams the blockchain too :(

As for dust attacks: I'm now kinda curious how much was paid in fees (and how much data got added to the blockchain) because of people who included the dust in their transactions. It's a lot of data (https://bitcointalk.org/index.php?topic=5307550.0) so I can't quickly get a number for this.

If wallets could start ignoring dust inputs by default, that would be great!
completely agree!

Quote
2,821,479 addresses hold 1000 sat or less. Total value: 1,403,085,278 satoshi (~14 BTC).
So do you think the dust attackers got their money worth?  based on your calculations for < 1000 addresses amounts to 14 BTC or  ~679K at present value for just the dust.  What exactly is their motive to justify this large outlay?  I suspect this represents a number of different 'dust attackers' over time so maybe hard to generalize, but I am curious what people think.


Title: Re: List of all Bitcoin addresses with a balance
Post by: willi9974 on March 04, 2021, 12:51:43 PM
i know (read) the list is from blockchair, but

what is the list
what is the point of the list
why blockchair created this list

Best regards,
Willi


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on March 06, 2021, 01:43:31 PM
what is the list
Read.

Quote
what is the point of the list
Up to the user.

Quote
why blockchair created this list
Ask them!



Due to growing daily file sizes, I've reduced the number of daily snapshots by 1:
Data retention and updates
I'll provide daily updates. I keep the latest 3 daily snapshots, and the latest 5 monthly snapshots. One a day (or once a month) I delete the oldest files.
The server ran out of disk space during the last download. Now it should be good for a while again.


Title: Re: List of all Bitcoin addresses with a balance
Post by: BrewMaster on March 29, 2021, 01:17:01 PM
Nodes shouldn't validate transactions to invalid addresses..

nodes don't relay non-standard transactions already but they must never not-validate any non-standard transaction (when they are confirmed) because that would break the backward compatibility of bitcoin and we no longer would be able to have soft forks.
SegWit addresses are non-standard by old client standards and yet they are valid.

however, this is a double edge sword. we can't prevent people from experimenting. that means they can pay to witness addresses with a version higher than what it is defined. it has been happening from early days too. for example the block 170060 contains  P2SH transaction while P2SH soft fork had never happened by then. now every full node treats this particular block as an "exception".


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on April 04, 2021, 09:13:38 AM
You guys have burned through roughly 1400 GB last month. Just a FYI: 2000 GB per month is the limit for my current hosting.


Title: Re: List of all Bitcoin addresses with a balance
Post by: Sanka555 on April 23, 2021, 07:08:41 AM
You guys have burned through roughly 1400 GB last month. Just a FYI: 2000 GB per month is the limit for my current hosting.

Maybe you just need to publish  the script ***.bat for windows for example, or on JAVA code (the best variant). which will simply take the Bitcoin Core wallet and make of it

blockchair_bitcoin_addresses_and_balance_*****.txt


I would be very grateful to you for that


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on April 23, 2021, 08:09:57 AM
Maybe you just need to publish  the script ***.bat for windows for example, or on JAVA code (the best variant). which will simply take the Bitcoin Core wallet and make of it
This data comes from Blockchair.com, I'm not sure what you're asking.


Title: Re: List of all Bitcoin addresses with a balance
Post by: Sanka555 on April 23, 2021, 08:14:48 AM
Maybe you just need to publish  the script ***.bat for windows for example, or on JAVA code (the best variant). which will simply take the Bitcoin Core wallet and make of it
This data comes from Blockchair.com, I'm not sure what you're asking.
I thought you create a base by decrypting files from Core wallet)


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on April 23, 2021, 08:17:08 AM
I thought you create a base by decrypting files from Core wallet)
Nope. See:
Credits
Blockchair Database Dumps (https://blockchair.com/dumps) has a staggering amount of data, easily accessible (at 10 kB/s (https://gz.blockchair.com/README.html)) with daily updates. All data presented in this topic comes from Blockchair.


Title: Re: List of all Bitcoin addresses with a balance
Post by: supika on April 28, 2021, 06:49:16 AM
Hi,

It's possible to add a column with the date of last spending tx?
Would be nice to identify the dormant addresses.
Thanks!


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on April 28, 2021, 07:11:51 AM
It's possible to add a column with the date of last spending tx?
Would be nice to identify the dormant addresses.
Someone asked a similar question by PM a couple weeks ago. This was my response:
I like the idea, and I could probably get it from this data:
The data
See blockdata.loyce.club (http://blockdata.loyce.club/)
blockdata.loyce.club/inputs/ (http://blockdata.loyce.club/inputs/) (386 GB)
blockdata.loyce.club/outputs/ (http://blockdata.loyce.club/outputs/) (121 GB)
blockdata.loyce.club/transactions/ (http://blockdata.loyce.club/transactions/) (44 GB)
But it's a bit more work than I currently want to do for this, especially because "inputs" doesn't show input addresses, so it takes some additional steps.
If you do get the data out of this, please post it on Bitcointalk :)
I could offer this as a (paid) service if you really want it, in that case make me an offer.



Out of curiosity: why do you need dormant addressess? If you're trying to brute-force private keys, does it really matter if they're long-term hodlers (or forgotten)?


Title: Re: List of all Bitcoin addresses with a balance
Post by: supika on April 29, 2021, 08:40:02 AM
Out of curiosity: why do you need dormant addressess? If you're trying to brute-force private keys, does it really matter if they're long-term hodlers (or forgotten)?

Thanks for your answer. It was a curiosity of how many addresses are untouched for more than 10 years.
From my point of view brute forcing of sha256 encrypted private keys is a waste of time.


Title: Re: List of all Bitcoin addresses with a balance
Post by: willi9974 on April 29, 2021, 08:44:27 AM
maybe another idea than bruteforce.
Generating billions of new BTC addresses and checking for one generated would be included in the list --> illegal gambling


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on May 20, 2021, 11:37:20 AM
This is new:
https://loyce.club/other/traffic4.png
Since a few days, it looks like someone is downloading the latest version multiple times per hour, while I only update it once a day. I don't like having to look at logfiles, but it seems to be coming from different IP-addresses.
If someone made a mistake in a cronjob: please set it to download once a day! If this doesn't stop I'll have to ban some IPs :(



Update: From the past few days, I've identified 101 different IP addresses that downloaded the same file in the same way 169 times. The IPs I checked are owned by Microsoft. Maybe a proxy server or VPN? Either way, I'll DROP their connections now. I expect to add more IPs to the list later. I don't like the way someone burns through 50 GB bandwidth per day.

Although unlikely: If I added your IP by mistake: please PM me.
If you don't want me to know which forum account belongs to which IP: setup a throw away email to contact me:
Code:
LoyceVswitzerland@protonmail.com



As expected: 2 new downloads with 2 new IP addresses. I'll now try to DROP their connection within a few seconds, so their download won't complete. I'm still curious who's doing this, and why. It's especially annoying that it's a new IP-address all the time, now I risk banning real users. But at the rate this is going, my 2 TB bandwidth won't make it to the end of the month.

I'd love to have a more professional way to handle this :)


Title: Re: List of all Bitcoin addresses with a balance
Post by: Sogolev on May 24, 2021, 02:01:20 PM
Greetings to the audience!

I ask experienced users to tell me what tools I can use to collect a file containing all the bitcoin addresses that have ever had a positive balance.

Maybe there is ready-made software that can do this.

Thank you in advance


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on May 24, 2021, 02:27:39 PM
I ask experienced users to tell me what tools I can use to collect a file containing all the bitcoin addresses that have ever had a positive balance.

Maybe there is ready-made software that can do this.
See my other topic: List of all Bitcoin addresses ever used (https://bitcointalk.org/index.php?topic=5265993.0).
However: I've been struggling to find an affordable host that can handle the data processing, so I haven't updated it in months. And since a few days (https://bitcointalk.org/index.php?topic=5307550.msg57050991#msg57050991), my temporary server is offline (and hasn't responded yet).
I don't really want to spin up a paid-by-the-hour VPS just for this.


Title: Re: List of all Bitcoin addresses with a balance
Post by: bitmover on June 03, 2021, 05:29:55 PM
I have been monitoring the total number of funded addresses here  (https://bitcoindata.science/bitcoin-funded-addresses.html), with the data provided by LoyceV here (http://addresses.loyce.club/total_number_of_funded_addresses.json)

Segwit adoption keeps growing, look at this chart:

https://i.imgur.com/z5E9GIS.png
https://bitcoindata.science/bitcoin-funded-addresses.html

Since Sep 2020, the total number of Bech32 addresses had more than 100% increase!
There were 1,552,770 total funded addresses (5%), and now we have 4,046,320 (10,8%).

The number of P2SH addresses had a small increase as well, from 19,4% to 22%.

And the total number of Legacy addresses is basically the same, 23 to 25 millions addresses.



We can conclude that most of the new created addresses are Bech32 and P2Sh addresses, while the total number of Legacy grows at a much slower rate

Just added this text to the website:
Quote
Since September 2020, the total number of addresses has changed from 31,115,394 to 37,551,189 (20.68%)

Legacy addresses change: 7.43%
P2SH addresses change: 36.25%
Bech32 addresses change: 161.17%


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on June 05, 2021, 10:21:31 AM
From the past few days, I've identified 101 different IP addresses that downloaded the same file in the same way 169 times. The IPs I checked are owned by Microsoft. Maybe a proxy server or VPN? Either way, I'll DROP their connections now. I expect to add more IPs to the list later. I don't like the way someone burns through 50 GB bandwidth per day.
Well, it got worse:
https://loyce.club/other/traffic5.png
What's wrong with people? :P The same 1 GB file got downloaded approximately 200 times in an hour, this time using Google servers. And since incoming bandwidth is free, they can just burn through my bandwidth with virtually no cost. But why? Just spam my feedback like normal people if you don't like me :P

I've increased the IP-bans, I now DROP entire IP ranges.

This still applies:
Although unlikely: If I added your IP by mistake: please PM me.
If you don't want me to know which forum account belongs to which IP: setup a throw away email to contact me:
Code:
LoyceVswitzerland@protonmail.com


Title: Re: List of all Bitcoin addresses with a balance
Post by: LeTH3knXoDArzm on July 07, 2021, 09:46:41 AM
Hi LoyceV,
I really appreciate your work, but I have a question... you can upload the content on https://wetransfer.com/ and use that for the main link, and somewhere in the page add the link on your host. It would save loooooots of traffic.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on July 07, 2021, 09:52:54 AM
you can upload the content on https://wetransfer.com/ and use that for the main link, and somewhere in the page add the link on your host. It would save loooooots of traffic.
Thanks for the suggestion, but I'm not going to do this. I don't like using a third party that requires additional attention from the downloader. If anything, I'll use Torrent in the future.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LeTH3knXoDArzm on July 07, 2021, 10:32:53 AM
you can upload the content on https://wetransfer.com/ and use that for the main link, and somewhere in the page add the link on your host. It would save loooooots of traffic.
Thanks for the suggestion, but I'm not going to do this. I don't like using a third party that requires additional attention from the downloader. If anything, I'll use Torrent in the future.

Or so! It's a great option, took a look on google for seedboxes and found this (first link) https://seedbox.io/shared-seedbox/ there's a monthly payment (6 euros) but there's unlimited traffic.
But there are some cons:
1) There's a fee to be paid monthly (6 euros)
2) The hassle to upload the files every day on that seedbox
3) You should implement some google captcha to download directly from loyce.club.
..
..
..
it worth the work? :)


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on July 07, 2021, 06:38:05 PM
1) There's a fee to be paid monthly (6 euros)
I want this to be a long-term project. If it's going to cost 720 euro for 10 years, I won't keep it up forever.

Quote
2) The hassle to upload the files every day on that seedbox
I've never used a seedbox, but a regular VPS is very convenient.

Quote
3) You should implement some google captcha to download directly from loyce.club.
Those are more annoying for humans than for bots, so I won't do that. I'd rather implement a 10 sat LN payment for each download :)


Title: Re: List of all Bitcoin addresses with a balance
Post by: GhostOfCommunism on August 08, 2021, 08:55:42 AM
Hi! How to get latest block numbers?


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on August 08, 2021, 09:05:14 AM
How to get latest block numbers?
You can choose from many different methods. For instance: in Bitcoin Core, click Window > Information, It shows: "Current block height 694768". Or use Blockchain.com's explorer (https://www.blockchain.com/explorer)'s Latest Blocks.

If you want a nice visualization with it, use Billfold's BitBonkers (https://bitbonkers.netlify.app/). Turn on your sound, put it to full screen, and watch transactions and blocks drop from the sky. Click balls for details.



This doesn't have much to do with this topic, so I'm not sure what exactly you're looking for.


Title: Re: List of all Bitcoin addresses with a balance
Post by: PrivatePerson on August 25, 2021, 11:06:55 AM
Need help :)

Code:
comm -12 bitcoin_addresses_25.08.2021.txt <(cat My-addresses.txt | sort | uniq)
comm: file 1 is not in sorted order
Need to sort the bitcoin_addresses_25.08.2021.txt file somehow?


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on August 25, 2021, 11:18:08 AM
Code:
comm -12 bitcoin_addresses_25.08.2021.txt <(cat My-addresses.txt | sort | uniq)
comm: file 1 is not in sorted order
Need to sort the bitcoin_addresses_25.08.2021.txt file somehow?
Your file name is different from mine: did you download (and uncompress) addresses.loyce.club/Bitcoin_addresses_August_24_2021.txt.gz (http://addresses.loyce.club/Bitcoin_addresses_August_24_2021.txt.gz)?
That file is sorted already, and I just checked the command to confirm it still works. What exactly did you do to get this file?


Title: Re: List of all Bitcoin addresses with a balance
Post by: PrivatePerson on August 25, 2021, 02:07:34 PM
i had blockchair_bitcoin_addresses_and_balance file
I removed the balances from it. Do you want both files to be sorted by name?


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on August 25, 2021, 02:35:23 PM
i had blockchair_bitcoin_addresses_and_balance file
I removed the balances from it.
That explains why it isn't sorted: the files with balance is sorted by balance (highest first). The comm command requires the file to be sorted alphabetically.
You can just use the sort command on it (make sure there are no tabs left in the file), or download the other file (http://addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz).


Title: Re: List of all Bitcoin addresses with a balance
Post by: PrivatePerson on August 25, 2021, 07:06:04 PM
Where should this command output the results?
I intentionally inserted one of my addresses into the list of addresses with balances and at the end of the check nothing was displayed.
Both lists are sorted.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceMobile on August 25, 2021, 07:21:31 PM
Where should this command output the results?
The matching addresses are shown on console.
Does your list have spaces or Windows formatting by any chance?


Title: Re: List of all Bitcoin addresses with a balance
Post by: PrivatePerson on August 25, 2021, 07:50:19 PM
formatted in windows > sublime Text - UTF8


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceMobile on August 25, 2021, 08:12:30 PM
formatted in windows > sublime Text - UTF8
Try to add the fromdos pipe.


Title: Re: List of all Bitcoin addresses with a balance
Post by: PrivatePerson on August 25, 2021, 08:28:11 PM
Eureka! It worked, not clear why, but it worked.
I use sort for Linux, it did not help.
What are the addresses in the list of the format:
m-f0038f17d4787569b404c383d2cdb031?


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on August 26, 2021, 07:54:40 AM
not clear why, but it worked
Windows uses a different end of line character (https://www.cs.toronto.edu/~krueger/csc209h/tut/line-endings.html) than Linux, and since comm compares the entire line, all your addresses will include a different character, and therefore won't match the list of funded addresses.
Using fromdos fixes it.

Quote
What are the addresses in the list of the format:
m-f0038f17d4787569b404c383d2cdb031?
See:
This is how Blockchair.com shows OP_RETURN. From the main page the search field doesn't show them, but you can replace a Bitcoin address in the URL to find them: https://blockchair.com/bitcoin/address/d-d0d953f2e7043342540a1407243e49fe.


Title: Re: List of all Bitcoin addresses with a balance
Post by: PrivatePerson on August 26, 2021, 11:26:55 AM
Thanks for the help!


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on October 06, 2021, 11:03:18 AM
This webhost is temporarily offline. See Twitter (https://twitter.com/GulloStatus/status/1445153848453124098) for my host's status updates. Current estimate is about 2 days of downtime to go.
Update: it's back online, currently running today's update.

Before it went down, I noticed several daily updates were missed in the past month. I don't know the cause of that, I only noticed it afterwards.



If anyone knows an affordable (storage) VPS with enough bandwidth allocated, please let me know. Say 1 TB storage and 10 TB bandwidth. "Affordable" means this is just a side-project that I'm not willing to spend hundreds of dollars worth of Bitcoin on.


Title: Re: List of all Bitcoin addresses with a balance
Post by: jovica888 on October 31, 2021, 09:36:13 PM
Can you upload this big files for example to "Wetransfer" or "Torrent"... I Have 1 Gb/s downloading 800MB 45 minutes


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on October 31, 2021, 09:52:00 PM
Can you upload this big files for example to "Wetransfer" or "Torrent"... I Have 1 Gb/s downloading 800MB 45 minutes
No: I won't use Wetransfer and Torrent won't really help for these small files :)
Addresses.loyce.club (http://addresses.loyce.club/) is working fine now. 800 MB should take you 1 or 2 minutes.

I'm hoping for a good Black Friday deal on a new storage servers, those have been scarce. Some say it's because of Chia hdd mining.


Title: Re: List of all Bitcoin addresses with a balance
Post by: CKKwan on November 03, 2021, 08:03:41 AM
New: Direct link to LATEST versions
blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz (http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz) (currently 862MB)
Bitcoin_addresses_LATEST.txt.gz (http://addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz) (currently 750 MB)

This is what I got after extracting the content of the file provided by the link above. May I know how can I get the balance?

Thx

Code:
block_id transaction_hash index time value value_usd recipient type script_hex is_from_coinbase is_spendable spending_block_id spending_transaction_hash spending_index spending_time spending_value_usd spending_sequence spending_signature_hex spending_witness lifespan cdd
707781 3b9e2e90f4fa4fe5bb88a02f6941556af8f96c8da24799f8247e00ea046c5dac 3 2021-11-01 23:57:34 495453564 304267.9375 17A16QmavnUfCW11DAApiJxp7ARnxN5pGX pubkeyhash 76a91443849383122ebb8a28268a89700c9f723663b5b888ac 0 -1 707782 a40e8ce34941857ab0a6d43608e4b3f46f9dec536b9996a0bbdaa4d03ee56483 0 2021-11-02 00:08:47 302479.3438 4294967295 4730440220657eaf1d7a7437bc43d5fed9fefe3a20050d851d20d84e49aae06ae77a282be302200de42995d1845351a51378af797995b417437f7b897d68f010be6d26a8bfb3c40141047146f0e0fcb3139947cf0beb870fe251930ca10d4545793d31033e801b5219abf56c11a3cf3406ca590e4c14b0dab749d20862b3adc4709153c280c2a78be10c 673 0.038592621362499996
707779 a2e73b31479e093808358d69fcd6ca7077784a221d0f2e1e012c81e64b2d4076 1 2021-11-01 23:45:32 8426099 5174.6357 3EJKvTDK4qfnmCZGTupkCJKwpYvcoPh3iC scripthash a9148a4f17a7a0a009defd16398346ff6676c56a54a087 0 -1 707782 2f3b6627353ad9b7a8f0e80b625bfee47658de46f3b92ca3b7ab6f836fb7618c 0 2021-11-02 00:08:47 5144.2178 4294967295 2200209d88df8678e20d25689764c898adf19acd527e6f3c3feb2a8357d92a7286eece ,3045022100c94a1fa096b44e433f28430622c072a85456620583b39134d95e914f21b1c90402204d2b33ad7f0fb23c3fdb9f7b0101efe7ae30b635c27db95d5a492c74e58f05a601,304402207ff9f64a641124f4141dda3283bbe4d26da670f6082745d4ac5e8bfb240cb1de022053b79f60104f6fa5381a1728ad102a6cf6ecacab68254f043ec28f0195a6478901,5221030daf2b128d8a9fb959664ee334b668ca43b74441367cea3616e208fbbaaf568c2103a34f5c21e98f801b0d848e6401538074e049f993e80c66b07c2f28d37e844eef2102778952c666d794865b42e39a2541781d216f9299477001023b8186a7faf57e0353ae 1395 0.0013604639010416665
707776 690c212f2010075bbaa393c3f2d7d3a8739d862ced53384eb596038c9f010109 0 2021-11-01 23:33:04 1029410682 632181.6875 115md8GZ5S8CmvUj8a8cW58YcKvv2SekYf pubkeyhash 76a91400e6e9c2ab476c627c157ccaa2a1a526b86fb79788ac 0 -1 707782 04ff6bbeabb0faa6fec86f7f2dd292b6f3d0524ddcc826b9fd1b62660d5c581e 0 2021-11-02 00:08:47 628465.5 4294967295 473044022053304e19fba6c8933139d647d06146d857a2483ba564f6c72440767a44260d6502204f43503fc1ccc5f4cf3ae2001ec825c3f0d7c85bde9a96e7dcee00bcda668b95012103b3c036c7b89095d00ddf9b23b93c4781d7c486420ea75ce9d6404344e4ddb4fb 2143 0.2553272096673611
707781 2289e6937ea7f96420f4f4a8303acfadb062be2cc9d40e1dc7ff7eee3d8baa0b 0 2021-11-01 23:57:34 1028078900 631363.8125 115md8GZ5S8CmvUj8a8cW58YcKvv2SekYf pubkeyhash 76a91400e6e9c2ab476c627c157ccaa2a1a526b86fb79788ac 0 -1 707782 6f34766bc533b55b108527254a37cdf883df3351a1d3063bca586bc08793305f 0 2021-11-02 00:08:47 627652.4375 4294967295 483045022100b88ac8d9e60b735bbf1cb7a46463257c847f4fa5e40874c2fa2a8a245e2e519e0220189ba4db5a8b95608eeaacdd906265719f006392fb4a94780e9bd4653921e084012103b3c036c7b89095d00ddf9b23b93c4781d7c486420ea75ce9d6404344e4ddb4fb 673 0.08008068283564815
707780 731c26594d1fa8048216542745ac8df127bad75d32289753c945f9adf4e9b218 5 2021-11-01 23:53:51 150000 92.118 32ECMLtoA4R92YLsZi3dVV3J3EhDdgSfkV scripthash a91405e5996d7a30bd3ba56b499e0359afd8679ac10f87 0 -1 707782 7d7607e72ebec8da5243785c5bcb238937dc97ae95198fa426ae5dc2e9de967f 0 2021-11-02 00:08:47 91.5765 4294967295 1600147532e84aaff981d8ac62f86769ba048f2733900b 304402201d96222e49702c434f60c81991169c5594a585dfc66799fd75aa4b6e416e3da70220385f85fdaca12c7b7ef59a6d9298c1a49203dd035424ec97b07f9f313148d71501,036cab780d415f14a6218b01cc435c4eb83d93e2faa2068a76c314485a6bfd4777 896 0.000015555555555555555
707776 a0269077bc0c5c824f5a33a20bc1bc0a912deeb8487cd7c15faa6df6e7e4f334 1 2021-11-01 23:33:04 249126 152.9933 3PbJsixkjmjzsjCpi4xAYxxaL5NnxrbF9B scripthash a914f03e6bf9b389bbd5d5669ff55c4dba30de99553587 0 -1 707782 7d7607e72ebec8da5243785c5bcb238937dc97ae95198fa426ae5dc2e9de967f 1 2021-11-02 00:08:47 152.0939 4294967295 160014f7bae6ee31d59e79da8e857fb63429637f9e0d57 3044022045be9d951d43a01075f501f9b2da5222597420cb9d2f4c9e86eb92ca836065fb022027a5e1f6347bb31253ad9f0725a4b8d129f2413ec4218ad79e6543b964396bf401,034ed258709969db6507e5b86568e6c57d903917034a853fd2f6b7604443431518 2143 0.00006179132152777778
707772 e8d7df31409460f4bc37f2a59389610cb6e34579fc171adadbf78f81a37da600 2 2021-11-01 23:14:31 11564045 7101.7114 bc1q4kmm0ud3qzml0frwcjxhgwlndmpfanzk48rkr2 witness_v0_scripthash 0014adb7b7f1b100b7f7a46ec48d743bf36ec29ecc56 0 -1 707782 73f16d7fadf951296d674880472e753000a6b0b35a362524f900ce13fe203a40 0 2021-11-02 00:08:47 7059.9653 4294967293 304402205761e08d6936715e98d6f1e939f19ab9776a56b1badecc79ab74811487dce17902206e42965c7fc5e0adbcb2556cc5f0dc9c9af37b1dfc46eed4eb0ff887333107b601,0279ac41a2bb9b0cedca41b9b7b9fad4cdff875d7fbb4a3474c3ad74588be19e27 3256 0.004357931773148148


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on November 03, 2021, 08:46:07 AM
This is what I got after extracting the content of the file provided by the link above. May I know how can I get the balance?
Thanks for this, it looks like gz.blockchair.com changed the file I use (https://gz.blockchair.com/bitcoin/addresses/) a few days ago: it's now much smaller and contains the wrong data. I'm pretty sure that's a mistake. I'll wait a bit to see if they fix it on their own before I contact them.

I'll restore older versions for now.
Update: done! This is the latest version: addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_October_28_2021.tsv.gz (http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_October_28_2021.tsv.gz). I'll keep an eye on Blockchair to see when I can continue daily updates.


Title: Re: List of all Bitcoin addresses with a balance
Post by: CKKwan on November 04, 2021, 11:40:52 AM
Thanks, the correct format is easy to parse. It is very helpful  :)


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on November 04, 2021, 02:49:13 PM
I'll wait a bit to see if they fix it on their own before I contact them.
It looks like they fixed it: gz.blockchair.com/bitcoin/addresses/ (https://gz.blockchair.com/bitcoin/addresses/) is 1GB again (the faulty version was around 200MB). I ran an update yesterday, and tomorrow daily updates should be back to normal.



I noticed (http://addresses.loyce.club/total_number_of_funded_addresses.txt) my script skips a few days once in a while (for instance, there were no updates between September 27 and October 08). I think it also has to do with Blockchair's data, that's why I keep a few older versions.


Title: Re: List of all Bitcoin addresses with a balance
Post by: jovica888 on November 04, 2021, 08:09:06 PM
At the moment the website is not working. Can you maybe split files so 1 file will contain only addresses that start with "1" for example "1jFkHrG18Hc6ae9jTxXyJxqHyzmUjNdyQ"?

Also can I donate via Paypal?


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on November 05, 2021, 08:48:11 AM
At the moment the website is not working.
I don't know why, I don't see any down time on the server-site.

Quote
Can you maybe split files so 1 file will contain only addresses that start with "1" for example "1jFkHrG18Hc6ae9jTxXyJxqHyzmUjNdyQ"?
For Linux users, that's very easy to get:
Code:
wget -qO- http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz | gunzip | grep "^1" > Bitcoin_legacy_addy.txt
I don't want to add additional files because this VPS only has 20 GB, and I prefer to keep a few older versions. I got you this (http://addresses.loyce.club/Bitcoin_legacy_addy.txt.gz) as a temporary solution, it's scheduled to be deleted in 72 hours.

Quote
Also can I donate via Paypal?
No, I like my privacy. I can setup some altcoin though if that's okay.


Title: Re: List of all Bitcoin addresses with a balance
Post by: jovica888 on November 05, 2021, 09:18:24 PM
I downloaded your link. Thank you


Title: Re: List of all Bitcoin addresses with a balance
Post by: willi9974 on November 05, 2021, 09:25:34 PM
Can i ask you all what you all do with the data Form the file?

Best regards
Willi


Title: Re: List of all Bitcoin addresses with a balance
Post by: albert0bsd on November 06, 2021, 04:52:36 PM
Hi LoyceV since you already have some useful data, did you believe that you can publish some list like "Public keys with balance".


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on November 06, 2021, 04:59:21 PM
can publish some list like "Public keys with balance".
As far as I know the public key is only known after a transaction was made from that address (or when a message was signed), and I haven't found this data in a format that I can easily scrape. It certainly won't be a very complete list.


Title: Re: List of all Bitcoin addresses with a balance
Post by: albert0bsd on November 06, 2021, 05:16:07 PM
Well, that is a pity, thanks for the fast reply.

I will stick to my current list of dormants-publickeys with subtracted values  Total (3.5 Million public keys) at some 400 Gkeys/s

Wonderful work you made.

Thanks!


Title: Re: List of all Bitcoin addresses with a balance
Post by: mausuv on November 13, 2021, 02:31:10 PM
Well, that is a pity, thanks for the fast reply.

I will stick to my current list of dormats-publickeys with subtracted values  Total (3.5 Million public keys) at some 400 Gkeys/s

Wonderful work you made.

Thanks!

subtracted problem i need update

i am use your code but very slow (subtracted) https://github.com/albertobsd/ecctools
                                                                         https://github.com/albertobsd/keysubtracter
hi share your new publickeys with subtracted code please send

and some example explain >>>>> Total (3.5 Million public keys) at some 400 Gkeys/s

my pc for 80 cpu 256 GP ram, 1TP stroage
 ./keyhunt -m bsgs -f 120.txt -b 120 -S - k 10 >>>>>>>>>> k value i am increaae system bloom filter error why????????
only one public key use 2 petakeys / per seconed ???
how to reached 10 Exabyte keys per second please solve my problem


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on November 13, 2021, 02:57:19 PM
@mausuv: you should post this in albert0bsd's topic: ecctools - a small collection of tools written in C (https://bitcointalk.org/index.php?topic=5361234.0).


Title: Re: List of all Bitcoin addresses with a balance
Post by: DeepComplex on November 28, 2021, 03:32:21 AM
Hi Albert0bsd,

Can you share your dormant-publickeys (3.5 million public keys)?

It would be greatly appreciated.


Well, that is a pity, thanks for the fast reply.

I will stick to my current list of dormats-publickeys with subtracted values  Total (3.5 Million public keys) at some 400 Gkeys/s

Wonderful work you made.

Thanks!

subtracted problem i need update

i am use your code but very slow (subtracted) https://github.com/albertobsd/ecctools
                                                                         https://github.com/albertobsd/keysubtracter
hi share your new publickeys with subtracted code please send

and some example explain >>>>> Total (3.5 Million public keys) at some 400 Gkeys/s

my pc for 80 cpu 256 GP ram, 1TP stroage
 ./keyhunt -m bsgs -f 120.txt -b 120 -S - k 10 >>>>>>>>>> k value i am increaae system bloom filter error why????????
only one public key use 2 petakeys / per seconed ???
how to reached 10 Exabyte keys per second please solve my problem


Title: Re: List of all Bitcoin addresses with a balance
Post by: SFR10 on December 02, 2021, 07:22:14 PM
@LoyceV
Do you know a way to narrow down addresses based on either their tx numbers or the number of txs that seem to have a change address in them [e.g. 2 outputs], by any chance?
- I do know the latter part is a long shot, but I'm dealing with a case that has limited information while the other involved party is ignoring everybody [so far] + I became impatient [had no luck with Blockchair dumps].


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on December 02, 2021, 08:46:21 PM
Do you know a way to narrow down addresses based on either their tx numbers or the number of txs that seem to have a change address in them [e.g. 2 outputs], by any chance?
Nope :(
But even if you get such a list, it's going to be so long it's not going to help you. Most transactions have multiple outputs, so whatever you're looking for, you'll have to narrow it down further.


Title: Re: List of all Bitcoin addresses with a balance
Post by: SFR10 on December 03, 2021, 04:48:20 AM
But even if you get such a list, it's going to be so long it's not going to help you.
You have a point, but since it's an address with a significant number of txs, I thought that might do the trick when it comes to tracking such addys [if there was a way].

Most transactions have multiple outputs, so whatever you're looking for, you'll have to narrow it down further.
Unfortunately, you're right :(


Title: Re: List of all Bitcoin addresses with a balance
Post by: TheArchaeologist on December 14, 2021, 01:12:54 PM
Hi LoyceV since you already have some useful data, did you believe that you can publish some list like "Public keys with balance".

Not OP, but are you looking for something like this:
Code:
address                                  balance         pubkey
12higDjoCCNXSA95xZMWUdPvXNmkAduhWv       0.00010000      044d05240cfbd8a2786eda9dadd520c1609b8593ff8641018d57703d02ba687cf2f187f0cee2221c3afb1b5ff7888caced2423916b61444666ca1216f26181398c
17afxUJouat3fkaaQ9tZrDThxdkXGL4WrM       0.00010000      0428ae82b094ace0e2753b55ade744f65580effe4be3940601cd5a6c51d5af7c5a3747741abcfebf48085010fc7983107da4fbc3dff8b72401779fcfbaa05bd5d3
1PhUXucRd8FzQved2KGK3g1eKfTHPGjgFu       50.00673279     04a2b3590604543737ef6a5cbdf438e44ef66d584efab1565759e1bafc72462d6b5f547466e4cbd610f2dcd65f2c0d6a419cf0170c15d7e783edf7d0247fa74f3b
1DNML6RyHdsqRS7te5SpCkMGoYEkVJU6wD       0.00000890      04ab6c62bcdc3ee4a5ded010007e48c6404347997624af22582147bf4ca7f2b7c3b725d1278f19c87f85652adcc8a198104c233353683bc6c2d1f8149c53c3782b
15J5MZxBkbo88hfHbY96UEDdWWtTJKTBgj       0.00000890      04aeb58ff7e7109ad1d2e8f4f574c733833a8c9c394082cec0f691d63a352ce56d3a1ac62473b62b70b238495808b8c904efff6c02ae9fc97e442f153404b64af2
1EGDbCKGbizMFgwjn4WNHTvAthsvrsixDv       1.00000000      042dfb87e38d92220d20fc181d358dca0f3677f2eabf64425b355deda0ae34a057127b369a80dcb69fb652ddbcb5e91b1c1bd43634be8c00fd1f0c74d3c6b9b8d6
1PyNKEyqtDFB4HBG8ueDVxdPZfgwC6gAqw       0.00001000      046bfe6fa9d6ab1383378cb1f30fb913e9bb18bc7f8cc81aadd2ac70bb5177d4dd73cd7a59373d58738602ba850622e1f470aa8ea45972673bd06b8bdc119f7ef4
1JXFXUBGs2ZtEDAQMdZ3tkCKo38nT2XSEp       0.00001000      04044ee968e705cba538788fd49bdd95ec4b8d96d6fe52c3c0788c4289d6c6ec3305cb603ac45462ffd6093aaddebd0f1201945165d8b0a2b9bb9a70e392aa4936
18FAoviy2XGWRBvDaA1QSUrKuPH2ST8pA2       2.38000000      04d21667e5ab6f05072fec925c9f598af3fcf40aafd31dd89b8994a63f516a7426da3cceb9ff437678dcd02abd8451b51d61075c58f0cc7e072fa71dd8f74b603c
14ccB2QFEKZnd1gLUtgr7vWg6JrMeXaQXK       0.00001000      045551caaca4caa1535fdea7e3dd6646720869e5c0187077aecfeafe60741aa61a35b013eb0f55081e30fc1b3b03bbc39e2430b31073574c39dbe60d122edb8a1b
12u6hecWRHEPceLYZ9yhubZvhgKYuLGf1J       151.00001435    046bd20f8fe764c28f5fe1c4eba69d26e0af42bf1853213d593a5098e075344d6fdc998e9c1e469cad7d799a0891f3bc47ae1f6a1eb6caabd62890a7994f4b6d5f
1XPTgDRhN8RFnzniWCddobD9iKZatrvH4        0.00015529      0434417dd8d89deaf0f6481c2c160d6de0921624ef7b956f38eef9ed4a64e36877be84b77cdee5a8d92b7d93694f89c3011bf1cbdf4fd7d8ca13b58a7bb4ab0804
1BiDsroMqEwDhBmzkKLvCSAddGn8QGmLFf       0.00001000      040bc7100ef382290a250654c0b77841ea844b5ad09843f55bf79fe6bb3a320f178a899280cf6f0a92befe316d922bffc3e8d6f93a8f1b788ac18c7ee44cf2f15a
1JCjFJckVFvN5EFs7oUmzKjPVLq8XyPiwc       10.00000000     04200af342e1bb4b50ada09233ebdca598446d4c7c3c92c915b3ec4fa36ef9c81937a204fe1909509d5b5b429ab00e378a3a1bf7e20e783974e0b1d5263c3bd38b
15Z5YJaaNSxeynvr6uW6jQZLwq3n1Hu6RX       7941.06426123   047a51392bace353f4c3788c9c090ef4f635ec211159ec3b9f1bb7da7679517e126e98e0012bcb4d2b023c479afaaa1ad703ea1b24e1910e2cdad38744ba7aab8a
1JiZxcwHpivk35obqHUYbdNoE1BA8ty5TB       5.00000000      04cb9648aa9f91795d111ded927f1a175c44a53537c817da4616eb071a9ec83a2670430628d20e70ee7a45b23494fc217d1020784d4d56b2f932754d79e4c3b808


Title: Re: List of all Bitcoin addresses with a balance
Post by: albert0bsd on December 16, 2021, 04:44:14 PM
Not OP, but are you looking for something like this:

Yes something like that, i already have some lists, but none of them are a exhaustive list or even updated

Regards


Title: Re: List of all Bitcoin addresses with a balance
Post by: bitxchange on December 29, 2021, 05:14:17 AM
You deserve praise for making the list available and keeping updates running.

Have fun converting base58 and bech32 addresses to hash160, python style:

Code:
import segwit_addr
import base58
import argparse
import sys
import hashlib
import binascii

parser = argparse.ArgumentParser()
parser.add_argument('address', help='Enter bech32 address')
args = parser.parse_args()

address = sys.argv[1]

def validate_base58(a):
    base58Decoder = base58.b58decode(a).hex()
    prefixAndHash = base58Decoder[:len(base58Decoder)-8]
    checksum = base58Decoder[len(base58Decoder)-8:]
    hash = prefixAndHash
    for x in range(1,3):
        hash = hashlib.sha256(binascii.unhexlify(hash)).hexdigest()
        #print("Hash#", x, " : ", hash)
    #print("--------------------------------------")
    if(checksum == hash[:8]):
        return True
    else:
        return False

if validate_base58(address) is True:
    base58_decoded = base58.b58decode_check(address).hex()[2:]
    print (base58_decoded)
else:
    decoded = segwit_addr.decode('bc', address)
    if decoded[0] == 0:
        #print(decoded[2])
        hash160_bin = []
        for x in decoded[1]:
            hash160_bin.append(f'{x:08b}')

        hash160_hex = []
        for x in hash160_bin:
            hash160_hex.append(f'{int(x, 2):X}')

        print(''.join(hash160_hex))

and running it:

Code:
# python3 segwit_addr_decode.py 1P5ZEDWTKTFGxQjZphgWPQUpe554WKDfHQ
f22f5563839ba6ba5aa8d3726fcbc675cb3e4c9e

# python3 segwit_addr_decode.py 34xp4vRoCGJym3xR7yCVPFHoCNxv4Twseo
23e522dfc6656a8fda3d47b4fa53f7585ac758cd

# python3 segwit_addr_decode.py bc1qxv55wuzz4qsfgss3uq2zwg5y88d7qv5hg67d2d
3329477042A820944211E01427228439DBE03297

useful information on bech32 encoding: https://en.bitcoin.it/wiki/Bech32
test segwit address: https://bitcoin.sipa.be/bech32/demo/demo.html
test base58 encoded address: http://lenschulwitz.com/base58


Title: Re: List of all Bitcoin addresses with a balance
Post by: majika on January 01, 2022, 12:13:04 PM
Hi LoyceV

I can see the utility of having the data in this format..
However call me stupid BUT a far simpler way would be to parse data directly from your own node?

> Dump out in raw addresses which have a positive int, then parse/query. Or am I missing something here??
I suppose another method would be to check for DATES where pub_key was involved in OUTPUT. to figure out addresses with positive balance.
I'm sure there is a far more elegant solution to the problem, with which a little touch of Python could go a long way to solving the BTC address > balance question.

PS. What about a "BTC Address Rich List" This is fundamentally based on a similar principle as what your wishing to achieve, is it not?
:)


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on January 01, 2022, 12:46:50 PM
However call me stupid BUT a far simpler way would be to parse data directly from your own node?
As far as I know, that's not so simple. But if you can figure it out, I'd love to see your results. I expect it to be quite slow.

Quote
PS. What about a "BTC Address Rich List" This is fundamentally based on a similar principle as what your wishing to achieve, is it not?
The list with balance is a "Rich List":
Code:
wget -qO- http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz | gunzip | grep -vP "address\tbalance" | head -n 50
34xp4vRoCGJym3xR7yCVPFHoCNxv4Twseo      26547968732321
bc1qgdjqv0av3q56jvd82tkdjpy7gdp9ut8tlqmgrpmv24sq90ecnvqqjwvw97  16800998536372
1P5ZEDWTKTFGxQjZphgWPQUpe554WKDfHQ      12038957274655
37XuVSEpWW4trkfmvWzegTHQt7BdktSKUs      9450511456719
38UmuUqPCrFmQo4khkomQwZ4VbY2nZMJ67      8406701772536
1FeexV6bAHb8ybZjqQMjJrcCrHGW9sb6uF      7995721697629
3Kzh9qAqVWQhEsfQz7zEQL1EuSx5tyNLNS      7642003887352
3LYJfcfHPXYJreMsASk2jkn69LWEYKzexb      7360113647202
3M219KR5vEneNb47ewrPfWyb5jQ2DjxRP6      7143400000000
bc1qa5wkgaew2dkv56kfvj49j0av5nml45x9ek9hz6      6937017585302
1LdRcdxfbSnmCYYNdeYpUnztiYzVfBEQeC      5388006072455
1AC4fMwgY8j9onSbXEWeH6Zan8QGMSdmtA      5183035998549
3FpYfDGJSdkMAvZvCrwPHDqdmGqUkTsJys      4579246787700
1LruNZjwamWJXThX2Y8C2d47QqhAkkc5os      4400008617726
3Gpex6g5FPmYWm26myFq7dW12ntd8zMcCY      4001396038553
bc1q5shngj24323nsrmxv99st02na6srekfctt30ch      4000032768183
385cR5DM96n1HvBDMzLHPYcw89fZAXULJP      3860186558087
3LQUu4v9z6KNch71j7kbj8GPeAGUo1FW6a      3792697744959
bc1q7ydrtdn8z62xhslqyqtyt38mm4e2c4h3mxjkug      3600041000054
159QgP4Ewvadjc4HDpaaR6pir2R4ZfzVfQ      3441500117745
bc1q5pucatprjrqltdp58f92mhqkfuvwpa43vhsjwpxlryude0plzyhqjkqazp  3369998788200
3LCGsSmfr24demGvriN4e3ft8wEcDuHFqh      3322004105156
12XqeqZRVkBDgmPLVY4ZC6Y4ruUUEug8Fx      3235400000564
bc1qx9t2l3pyny2spqpqlye8svce70nppwtaxwdrp4      3141523355441
3FHNBLobJnbCTFTVakh5TXmEneyf5PT61B      3127534535966
12ib7dApVFvg82TXKycWBNpN8kFyiAN1dr      3100007042388
3FupZp77ySr7jwoLYEJ9mwzJpvoNBXsBnE      2949754539921
12tkqA9xSoowkzoERHMWNKsTey55YEBqkv      2815105848732
3JJpCZCk4h4TpQeU7SA1yhH768Xgbtdbfg      2660550102022
1ftuUgzrr5hnVzXnFBDjELvVe69rPtBRi       2589500002000
3JZq4atUahhuA9rLhXLMhhTo133J9rF97j      2550364890459
17MWdxfjPYP2PYhdy885QtihfbW181r1rn      2449535042970
35ULMyVnFoYaPaMxwHTRmaGdABpAThM4QR      2430500118088
19D5J8c59P2bAkWKvxSYw8scD3KUNWoZ1C      2396900153670
1aXzEKiDJKzkPxTZy9zGc3y1nCDwDPub2       2363051758452
bc1qjysjfd9t9aspttpjqzv68k0ydpe7pvyd5vlyn37868473lell5tqkz456m  2328178254950
1932eKraQ3Ad9MeNBHb14WFQbNrLaKeEpT      2289999043553
19iqYbeATe4RxghQZJnYVFU4mjUUu76EA6      2243324021699
1MDq7zyLw6oKichbFiDDZ3aaK59byc6CT8      2240000017757
bc1qd4ysezhmypwty5dnw7c8nqy5h5nxg0xqsvaefd0qn5kq32vwnwqqgv4rzr  2200000010000
3DVJfEsDTPkGDvqPCLC41X85L1B1DQWDyh      2040626414582
3HSMPBUuAPQf6CU5B3qa6fALrrZXswHaF1      2020488810100
bc1q4jchcr7nla277su5lpjzttxp3xg5j8wds5lcwg      2017184466165
bc1qtw30nantkrh7y5ue73gm4mmy0zezfqxug3psr94sd967qwg7f76scfmr9p  2013623538292
17rm2dvb439dZqyMe2d4D6AQJSgg6yeNRn      2000800192617
1PeizMg76Cf96nUQrYg8xuoZWLQozU5zGW      1941443070193
3K5dmrkBMS8ZVgERMLwiw7PJuG8GWTbo8e      1860004653628
bc1qvpgyac88vqtslewxu7yu9dqwp8rd83zch55zpm3xgn3mgg72w3kqv0s8qa  1792998389600
1AjPJ1eq4VMt3NxBZwAD5BvpajxoGUJ89n      1727771271755
15HEcdBfaDD9tUxfPxgiPbTJi6ooQb11B2      1681996864863


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on January 05, 2022, 06:36:29 PM
I have been monitoring the total number of funded addresses here  (https://bitcoindata.science/bitcoin-funded-addresses.html), with the data provided by LoyceV here (http://addresses.loyce.club/total_number_of_funded_addresses.json)
Time to add Taproot addresses:
I know that Taproot addresses are starting with bc1p, and Native Sewgit addresses start with bc1q.
Starting today, I've added bc1p-addresses to total_number_of_funded_addresses.txt (http://addresses.loyce.club/total_number_of_funded_addresses.txt) and total_number_of_funded_addresses.json (http://addresses.loyce.club/total_number_of_funded_addresses.json):
Code:
"03/01/2022":[
{"total": 39644562,
  "1": 23810601,
  "3": 8474856,
  "bc1q": 7342678}],
"04/01/2022":[
{"total": 39631173,
  "1": 23822116,
  "3": 8429494,
  "bc1q": 7362835}],
"05/01/2022":[
{"total": 39633563,
  "1": 23822618,
  "3": 8421989,
  "bc1q": 7371925,
  "bc1p": 17027}]
}
I'm curious to see how fast those numbers will go up.


Title: Re: List of all Bitcoin addresses with a balance
Post by: mynonce on January 05, 2022, 07:26:44 PM
I like what you're doing, but only a proposal:

Code:
"05/01/2022":[
{"total": 39633563, 100.00%,
 "1": 23822618, 60.11%,
 "3": 8421989, 21.25%,
 "bc1q": 7371925, 18.60%
 "bc1p": 17027, 0.04%}]

Maybe with percentage?  :)


Title: Re: List of all Bitcoin addresses with a balance
Post by: bitmover on January 05, 2022, 08:03:20 PM
I like what you're doing, but only a proposal:

Code:
"05/01/2022":[
{"total": 39633563, 100.00%,
 "1": 23822618, 60.11%,
 "3": 8421989, 21.25%,
 "bc1q": 7371925, 18.60%
 "bc1p": 17027, 0.04%}]

Maybe with percentage?  :)

I am doing this in the front end, no need to add this information in the data itself. I think the data should e as raw as possible.
You can see the percentage here
https://bitcoindata.science/bitcoin-funded-addresses.html


Time to add Taproot addresses:
I know that Taproot addresses are starting with bc1p, and Native Sewgit addresses start with bc1q.
Starting today, I've added bc1p-addresses to total_number_of_funded_addresses.txt (http://addresses.loyce.club/total_number_of_funded_addresses.txt) and total_number_of_funded_addresses.json (http://addresses.loyce.club/total_number_of_funded_addresses.json):
Code:
"03/01/2022":[
{"total": 39644562,
 "1": 23810601,
 "3": 8474856,
 "bc1q": 7342678}],
"04/01/2022":[
{"total": 39631173,
 "1": 23822116,
 "3": 8429494,
 "bc1q": 7362835}],
"05/01/2022":[
{"total": 39633563,
 "1": 23822618,
 "3": 8421989,
 "bc1q": 7371925,
 "bc1p": 17027}]
}
I'm curious to see how fast those numbers will go up.

I will add this by tomorrow
 Thanks for the heads-up


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on January 05, 2022, 09:46:25 PM
Maybe with percentage?  :)
I created the json-file for bitmover, and I prefer his solution:
I am doing this in the front end, no need to add this information in the data itself. I think the data should e as raw as possible.
You can see the percentage here https://bitcoindata.science/bitcoin-funded-addresses.html


Title: Re: List of all Bitcoin addresses with a balance
Post by: mynonce on January 05, 2022, 10:32:03 PM
I am doing this in the front end, no need to add this information in the data itself. I think the data should e as raw as possible.
Maybe with percentage?  :)
I created the json-file for bitmover, and I prefer his solution:
I am doing this in the front end, no need to add this information in the data itself. I think the data should e as raw as possible.
You can see the percentage here https://bitcoindata.science/bitcoin-funded-addresses.html
Thanks.
I didn't know this site https://bitcoindata.science/bitcoin-funded-addresses.html (https://bitcoindata.science/bitcoin-funded-addresses.html)


Title: Re: List of all Bitcoin addresses with a balance
Post by: bitmover on January 06, 2022, 01:31:57 PM
Maybe with percentage?  :)
I created the json-file for bitmover, and I prefer his solution:
I am doing this in the front end, no need to add this information in the data itself. I think the data should e as raw as possible.
You can see the percentage here https://bitcoindata.science/bitcoin-funded-addresses.html

just added the taproot addresses to the front end:

https://i.imgur.com/Ai1YXyF.png

The problem is that the numbers are still very small. Let's see how fast those numbers will grow so we can actually see those addresses in the charts.


Title: Re: List of all Bitcoin addresses with a balance
Post by: mynonce on January 06, 2022, 09:36:46 PM
@LoyceV, @bitmover
Have you ever looked for mined but unmoved Bitcoin? Would be interesting to know how many there are.


Title: Re: List of all Bitcoin addresses with a balance
Post by: bitmover on January 07, 2022, 01:40:45 AM
@LoyceV, @bitmover
Have you ever looked for mined but unmoved Bitcoin? Would be interesting to know how many there are.

I just made a search here:
https://blockchair.com/bitcoin/outputs?s=time(desc)&q=is_from_coinbase(true),is_spendable(true)#f=is_from_coinbase,time,is_spendable

According to blockchair.com, there are 196,479 outputs of coinbase transactions which are spendable. I.e, they are unmoved.


Title: Re: List of all Bitcoin addresses with a balance
Post by: mynonce on January 07, 2022, 02:10:14 AM
@LoyceV, @bitmover
Have you ever looked for mined but unmoved Bitcoin? Would be interesting to know how many there are.
I just made a search here:
https://blockchair.com/bitcoin/outputs?s=time(desc)&q=is_from_coinbase(true),is_spendable(true)#f=is_from_coinbase,time,is_spendable
According to blockchair.com, there are 196,479 outputs of coinbase transactions which are spendable. I.e, they are unmoved.

We have block 0 in that list  ??? But block 0 output is unspendable.

These are the number of outputs but not how many Bitcoin. It should be > 1 million Bitcoin (at least)


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on January 07, 2022, 07:42:32 AM
Have you ever looked for mined but unmoved Bitcoin? Would be interesting to know how many there are.
You can probably get this from Bitcoin block data (550 GB): inputs, outputs and transactions (https://bitcointalk.org/index.php?topic=5307550.0), but I still don't have a(n affordable) webhost for this data, and https://gz.blockchair.com/ has been offline for a few days.
It would be interesting to create an all-time graph, showing how the number of unspent new Bitcoins changes since the creation of Bitcoin. But I'll skip this one: too much data to process.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on January 11, 2022, 03:38:52 PM
My LATEST version (http://addresses.loyce.club/) is currently stuck at January 5, 2022. It turns out Blockchair (https://gz.blockchair.com/bitcoin/addresses/) hasn't updated their archives in the last 6 days, and before that their archives were offline for a few days. I'm hoping it will be back online eventually.


Title: Re: List of all Bitcoin addresses with a balance
Post by: PawGo on January 11, 2022, 04:01:52 PM
My LATEST version (http://addresses.loyce.club/) is currently stuck at January 5, 2022. It turns out Blockchair (https://gz.blockchair.com/bitcoin/addresses/) hasn't updated their archives in the last 6 days, and before that their archives were offline for a few days. I'm hoping it will be back online eventually.

They informed about troubles, AFAIK something related to situation in Kazakhstan. Archive pages were completely inaccessible recently, now they are back, but with old data.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on January 11, 2022, 04:08:15 PM
situation in Kazakhstan.
What are the odds they host this in Kazakhstan O0


Title: Re: List of all Bitcoin addresses with a balance
Post by: PawGo on January 11, 2022, 04:23:32 PM
situation in Kazakhstan.
What are the odds they host this in Kazakhstan O0

https://status.blockchair.com/
Quote
General status: Some of our servers are currently unavailable due to a global Internet shutdown in Kazakhstan: https://twitter.com/disclosetv/status/1478694856402817026. We're working on activating our reserve infrastructure. Some historical Ethereum transactions may be missing. Situation updates will be available at https://status.blockchair.com/ and at https://twitter.com/Blockchair/status/1478698203792949249


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on January 25, 2022, 09:49:33 AM
Blockchair's problems have long been resolved.

I'm closing in on another potential problem: this VPS might run out of bandwidth (2TB) before the end of this month. If that happens it will get suspended, and be back online next month. This hosting package doesn't allow adding more bandwidth, and I've paid this server until November 2023, so I'm not switching elsewhere any time soon.

I'm glad to see this data is in high demand though:
https://loyce.club/other/bandwidth.gif


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on February 05, 2022, 09:20:02 AM
Last month, I killed my webserver for the last days to stop my VPS from being suspended. I turned it back on this month, but Blockchair (https://gz.blockchair.com/bitcoin/addresses/) hasn't updated their data dumps yet this month so you're all downloading the same version over and over again :P


Title: Re: List of all Bitcoin addresses with a balance
Post by: PawGo on February 05, 2022, 09:26:58 AM
Last month, I killed my webserver for the last days to stop my VPS from being suspended. I turned it back on this month, but Blockchair (https://gz.blockchair.com/bitcoin/addresses/) hasn't updated their data dumps yet this month so you're all downloading the same version over and over again :P

https://www.youtube.com/watch?v=JKDo6g4CCeU

If you want to keep "fixed filename", maybe add one more, small file with "update date" inside.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on February 05, 2022, 09:53:02 AM
If you want to keep "fixed filename", maybe add one more, small file with "update date" inside.
The update date is in the file's time stamp. It used to be easy to see, but since I added an index.html it doesn't show anymore.
I had to search a bit, Super User.com (https://superuser.com/questions/619592/get-modification-time-of-remote-file-over-http-in-bash-script) gave this command:
Code:
wget --server-response --spider  http://addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz 2>&1 | grep -i Last-Modified
  Last-Modified: Mon, 31 Jan 2022 00:57:38 GMT


Title: Re: List of all Bitcoin addresses with a balance
Post by: PawGo on February 05, 2022, 11:42:30 AM
If you want to keep "fixed filename", maybe add one more, small file with "update date" inside.
The update date is in the file's time stamp. It used to be easy to see, but since I added an index.html it doesn't show anymore.
I had to search a bit, Super User.com (https://superuser.com/questions/619592/get-modification-time-of-remote-file-over-http-in-bash-script) gave this command:
Code:
wget --server-response --spider  http://addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz 2>&1 | grep -i Last-Modified
  Last-Modified: Mon, 31 Jan 2022 00:57:38 GMT

Too technical. But maybe will be enough. You should check in accessLog how many requests you have from curl/wget type tools and how many from normal browser.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on February 05, 2022, 01:19:06 PM
You should check in accessLog how many requests you have from curl/wget type tools and how many from normal browser.
I never look at logs. Now that I did, it turns out logs don't get rotated, so it goes back more than a year.
Some counts:
Code:
"mozill": 688k
"chrom": 539k
"wget": 415k
I've wiped the logs now.

I assume people who use a normal browser will see the dates typed on addresses.loyce.club/ (http://addresses.loyce.club/).


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on March 19, 2022, 12:51:55 PM
This project has already used up 1.5 out of it's 2 TB bandwidth for this month. To prevent the server from being suspended, I've scheduled to take down the Apache webserver in 5 days. I'll restart it on April 1st.

It's now the second time I have to take this precaution. I've paid this webhost for the next 1.5 years, and I'm at the maximum bandwidth for this package already. I don't really want to upgrade to another server yet (I use 4 different servers for 4 different projects).
If anyone's in urgent need for this data when it's offline: post here :)


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on March 22, 2022, 09:49:48 AM
This project has already used up 1.5 out of it's 2 TB bandwidth for this month. To prevent the server from being suspended, I've scheduled to take down the Apache webserver in 5 days. I'll restart it on April 1st.
The above post opened a new door: an anonymous Bitcointalk reader donated a dedicated server for this project! That means lots of storage (1 TB), lots of bandwidth (50 TB/month), and potentially more than once a day updates by parsing the Bitcoin blockchain instead of getting data from Blockchair.com!
I'm still setting it up now.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on March 26, 2022, 01:14:04 PM
Update: the new server is online, at the old location: addresses.loyce.club (https://addresses.loyce.club).
Because of the much larger disk space, I'll now keep more daily and monthly snapshots. It will take a couple of months to collect them all.
Enjoy the bandwidth :)


Title: Re: List of all Bitcoin addresses with a balance
Post by: PawGo on March 27, 2022, 09:19:10 AM
The above post opened a new door: an anonymous Bitcointalk reader donated a dedicated server for this project! That means lots of storage (1 TB), lots of bandwidth (50 TB/month), and potentially more than once a day updates by parsing the Bitcoin blockchain instead of getting data from Blockchair.com!
I'm still setting it up now.

WOW, congratulations!
Open way to mine Monero on free cpu ;-)
Do you plan to run a full-node (with incoming connections) and maybe electrum server?


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on March 27, 2022, 01:50:21 PM
Open way to mine Monero on free cpu ;-)
It can produce $0.06 per day. I'm not turning electricity into heat for that, even if someone else pays for it.

Quote
Do you plan to run a full-node (with incoming connections) and maybe electrum server?
I've already almost filled it's disk, so that won't work.


Title: Re: List of all Bitcoin addresses with a balance
Post by: mrx23dot on April 13, 2022, 09:28:54 PM
Hello,
What's the RPC command to do the dump on local full node?

Is it possible to use remote RPC server for it, like electrum?

Cheers


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on April 14, 2022, 11:36:42 AM
What's the RPC command to do the dump on local full node?
You may be interested in Bitcoin UTXO Dump (https://github.com/in3rsha/bitcoin-utxo-dump).
I haven't had the time yet to test it (use at your own risk).


Title: Re: List of all Bitcoin addresses with a balance
Post by: PawGo on April 14, 2022, 08:19:36 PM
Hello,
What's the RPC command to do the dump on local full node?

It is not RPC way, but I was using rusty-blockparser for doing dumps from local node. It uses local files, not rpc connection.
https://github.com/gcarq/rusty-blockparser


Title: Re: List of all Bitcoin addresses with a balance
Post by: DeepComplex on April 15, 2022, 04:16:46 AM
How could I generate a dump like this from the blockchain? Any script available?

Hi LoyceV since you already have some useful data, did you believe that you can publish some list like "Public keys with balance".

Not OP, but are you looking for something like this:
Code:
address                                  balance         pubkey
12higDjoCCNXSA95xZMWUdPvXNmkAduhWv       0.00010000      044d05240cfbd8a2786eda9dadd520c1609b8593ff8641018d57703d02ba687cf2f187f0cee2221c3afb1b5ff7888caced2423916b61444666ca1216f26181398c
17afxUJouat3fkaaQ9tZrDThxdkXGL4WrM       0.00010000      0428ae82b094ace0e2753b55ade744f65580effe4be3940601cd5a6c51d5af7c5a3747741abcfebf48085010fc7983107da4fbc3dff8b72401779fcfbaa05bd5d3
1PhUXucRd8FzQved2KGK3g1eKfTHPGjgFu       50.00673279     04a2b3590604543737ef6a5cbdf438e44ef66d584efab1565759e1bafc72462d6b5f547466e4cbd610f2dcd65f2c0d6a419cf0170c15d7e783edf7d0247fa74f3b
1DNML6RyHdsqRS7te5SpCkMGoYEkVJU6wD       0.00000890      04ab6c62bcdc3ee4a5ded010007e48c6404347997624af22582147bf4ca7f2b7c3b725d1278f19c87f85652adcc8a198104c233353683bc6c2d1f8149c53c3782b
15J5MZxBkbo88hfHbY96UEDdWWtTJKTBgj       0.00000890      04aeb58ff7e7109ad1d2e8f4f574c733833a8c9c394082cec0f691d63a352ce56d3a1ac62473b62b70b238495808b8c904efff6c02ae9fc97e442f153404b64af2
1EGDbCKGbizMFgwjn4WNHTvAthsvrsixDv       1.00000000      042dfb87e38d92220d20fc181d358dca0f3677f2eabf64425b355deda0ae34a057127b369a80dcb69fb652ddbcb5e91b1c1bd43634be8c00fd1f0c74d3c6b9b8d6
1PyNKEyqtDFB4HBG8ueDVxdPZfgwC6gAqw       0.00001000      046bfe6fa9d6ab1383378cb1f30fb913e9bb18bc7f8cc81aadd2ac70bb5177d4dd73cd7a59373d58738602ba850622e1f470aa8ea45972673bd06b8bdc119f7ef4
1JXFXUBGs2ZtEDAQMdZ3tkCKo38nT2XSEp       0.00001000      04044ee968e705cba538788fd49bdd95ec4b8d96d6fe52c3c0788c4289d6c6ec3305cb603ac45462ffd6093aaddebd0f1201945165d8b0a2b9bb9a70e392aa4936
18FAoviy2XGWRBvDaA1QSUrKuPH2ST8pA2       2.38000000      04d21667e5ab6f05072fec925c9f598af3fcf40aafd31dd89b8994a63f516a7426da3cceb9ff437678dcd02abd8451b51d61075c58f0cc7e072fa71dd8f74b603c
14ccB2QFEKZnd1gLUtgr7vWg6JrMeXaQXK       0.00001000      045551caaca4caa1535fdea7e3dd6646720869e5c0187077aecfeafe60741aa61a35b013eb0f55081e30fc1b3b03bbc39e2430b31073574c39dbe60d122edb8a1b
12u6hecWRHEPceLYZ9yhubZvhgKYuLGf1J       151.00001435    046bd20f8fe764c28f5fe1c4eba69d26e0af42bf1853213d593a5098e075344d6fdc998e9c1e469cad7d799a0891f3bc47ae1f6a1eb6caabd62890a7994f4b6d5f
1XPTgDRhN8RFnzniWCddobD9iKZatrvH4        0.00015529      0434417dd8d89deaf0f6481c2c160d6de0921624ef7b956f38eef9ed4a64e36877be84b77cdee5a8d92b7d93694f89c3011bf1cbdf4fd7d8ca13b58a7bb4ab0804
1BiDsroMqEwDhBmzkKLvCSAddGn8QGmLFf       0.00001000      040bc7100ef382290a250654c0b77841ea844b5ad09843f55bf79fe6bb3a320f178a899280cf6f0a92befe316d922bffc3e8d6f93a8f1b788ac18c7ee44cf2f15a
1JCjFJckVFvN5EFs7oUmzKjPVLq8XyPiwc       10.00000000     04200af342e1bb4b50ada09233ebdca598446d4c7c3c92c915b3ec4fa36ef9c81937a204fe1909509d5b5b429ab00e378a3a1bf7e20e783974e0b1d5263c3bd38b
15Z5YJaaNSxeynvr6uW6jQZLwq3n1Hu6RX       7941.06426123   047a51392bace353f4c3788c9c090ef4f635ec211159ec3b9f1bb7da7679517e126e98e0012bcb4d2b023c479afaaa1ad703ea1b24e1910e2cdad38744ba7aab8a
1JiZxcwHpivk35obqHUYbdNoE1BA8ty5TB       5.00000000      04cb9648aa9f91795d111ded927f1a175c44a53537c817da4616eb071a9ec83a2670430628d20e70ee7a45b23494fc217d1020784d4d56b2f932754d79e4c3b808


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on April 15, 2022, 04:35:11 PM
How could I generate a dump like this from the blockchain?
I don't know how to do this. I don't think this data (https://bitcointalk.org/index.php?topic=5307550.0) has the information, so you'll really have to get it from your own blockchain. If rusty-blockparser (see 2 posts up) can do it, I might be able to add it to my daily updates. Let me know if you find a way, I'm not checking it myself.


Title: Re: List of all Bitcoin addresses with a balance
Post by: DeepComplex on April 15, 2022, 04:53:15 PM
Hi @LoyceV,

I've perused your dump and indeed that info cannot be generated from them. Nevertheless, your blockchain dumps are cool and useful. Thanks again for that!

I was hoping that @albert0bsd or @TheArchaeologist could help with a script that can parse the blockchain.

I'll check the rusty-blockparser. If I do get something I'll share the same here.

Regards,

How could I generate a dump like this from the blockchain?
I don't know how to do this. I don't think this data (https://bitcointalk.org/index.php?topic=5307550.0) has the information, so you'll really have to get it from your own blockchain. If rusty-blockparser (see 2 posts up) can do it, I might be able to add it to my daily updates. Let me know if you find a way, I'm not checking it myself.


Title: Re: List of all Bitcoin addresses with a balance
Post by: DeepComplex on April 16, 2022, 01:30:25 AM
@LoyceV,

The only method I've found that gives me something close to the aforementioned is on Blockchair.com (https://blockchair.com/bitcoin/outputs?s=block_id(asc)&q=is_spendable(true),value(100000000..),is_spent(false),time(2009-01-01..),block_id(122..201865)#f=recipient,block_id,is_spent,type,is_spendable,script_hex,value,is_from_coinbase,time). Can you export that dataset to a  tsv/csv  file? The option "Get this dataset in .tsv format" on the site takes you to the Database Dumps (https://blockchair.com/dumps).

I have no idea how to extract the data in that format from the outputs dump files. If you know some linux commands to parse them and give a similar output, that would be appreciated.

Regards,

Hi @LoyceV,

I've perused your dump and indeed that info cannot be generated from them. Nevertheless, your blockchain dumps are cool and useful. Thanks again for that!

I was hoping that @albert0bsd or @TheArchaeologist could help with a script that can parse the blockchain.

I'll check the rusty-blockparser. If I do get something I'll share the same here.

Regards,

How could I generate a dump like this from the blockchain?
I don't know how to do this. I don't think this data (https://bitcointalk.org/index.php?topic=5307550.0) has the information, so you'll really have to get it from your own blockchain. If rusty-blockparser (see 2 posts up) can do it, I might be able to add it to my daily updates. Let me know if you find a way, I'm not checking it myself.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on April 16, 2022, 08:02:34 AM
The only method I've found that gives me something close to the aforementioned is on Blockchair.com (https://blockchair.com/bitcoin/outputs?s=block_id(asc)&q=is_spendable(true),value(100000000..),is_spent(false),time(2009-01-01..),block_id(122..201865)#f=recipient,block_id,is_spent,type,is_spendable,script_hex,value,is_from_coinbase,time). Can you export that dataset to a  tsv/csv  file? The option "Get this dataset in .tsv format" on the site takes you to the Database Dumps (https://blockchair.com/dumps).
I don't think I can easily get this data, other than clicking "Load more" 3000 times.
But even if you get everything: the address I checked hasn't sent funds yet, so if you're looking for the pubkey, it's not what you need, right?

Quote
I have no idea how to extract the data in that format from the outputs dump files. If you know some linux commands to parse them and give a similar output, that would be appreciated.
If you tell me exactly which part of the data dump you need, I can quite easily get it for you. It just takes a few hours to process all the data. Or I'll get you a short script that loops through the data, but to use it you'll need to download everything first.

If I take a random output (https://gz.blockchair.com/bitcoin/outputs/blockchair_bitcoin_outputs_20111219.tsv.gz), it has this data:
Code:
block_id transaction_hash index time value value_usd recipient type script_hex is_from_coinbase is_spendable
158078 968937595d4777356aa3dbd61148c4ddcf0aef44e6295a479731f34998ee392c 0 2011-12-19 00:08:29 5008750000 160.28 1J6iKnrn9RPM6D23VZ6o1z5muVLUjJ9NuT pubkey 4104eacd3bfaf2d76524763c4416db596cc26e4ff20c13f12ac7d3cc5b5cec3ec88b53ff35b349fb47421976facdc25e63ea365416c3a4ace7acb7884caa6a6d7a52ac 1 1
158078 5c717d331dae3919337f11722e0e3f62b8c27230ee154f4c6c67b36a60bc4dd5 0 2011-12-19 00:08:29 1000000 0.032 1KSiuy6DeeDefyj2Ls8mnMvhRPb25xs6jx pubkeyhash 76a914ca4fa68a872fb740f16ca1caaa414bd989362aa788ac 0 -1
158078 5c717d331dae3919337f11722e0e3f62b8c27230ee154f4c6c67b36a60bc4dd5 1 2011-12-19 00:08:29 119721000000 3831.072 1LaN6ooQJpCqC5nFb5TCinceDkCc24eetR pubkeyhash 76a914d6b9bae958a79b931072b93a52e632f2464ae37e88ac 0 -1
158078 1c4e4dfa580566cb92ec3eb43e3650e542ba1ce7a35415b671f59794da702de4 0 2011-12-19 00:08:29 78812408397 2521.9971 1HT5UX8eSiT65fUZSskZYJ4nPPPL58Rzic pubkeyhash 76a914b4709009d5e5844f643b02ecafa7d55d7acb3f7d88ac 0 -1
158078 1c4e4dfa580566cb92ec3eb43e3650e542ba1ce7a35415b671f59794da702de4 1 2011-12-19 00:08:29 100000000 3.2 17gN64BPHtxi4mEM3qWrxdwhieUvRq8R2r pubkeyhash 76a914494294730abf03c846988654f15d1864469c737a88ac 0 -1
158078 bc9822da9fb69f43f1fe33682cd8007ad247deb75f5a6edb641751e456e18b93 0 2011-12-19 00:08:29 3578400000 114.5088 1KDE7sAtyhzJnLXcmJmAMYN9Xk3MRLsDxT pubkeyhash 76a914c7c234d085dd3d5dbc342d59cca174335f554fa988ac 0 -1
Would you be looking for "4104eacd3bfaf2d76524763c4416db596cc26e4ff20c13f12ac7d3cc5b5cec3ec88b53ff35b349f b47421976facdc25e63ea365416c3a4ace7acb7884caa6a6d7a52ac"?


Title: Re: List of all Bitcoin addresses with a balance
Post by: DeepComplex on April 16, 2022, 03:07:16 PM
Hi LoyceV,

I'm looking for all of the entries with script-type pubkey (P2PK) only and is unspent. Primarly between blocks 0 to 278,298. It is about ~34,300 entries. The fields I would like are: block_id    transaction_hash   time   value   value_usd   recipient   type   script_hex   is_from_coinbase is_spendable.

This is what I ultimately need.
Quote
Would you be looking for "4104eacd3bfaf2d76524763c4416db596cc26e4ff20c13f12ac7d3cc5b5cec3ec88b53ff35b349f b47421976facdc25e63ea365416c3a4ace7acb7884caa6a6d7a52ac"?

Thank you for responding!

Regards,

The only method I've found that gives me something close to the aforementioned is on Blockchair.com (https://blockchair.com/bitcoin/outputs?s=block_id(asc)&q=is_spendable(true),value(100000000..),is_spent(false),time(2009-01-01..),block_id(122..201865)#f=recipient,block_id,is_spent,type,is_spendable,script_hex,value,is_from_coinbase,time). Can you export that dataset to a  tsv/csv  file? The option "Get this dataset in .tsv format" on the site takes you to the Database Dumps (https://blockchair.com/dumps).
I don't think I can easily get this data, other than clicking "Load more" 3000 times.
But even if you get everything: the address I checked hasn't sent funds yet, so if you're looking for the pubkey, it's not what you need, right?

Quote
I have no idea how to extract the data in that format from the outputs dump files. If you know some linux commands to parse them and give a similar output, that would be appreciated.
If you tell me exactly which part of the data dump you need, I can quite easily get it for you. It just takes a few hours to process all the data. Or I'll get you a short script that loops through the data, but to use it you'll need to download everything first.

If I take a random output (https://gz.blockchair.com/bitcoin/outputs/blockchair_bitcoin_outputs_20111219.tsv.gz), it has this data:
Code:
block_id transaction_hash index time value value_usd recipient type script_hex is_from_coinbase is_spendable
158078 968937595d4777356aa3dbd61148c4ddcf0aef44e6295a479731f34998ee392c 0 2011-12-19 00:08:29 5008750000 160.28 1J6iKnrn9RPM6D23VZ6o1z5muVLUjJ9NuT pubkey 4104eacd3bfaf2d76524763c4416db596cc26e4ff20c13f12ac7d3cc5b5cec3ec88b53ff35b349fb47421976facdc25e63ea365416c3a4ace7acb7884caa6a6d7a52ac 1 1
158078 5c717d331dae3919337f11722e0e3f62b8c27230ee154f4c6c67b36a60bc4dd5 0 2011-12-19 00:08:29 1000000 0.032 1KSiuy6DeeDefyj2Ls8mnMvhRPb25xs6jx pubkeyhash 76a914ca4fa68a872fb740f16ca1caaa414bd989362aa788ac 0 -1
158078 5c717d331dae3919337f11722e0e3f62b8c27230ee154f4c6c67b36a60bc4dd5 1 2011-12-19 00:08:29 119721000000 3831.072 1LaN6ooQJpCqC5nFb5TCinceDkCc24eetR pubkeyhash 76a914d6b9bae958a79b931072b93a52e632f2464ae37e88ac 0 -1
158078 1c4e4dfa580566cb92ec3eb43e3650e542ba1ce7a35415b671f59794da702de4 0 2011-12-19 00:08:29 78812408397 2521.9971 1HT5UX8eSiT65fUZSskZYJ4nPPPL58Rzic pubkeyhash 76a914b4709009d5e5844f643b02ecafa7d55d7acb3f7d88ac 0 -1
158078 1c4e4dfa580566cb92ec3eb43e3650e542ba1ce7a35415b671f59794da702de4 1 2011-12-19 00:08:29 100000000 3.2 17gN64BPHtxi4mEM3qWrxdwhieUvRq8R2r pubkeyhash 76a914494294730abf03c846988654f15d1864469c737a88ac 0 -1
158078 bc9822da9fb69f43f1fe33682cd8007ad247deb75f5a6edb641751e456e18b93 0 2011-12-19 00:08:29 3578400000 114.5088 1KDE7sAtyhzJnLXcmJmAMYN9Xk3MRLsDxT pubkeyhash 76a914c7c234d085dd3d5dbc342d59cca174335f554fa988ac 0 -1
Would you be looking for "4104eacd3bfaf2d76524763c4416db596cc26e4ff20c13f12ac7d3cc5b5cec3ec88b53ff35b349f b47421976facdc25e63ea365416c3a4ace7acb7884caa6a6d7a52ac"?


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on April 16, 2022, 03:44:32 PM
I'm looking for all of the entries with script-type pubkey (P2PK) only and is unspent. Primarly between blocks 0 to 278,298. It is about ~34,300 entries. The fields I would like are: block_id    transaction_hash   time   value   value_usd   recipient   type   script_hex   is_from_coinbase is_spendable.
If I get this from "outputs", you get it for all outputs when they were created. So they're all "spendable" (unless they were used again in the same block), even though most of them were spent later on already.
If you need a list similar to what TheArchaeologist (https://bitcointalk.org/index.php?topic=5254914.msg58714326#msg58714326) showed, you don't need most of the data you're asking for. Now the output looks like this:
Code:
160345  3abd1b599fe1e69147c6d375d0c85f47999ba99b5ddfbc090370bcb1d1775760        2012-01-03 00:38:50     1375309318      71.6536 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160345  ff0b0816051a55f0e50820f3551df083491114bff0048e214dca75adf52f7506        2012-01-03 00:38:50     2943300000      153.3459        1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160345  61622dff47f8f242ffeeae9f3b382fa975d309be45248af9f044e5350323b014        2012-01-03 00:38:50     1174309318      61.1815 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160345  a91453015e530f092013d24b063d821d46a739b6cddb913bd7f7fd30710b1971        2012-01-03 00:38:50     2780300000      144.8536        1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160345  bcec7c219b25a598750a85c136446d2992ba336f8e2c2c0a2e1c2248adeacef5        2012-01-03 00:38:50     1072309318      55.8673 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  c3d575d5773035585fcd65d9f8f3291e2dc0aa2f52e28f3a6c6c2ad11ffb4e29        2012-01-03 00:39:37     5001000000      260.5521        1CkjArEYChyvBhjrfw69SZXeMbCLooRvyD      pubkey  410408e007dd2c04b76beeb5a17c43ed81a8393174575bd80ba38b82097611f2ea0ff6bea432e69c22d5c67694462dc9d424477f130d55b35c595e8ec36ef774637cac  1       1
160346  7ea972403219e1b29db5b2dc7370ef3ef01317adca2520520f9934e0a35816ae        2012-01-03 00:39:37     2294300000      119.533 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  6f7a8a7733a44710b239f914b902d47013abf768c09a92c3c82448567dae0e57        2012-01-03 00:39:37     968309318       50.4489 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  5e434a0b0b670073b7c3b9983b73f6b71fccf4d88b200205451ada84da776ec0        2012-01-03 00:39:37     2187300000      113.9583        1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  80bdaf438a966115587de07654e271b89bd6c2be4f158d114ff23f9f09295c96        2012-01-03 00:39:37     845309318       44.0406 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  c2827d8e852bb0e0b0938fba7a214ea6387030773e42f524c805ebc622f9225a        2012-01-03 00:39:37     2159300000      112.4995        1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  caedb38e300073a08fa7cb04bf4da34018a8016506eb6c0fac15601ed375ccca        2012-01-03 00:39:37     745309318       38.8306 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  b53520a99ea46c541f8bcb2c3327d439a525ed897928a3771385550312fafdd1        2012-01-03 00:39:37     2142300000      111.6138        1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  0d11d8a4cd909bb7aba2b0e39fa3ee3f26a4253a6ffe34ffb1b948206826aaba        2012-01-03 00:39:37     741309318       38.6222 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  a0107298cf53abfeb699402def5949763721186ddac81e88e4f55154043bea91        2012-01-03 00:39:37     2025300000      105.5181        1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  7d5e8972f79b61ce3004a74fc5a1bda1e8d5531099ed5dfad3bb675c1982841c        2012-01-03 00:39:37     726309318       37.8407 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  f89d13dfcc18724e1d99a34e0149e7320277ff3e75ffacdd9008573d6c2ac8b2        2012-01-03 00:39:37     1834300000      95.567  1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  54e0dd5717434d69c0e761dbf3791b8df39d13fb8e8578f22e6f161475069280        2012-01-03 00:39:37     712309318       37.1113 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  588329950a7cd00e492ea40c1f2b79c9b3648767dd5c0d8be6b1e51030ace4a5        2012-01-03 00:39:37     1707300000      88.9503 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  4c7636192f99c3e49cba193615e0a9f9e679725f0a545b2ef53206d321b7674c        2012-01-03 00:39:37     556309318       28.9837 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  5960134323fef0e8a6f945306eb3d6d8a1283265de1e61d1068c041c5f5d3d9f        2012-01-03 00:39:37     1686300000      87.8562 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  559b2e1ccbe450d3ca276955dd76677226c00ba82a514397159646a0722c6d6a        2012-01-03 00:39:37     505309318       26.3266 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  aacfd9b0f72603d533bdce3c217f4c291be181b733a7930aa16876dba768270f        2012-01-03 00:39:37     1670300000      87.0226 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  40092b4c718d1078a20edb7e9ead9456314cdedb4daea9d868cab53d7a66f396        2012-01-03 00:39:37     358309318       18.6679 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  5963ff2d7969842dd60a82b47b208037d09e6c47c6db2f4225167a4cade79b7d        2012-01-03 00:39:37     1585300000      82.5941 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
I'm still not sure if the long hash at the end of each line is really what you need, but if it is, I could get only the address and hex from each line, remove the duplicates, match the addresses with the current list of addresses with a balance, and only show those. Would that be more useful? Otherwise you'll end up with a couple hundred GB of data.

Why end at block 278,298 if I you don't mind me asking?


Title: Re: List of all Bitcoin addresses with a balance
Post by: DeepComplex on April 16, 2022, 06:22:55 PM
The sample that you've shown is exactly how I want it. Your suggestion to remove the duplicates and match them with those with a balance of  1btc and more would be appreciated.

Based on what I've seen on the blockchain, p2pk addresses were used up to block 278,298.

Your efforts are truly appreciated.

I am away from my computer so my response is short.

Regards,


I'm looking for all of the entries with script-type pubkey (P2PK) only and is unspent. Primarly between blocks 0 to 278,298. It is about ~34,300 entries. The fields I would like are: block_id    transaction_hash   time   value   value_usd   recipient   type   script_hex   is_from_coinbase is_spendable.
If I get this from "outputs", you get it for all outputs when they were created. So they're all "spendable" (unless they were used again in the same block), even though most of them were spent later on already.
If you need a list similar to what TheArchaeologist (https://bitcointalk.org/index.php?topic=5254914.msg58714326#msg58714326) showed, you don't need most of the data you're asking for. Now the output looks like this:
Code:
160345  3abd1b599fe1e69147c6d375d0c85f47999ba99b5ddfbc090370bcb1d1775760        2012-01-03 00:38:50     1375309318      71.6536 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160345  ff0b0816051a55f0e50820f3551df083491114bff0048e214dca75adf52f7506        2012-01-03 00:38:50     2943300000      153.3459        1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160345  61622dff47f8f242ffeeae9f3b382fa975d309be45248af9f044e5350323b014        2012-01-03 00:38:50     1174309318      61.1815 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160345  a91453015e530f092013d24b063d821d46a739b6cddb913bd7f7fd30710b1971        2012-01-03 00:38:50     2780300000      144.8536        1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160345  bcec7c219b25a598750a85c136446d2992ba336f8e2c2c0a2e1c2248adeacef5        2012-01-03 00:38:50     1072309318      55.8673 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  c3d575d5773035585fcd65d9f8f3291e2dc0aa2f52e28f3a6c6c2ad11ffb4e29        2012-01-03 00:39:37     5001000000      260.5521        1CkjArEYChyvBhjrfw69SZXeMbCLooRvyD      pubkey  410408e007dd2c04b76beeb5a17c43ed81a8393174575bd80ba38b82097611f2ea0ff6bea432e69c22d5c67694462dc9d424477f130d55b35c595e8ec36ef774637cac  1       1
160346  7ea972403219e1b29db5b2dc7370ef3ef01317adca2520520f9934e0a35816ae        2012-01-03 00:39:37     2294300000      119.533 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  6f7a8a7733a44710b239f914b902d47013abf768c09a92c3c82448567dae0e57        2012-01-03 00:39:37     968309318       50.4489 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  5e434a0b0b670073b7c3b9983b73f6b71fccf4d88b200205451ada84da776ec0        2012-01-03 00:39:37     2187300000      113.9583        1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  80bdaf438a966115587de07654e271b89bd6c2be4f158d114ff23f9f09295c96        2012-01-03 00:39:37     845309318       44.0406 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  c2827d8e852bb0e0b0938fba7a214ea6387030773e42f524c805ebc622f9225a        2012-01-03 00:39:37     2159300000      112.4995        1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  caedb38e300073a08fa7cb04bf4da34018a8016506eb6c0fac15601ed375ccca        2012-01-03 00:39:37     745309318       38.8306 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  b53520a99ea46c541f8bcb2c3327d439a525ed897928a3771385550312fafdd1        2012-01-03 00:39:37     2142300000      111.6138        1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  0d11d8a4cd909bb7aba2b0e39fa3ee3f26a4253a6ffe34ffb1b948206826aaba        2012-01-03 00:39:37     741309318       38.6222 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  a0107298cf53abfeb699402def5949763721186ddac81e88e4f55154043bea91        2012-01-03 00:39:37     2025300000      105.5181        1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  7d5e8972f79b61ce3004a74fc5a1bda1e8d5531099ed5dfad3bb675c1982841c        2012-01-03 00:39:37     726309318       37.8407 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  f89d13dfcc18724e1d99a34e0149e7320277ff3e75ffacdd9008573d6c2ac8b2        2012-01-03 00:39:37     1834300000      95.567  1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  54e0dd5717434d69c0e761dbf3791b8df39d13fb8e8578f22e6f161475069280        2012-01-03 00:39:37     712309318       37.1113 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  588329950a7cd00e492ea40c1f2b79c9b3648767dd5c0d8be6b1e51030ace4a5        2012-01-03 00:39:37     1707300000      88.9503 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  4c7636192f99c3e49cba193615e0a9f9e679725f0a545b2ef53206d321b7674c        2012-01-03 00:39:37     556309318       28.9837 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  5960134323fef0e8a6f945306eb3d6d8a1283265de1e61d1068c041c5f5d3d9f        2012-01-03 00:39:37     1686300000      87.8562 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  559b2e1ccbe450d3ca276955dd76677226c00ba82a514397159646a0722c6d6a        2012-01-03 00:39:37     505309318       26.3266 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  aacfd9b0f72603d533bdce3c217f4c291be181b733a7930aa16876dba768270f        2012-01-03 00:39:37     1670300000      87.0226 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  40092b4c718d1078a20edb7e9ead9456314cdedb4daea9d868cab53d7a66f396        2012-01-03 00:39:37     358309318       18.6679 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
160346  5963ff2d7969842dd60a82b47b208037d09e6c47c6db2f4225167a4cade79b7d        2012-01-03 00:39:37     1585300000      82.5941 1VayNert3x1KzbpzMGt2qdqrAThiRovi8       pubkey  4104a39b9e4fbd213ef24bb9be69de4a118dd0644082e47c01fd9159d38637b83fbcdc115a5d6e970586a012d1cfe3e3a8b1a3d04e763bdc5a071c0e827c0bd834a5ac  0       1
I'm still not sure if the long hash at the end of each line is really what you need, but if it is, I could get only the address and hex from each line, remove the duplicates, match the addresses with the current list of addresses with a balance, and only show those. Would that be more useful? Otherwise you'll end up with a couple hundred GB of data.

Why end at block 278,298 if I you don't mind me asking?


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on April 17, 2022, 06:16:33 PM
It took me longer than I expected (because it was more work). Next time it's easy :D

Based on my what I've seen on the block chain, p2pk addresses were used up to block 278,298.
Searching through the first 278,298 blocks (about 1821 daily "outputs" data dumps), I get 901,980 lines that contain the word pubkey. Without duplicate addresses, it's 203,283.
If I search all data, I get 3,669,383 lines. I don't think any address type ever stopped being used, it's still backwards compatible.
If I check how many of those addresses currently have a balance, I get 39,208 from the shortest list. The full list has 39,979 addresses. I'll continue with this one.

I'm not sure why the hash length varies this much:
Code:
367569  c59bd606c40e325841f17d1365d917f953507040ccb540b351967a402999a02e        2015-07-30 00:50:12     2450    0.0072  1aajpztNecNzGBDjJhGfkrsUJFgeyhBgY       pubkey  2103d68f90ba81455256cb7a0df14fb3930d6df61393207f2f3e71659414d296e0f0ac  0       1
367631  72f822c0a805d88706a5d696a1715d9e08006dcc2433a278a4a91f41db587b75        2015-07-30 12:50:57     70000   0.2048  1aajpztNecNzGBDjJhGfkrsUJFgeyhBgY       pubkey  2103d68f90ba81455256cb7a0df14fb3930d6df61393207f2f3e71659414d296e0f0ac  0       1
367679  3fecf7434af30ca54fdadf03dd7e12c52fcdc753f82afe1f545ebf5fc04388fb        2015-07-30 20:53:11     2435048 7.1252  1Kz5QaUPDtKrj5SqW5tFkn7WZh8LmQaQi4      pubkey  4104ffd03de44a6e11b9917f3a29f9443283d9871c9d743ef30d5eddcd37094b64d1b3d8090496b53256786bf5c82932ec23c3b74d9f05a6f95a8b5529352656664bac  1       1
367699  2bb1cf36f7004de8b93978e1b57a924e9e410ba5f1ac1fcf42abe64d9b51decb        2015-07-31 00:54:39     2340276 6.7798  1Kz5QaUPDtKrj5SqW5tFkn7WZh8LmQaQi4      pubkey  4104ffd03de44a6e11b9917f3a29f9443283d9871c9d743ef30d5eddcd37094b64d1b3d8090496b53256786bf5c82932ec23c3b74d9f05a6f95a8b5529352656664bac  1       1
Maybe it's the difference between compressed and uncompressed addresses?

Final result: balance_addy_pubkey.txt (https://loyce.club/other/balance_addy_pubkey.txt) (6.5 MB). It's sorted by balance, highest first.
Sample:
Code:
323317181009 1PTYXwamXXgQoAhDbmUf98rY2Pg1pYXhin 4104633280c0a93b45217059013ddadab8d35b9a858336028fecdff64c6a5e068fadaf7d2b73bc22795fa160c2304703320516e1b0b20e43d613fa5975787c8287e4ac
87500062957 16mEzobs4wQPuAMq1C8QSQafcDHvzczVcs 410469119c7e7d8e8b9f404d70c36cc09cae033e86d259554fd945c9263560fdc9ddea108db86e5673b3894b0bf97807ff8ac6e322791cf8d4d22f7c55a0e812720aac
80000077848 1P2ZAuW9nUrFfwgVjfL2SA9sPXSruCfzp8 4104a9c9c642db7941be7d9d467786c05b66d13c4fdafeb81d66815fb336e7e19d42981f62f2624826898152444b99b590476c90ea7f2e15f50a00e726368d6ec7c1ac
75000064054 15UkFYLMs5nytwiKWqGgkkVo1fjLFAeJhs 41044da39d1810bb01f7039fa7cdfa3dca6de313bbf2d9db9f24170e87b0206f079adb64cccc5cdf6283a54826adaf265087b3f8eba02577153aeaa6cdea8165d6dfac
55000168944 15BKWJjL5YWXtaP449WAYqVYZQE1szicTn 41042e3d29ccb78680a0da35bac8dc7adf895feb2b8eb9dbc5058400d811f6d7ce46faf0099ef63a05f3a8340e7e95530515303d652f27c2d9b379174b41bfd7046bac
.........
.........
.........
1 13P4fnmSVrHa7g6SiYh6mYF8ucB273TqXR 210354e528107cddb35020d91d28549818cc6a0ac57627eec88d48e248ce381fed98ac
1 13DAZmeDhHreHYTqeY7ZdpZt8oW84BwXEQ 2102fd985d1b8352ac5ae0426d4037cfdc34b6d71043e765ef2879d0d6c00aa88285ac
1 12fBP1Dck3pLUqGG7czQqVKAGHNwUCBZxT 21038b9bdf9d80e0e5f40880fc26d425700cbd71de87b232b2cffb6fecf97fa70214ac
1 12f7dZYL1pTkB7smfU6jf6hNhzP3sTASso 21020454e3b33d1df7037ab7767665219bfe9a99ff4aea5672f3a1e9379ef9703a16ac
1 12d9tPargbPamzfhxEh4wdh5jauA5dtobM 210375106eb0bdf4ed504356e8debd010050c57f9c5a03cea95a8dc6d0d6f0cd48ccac


Title: Re: List of all Bitcoin addresses with a balance
Post by: DeepComplex on April 18, 2022, 10:14:41 PM
LoyceV,

Thanks for the list. The list is good.


Can you say how you've compiled it, scripts or Linux commands, etc?
did you pull it from the blockchain, utxo?

I would like to learn how to manipulate the data as you did.


Regards,


It took me longer than I expected (because it was more work). Next time it's easy :D

Based on my what I've seen on the block chain, p2pk addresses were used up to block 278,298.
Searching through the first 278,298 blocks (about 1821 daily "outputs" data dumps), I get 901,980 lines that contain the word pubkey. Without duplicate addresses, it's 203,283.
If I search all data, I get 3,669,383 lines. I don't think any address type ever stopped being used, it's still backwards compatible.
If I check how many of those addresses currently have a balance, I get 39,208 from the shortest list. The full list has 39,979 addresses. I'll continue with this one.

I'm not sure why the hash length varies this much:
Code:
367569  c59bd606c40e325841f17d1365d917f953507040ccb540b351967a402999a02e        2015-07-30 00:50:12     2450    0.0072  1aajpztNecNzGBDjJhGfkrsUJFgeyhBgY       pubkey  2103d68f90ba81455256cb7a0df14fb3930d6df61393207f2f3e71659414d296e0f0ac  0       1
367631  72f822c0a805d88706a5d696a1715d9e08006dcc2433a278a4a91f41db587b75        2015-07-30 12:50:57     70000   0.2048  1aajpztNecNzGBDjJhGfkrsUJFgeyhBgY       pubkey  2103d68f90ba81455256cb7a0df14fb3930d6df61393207f2f3e71659414d296e0f0ac  0       1
367679  3fecf7434af30ca54fdadf03dd7e12c52fcdc753f82afe1f545ebf5fc04388fb        2015-07-30 20:53:11     2435048 7.1252  1Kz5QaUPDtKrj5SqW5tFkn7WZh8LmQaQi4      pubkey  4104ffd03de44a6e11b9917f3a29f9443283d9871c9d743ef30d5eddcd37094b64d1b3d8090496b53256786bf5c82932ec23c3b74d9f05a6f95a8b5529352656664bac  1       1
367699  2bb1cf36f7004de8b93978e1b57a924e9e410ba5f1ac1fcf42abe64d9b51decb        2015-07-31 00:54:39     2340276 6.7798  1Kz5QaUPDtKrj5SqW5tFkn7WZh8LmQaQi4      pubkey  4104ffd03de44a6e11b9917f3a29f9443283d9871c9d743ef30d5eddcd37094b64d1b3d8090496b53256786bf5c82932ec23c3b74d9f05a6f95a8b5529352656664bac  1       1
Maybe it's the difference between compressed and uncompressed addresses?

Final result: balance_addy_pubkey.txt (https://loyce.club/other/balance_addy_pubkey.txt) (6.5 MB). It's sorted by balance, highest first.
Sample:
Code:
323317181009 1PTYXwamXXgQoAhDbmUf98rY2Pg1pYXhin 4104633280c0a93b45217059013ddadab8d35b9a858336028fecdff64c6a5e068fadaf7d2b73bc22795fa160c2304703320516e1b0b20e43d613fa5975787c8287e4ac
87500062957 16mEzobs4wQPuAMq1C8QSQafcDHvzczVcs 410469119c7e7d8e8b9f404d70c36cc09cae033e86d259554fd945c9263560fdc9ddea108db86e5673b3894b0bf97807ff8ac6e322791cf8d4d22f7c55a0e812720aac
80000077848 1P2ZAuW9nUrFfwgVjfL2SA9sPXSruCfzp8 4104a9c9c642db7941be7d9d467786c05b66d13c4fdafeb81d66815fb336e7e19d42981f62f2624826898152444b99b590476c90ea7f2e15f50a00e726368d6ec7c1ac
75000064054 15UkFYLMs5nytwiKWqGgkkVo1fjLFAeJhs 41044da39d1810bb01f7039fa7cdfa3dca6de313bbf2d9db9f24170e87b0206f079adb64cccc5cdf6283a54826adaf265087b3f8eba02577153aeaa6cdea8165d6dfac
55000168944 15BKWJjL5YWXtaP449WAYqVYZQE1szicTn 41042e3d29ccb78680a0da35bac8dc7adf895feb2b8eb9dbc5058400d811f6d7ce46faf0099ef63a05f3a8340e7e95530515303d652f27c2d9b379174b41bfd7046bac
.........
.........
.........
1 13P4fnmSVrHa7g6SiYh6mYF8ucB273TqXR 210354e528107cddb35020d91d28549818cc6a0ac57627eec88d48e248ce381fed98ac
1 13DAZmeDhHreHYTqeY7ZdpZt8oW84BwXEQ 2102fd985d1b8352ac5ae0426d4037cfdc34b6d71043e765ef2879d0d6c00aa88285ac
1 12fBP1Dck3pLUqGG7czQqVKAGHNwUCBZxT 21038b9bdf9d80e0e5f40880fc26d425700cbd71de87b232b2cffb6fecf97fa70214ac
1 12f7dZYL1pTkB7smfU6jf6hNhzP3sTASso 21020454e3b33d1df7037ab7767665219bfe9a99ff4aea5672f3a1e9379ef9703a16ac
1 12d9tPargbPamzfhxEh4wdh5jauA5dtobM 210375106eb0bdf4ed504356e8debd010050c57f9c5a03cea95a8dc6d0d6f0cd48ccac


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on April 19, 2022, 10:34:38 AM
Can you say how you've compiled it, scripts or Linux commands, etc?
did you pull it from the blockchain, utxo?
I don't want to keep going off-topic, so I'll make a new topic for it.
See Get pubkeys out of Bitcoin block data (https://bitcointalk.org/index.php?topic=5395212.msg59906483#msg59906483).


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on April 22, 2022, 07:45:08 AM
The list is good.
After thinking it over: I don't think it's correct. This address (https://blockchair.com/bitcoin/address/14a52a7AJccWjNCeS5PGwov7C2xnP7j21R) for example has never sent anything, so as far as I know, it's pubkey can't be known. And yet, it's in my list, while other addresses that have been used to send funds aren't in there.
You're looking for public keys that can be used to brute-force private keys, right? I'm still not sure what the pubkey in my list belongs to.


Title: Re: List of all Bitcoin addresses with a balance
Post by: PawGo on April 22, 2022, 07:52:39 AM
The list is good.
After thinking it over: I don't think it's correct. This address (https://blockchair.com/bitcoin/address/14a52a7AJccWjNCeS5PGwov7C2xnP7j21R) for example has never sent anything, so as far as I know, it's pubkey can't be known. And yet, it's in my list, while other addresses that have been used to send funds aren't in there.
You're looking for public keys that can be used to brute-force private keys, right? I'm still not sure what the pubkey in my list belongs to.

I did not check your list, but do you have addresses like that:
https://blockchair.com/bitcoin/address/1H5G1HXeMMQGowH98rF6FpGWDnxvdVeTUr
Code:
04b1aac4d864cad13f9d168e2fed8a8f1db59d33f5e4041f32882aa75d5b083e80bba9e05ba25a61f68053fe7e0889e822c56519c9f0653c4f9534ca91f4f56ad6


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on April 22, 2022, 08:45:38 AM
do you have addresses like that:
It's in there:
Code:
5000000000 1H5G1HXeMMQGowH98rF6FpGWDnxvdVeTUr 4104b1aac4d864cad13f9d168e2fed8a8f1db59d33f5e4041f32882aa75d5b083e80bba9e05ba25a61f68053fe7e0889e822c56519c9f0653c4f9534ca91f4f56ad6ac


Title: Re: List of all Bitcoin addresses with a balance
Post by: DeepComplex on April 24, 2022, 12:33:57 AM
LoyceV,

Those addresses are V1 bitcoin addresses used in the inception up to around the end of ~ 2012. Presumably by Satoshi et al. They are P2PK  (https://learnmeabitcoin.com/technical/p2pk) addresses.

Regards,

The list is good.
After thinking it over: I don't think it's correct. This address (https://blockchair.com/bitcoin/address/14a52a7AJccWjNCeS5PGwov7C2xnP7j21R) for example has never sent anything, so as far as I know, it's pubkey can't be known. And yet, it's in my list, while other addresses that have been used to send funds aren't in there.
You're looking for public keys that can be used to brute-force private keys, right? I'm still not sure what the pubkey in my list belongs to.


Title: Re: List of all Bitcoin addresses with a balance
Post by: DeepComplex on April 24, 2022, 01:31:14 AM
Can you prepare another list with all addresses with known pubkey and positive balance in the same format as you did before?

Regards,

LoyceV,

Those addresses are V1 bitcoin addresses used in the inception up to around the end of ~ 2012. Presumably by Satoshi et al. They are P2PK  (https://learnmeabitcoin.com/technical/p2pk) addresses.

Regards,

The list is good.
After thinking it over: I don't think it's correct. This address (https://blockchair.com/bitcoin/address/14a52a7AJccWjNCeS5PGwov7C2xnP7j21R) for example has never sent anything, so as far as I know, it's pubkey can't be known. And yet, it's in my list, while other addresses that have been used to send funds aren't in there.
You're looking for public keys that can be used to brute-force private keys, right? I'm still not sure what the pubkey in my list belongs to.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on April 24, 2022, 06:38:53 AM
They are P2PK  (https://learnmeabitcoin.com/technical/p2pk) addresses.
Thanks, that link explains why the pubkey for those addresses is known without sending a transaction:
Quote
I think P2PKH was used as a convenient way for people to make transactions, as it allowed you to use addresses instead of passing around your public key (which is longer).

Can you prepare another list with all addresses with known pubkey and positive balance in the same format as you did before?
If you can get me an example showing what part of the block data (https://bitcointalk.org/index.php?topic=5307550.0) you need, I can get it for you. See this post (https://bitcointalk.org/index.php?topic=5307550.msg56378771#msg56378771) for a sample of the data.
Please edit your post instead of making 2 posts in a row, see forum rules (https://bitcointalk.org/index.php?topic=703657.0).


Title: Re: List of all Bitcoin addresses with a balance
Post by: racminer on May 04, 2022, 12:02:20 PM
Maybe a dumb question:
 I downloaded this file all_Bitcoin_addresses_ever_used_sorted.txt.gz from  this site http://addresses.loyce.club and I found 18635073 wallets having this form x-ff8bb9190e0e969f5b8176dba0c0f161
where  x can be s,d,m or another letter followed by - and a string of hex digits ...
they don't look like hash160 representation so  what are they ?

Thank you:


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on May 04, 2022, 02:29:29 PM
I found 18635073 wallets having this form x-ff8bb9190e0e969f5b8176dba0c0f161
where  x can be s,d,m or another letter followed by - and a string of hex digits ...
they don't look like hash160 representation so  what are they ?
It's Blockchair.com's way of showing OP_RETURN outputs. I'm not sure how "real" those are, and if you'd get the same result if you'd get the data from your own Bitcoin Core installation. For most purposes you can ignore them.


Title: Re: List of all Bitcoin addresses with a balance
Post by: racminer on May 04, 2022, 08:02:42 PM
I found 18635073 wallets having this form x-ff8bb9190e0e969f5b8176dba0c0f161
where  x can be s,d,m or another letter followed by - and a string of hex digits ...
they don't look like hash160 representation so  what are they ?
It's Blockchair.com's way of showing OP_RETURN outputs. I'm not sure how "real" those are, and if you'd get the same result if you'd get the data from your own Bitcoin Core installation. For most purposes you can ignore them.

To be precise:
18125075  d-
     10092   s-
   499906   m-

18635073 Total


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on June 22, 2022, 08:42:21 AM
I'll take this quote from another topic here:
In the list of founded addresses there is unnecessary line "address"  (kind of column header) between addresses "3.." and "bc..", line 31808561. Like you had separated lists and you concatenated them.
Thanks! That "address" should have been removed (but due to copying code I tried to remove the header from another file). It will be fixed in tomorrow's update.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on June 22, 2022, 11:27:24 AM
how did you compile this list?
See:
Credits
Blockchair Database Dumps (https://blockchair.com/dumps) has a staggering amount of data, easily accessible (at 10 kB/s (https://gz.blockchair.com/README.html)) with daily updates. All data presented in this topic comes from Blockchair.


Title: Re: List of all Bitcoin addresses with a balance
Post by: andresteves on August 08, 2022, 02:40:06 PM
Thanks @LoyceV, you saved me a lot of time as I was searching for something like for a while and stepped on it today :).

Really appreciate.


Title: Re: List of all Bitcoin addresses with a balance
Post by: citb0in on August 30, 2022, 03:18:56 PM
Hi LoyceV and rest,

... You're looking for public keys that can be used to brute-force private keys, right? ...

you replied to DeepComplex as he was asking for the pubkey of an address. Can you (or DeepComplex or anyone else) explain to me please, why the pubkey is such a help and thus interesting for assisting in finding out the private key for this particular address? Where can I find more information about this, I am trying to understand. Wouldn't that be a serious concern when I spend some BTC to someone and my pubkey is revealed within the transaction? Thanks for any helpful information to all in advance.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on August 30, 2022, 03:26:18 PM
Can you (or DeepComplex or anyone else) explain to me please, why the pubkey is such a help and thus interesting for assisting in finding out the private key for this particular address? Where can I find more information about this
I don't know the technical details, but knowing the public key makes brute-forcing a partial private key much easier.

Quote
Wouldn't that be a serious concern when I spend some BTC to someone and my pubkey is revealed within the transaction?
No, not without quantum computers.


Title: Re: List of all Bitcoin addresses with a balance
Post by: citb0in on August 30, 2022, 03:31:09 PM
I don't know the technical details, but knowing the public key makes brute-forcing a partial private key much easier.

anyone can provide some more information on this topic, please ?


Title: Re: List of all Bitcoin addresses with a balance
Post by: albert0bsd on September 02, 2022, 08:33:03 PM
anyone can provide some more information on this topic, please ?

A public key for ECC is a point in the curve it is just a set of coordinates (X,Y) but it represent the privatekey that is just a number, we can add, subtract, multiply and divide values to the publickey and the result is another publickey this is useful because there are some algorithms that can SPEED the search if you know part of the range of the original publickey. This is only useful for partial WIF and puzzles

Examples:
Looking for address of the puzzle 64 without publickey available will take some 292 thousand years with a CPU

Looking for a publickey in the 64 bit range will take 5 to 10 minutes with a CPU. For a GPU this taask can take some 20 Seconds to one minute.

The main problem with Publickey as a target is that all the calculations are made specific per publickey, this means that you can't take a list of 10 Millions of publickeys and expected the same performance.

More information:
https://andrea.corbellini.name/2015/06/08/elliptic-curve-cryptography-breaking-security-and-a-comparison-with-rsa/


Title: Re: List of all Bitcoin addresses with a balance
Post by: mausuv on December 23, 2022, 04:14:22 PM
@LoyceV (OP)
Hi, I need Bitcoin Public Key Database with balance
Please upload Bitcoin Public Key Database with balance your site http://addresses.loyce.club/

Example Publickeys.txt
Code:
02ceb6cbbcdbdf5ef7150682150f4ce2c6f4807b349827dcdbdd1f2efa885a2630
02ceb6cbbcdbdf5ef7150682150f4ce2c6f4807b349827dcdbdd1f2efa885a2630
02ceb6cbbcdbdf5ef7150682150f4ce2c6f4807b349827dcdbdd1f2efa885a2630

Bitcoin Public Key Database please any  gdrive github other site send link
i need 1M < 10M public keys .txt formet


Title: Re: List of all Bitcoin addresses with a balance
Post by: albert0bsd on December 24, 2022, 03:21:24 AM
Hi, I need Bitcoin Public Key Database with balance

I already ask if LoyceV has or can share that data but seem that it is a lot of work.

The best way is do this with an script, take your list of address of interest and get the publickey with an API if it is available.

Regards!


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on December 24, 2022, 06:50:48 AM
@LoyceV (OP)
Hi, I need Bitcoin Public Key Database with balance
As albert0bsd said, it's complicated. Read the last few pages of this topic, read LoyceV's small Linux commands for handling big data (https://bitcointalk.org/index.php?topic=5395212.0) and most importantly read this post (https://bitcointalk.org/index.php?topic=5395212.msg59976653#msg59976653) on the problems I ran into at my attempt to get all known pubkeys.
I haven't found a solution, which is why I abandoned the pubkey project (https://bitcointalk.org/index.php?topic=5395212.msg60139427#msg60139427). If you can tell me how to get pubkeys out of the data I've published, I'll may it another try.


Title: Re: List of all Bitcoin addresses with a balance
Post by: pbies on January 23, 2023, 04:58:45 AM
Is it possible to gain RIPEMD160 version of these addresses?

Eventually how to convert these addresses to RIPEMD160?


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on January 23, 2023, 06:37:01 AM
Is it possible to gain RIPEMD160 version of these addresses?

Eventually how to convert these addresses to RIPEMD160?
I'm not sure what RIPEMD160 is, and how it relates to pubkeys, but I think this applies:
The very act of hashing destroys any relation there could have been with inputs, no matter what the hashing algorithm is.


Title: Re: List of all Bitcoin addresses with a balance
Post by: wordbustingmistake on January 26, 2023, 06:58:17 PM
Is it possible to gain RIPEMD160 version of these addresses?

Eventually how to convert these addresses to RIPEMD160?

P2PKH (26-35 characters long addresses starting 1, base58check encoded) and P2WPKH (42 characters long addresses starting bc1q, bech32 encoded) addresses are basically Ripemd160 hashes of public keys just encoded and with a little overhead (e.g. checksums). From loyce set of addresses you need to select only addresses matching proper length and starting characters, then use a decoder to verify checksums, decode addresses to hex and remove whatever overhead is leftover. Bit library for Python has base58 and base32 classes with methods to do that, so it's going to be something like:

Code:
print(base58.b58decode_check(P2PKH_address)[1:].hex())
print(bytes(base32.decode(P2WPKH_address)[1]).hex())


Title: Re: List of all Bitcoin addresses with a balance
Post by: pbies on August 14, 2023, 09:30:18 PM
...

Hi, I need Bitcoin Public Key Database with balance
Please upload Bitcoin Public Key Database with balance your site http://addresses.loyce.club/

...

I can share such database. It is 5 MB as .rar.

Inside are: transaction ids, amounts, public keys.

You can obtain such file by go program with name "bitcoin-utxo-dump".


Title: Re: List of all Bitcoin addresses with a balance
Post by: seoincorporation on August 15, 2023, 03:13:32 AM
...

Hi, I need Bitcoin Public Key Database with balance
Please upload Bitcoin Public Key Database with balance your site http://addresses.loyce.club/

...

I can share such database. It is 5 MB as .rar.

Inside are: transaction ids, amounts, public keys.

You can obtain such file by go program with name "bitcoin-utxo-dump".

I was reading about bitcoin-utxo-dump and looks like the right tool for this project, the files are not 5 MB, the file should be around 7GB and it takes close to 20 minutes to get all the unspent coins. After that we should group the unspent transactions from the same address and that way we can know the balance from each address...

The thing here is, you should run the dump each hour, then group the address and make a new csv file with the data, then update the database from the csv file, sounds as a busy cron job, but is possible.

Source: https://github.com/in3rsha/bitcoin-utxo-dump


Title: Re: List of all Bitcoin addresses with a balance
Post by: Mek on August 16, 2023, 06:37:12 AM
Hello
if I run my own full node, how can I generate the same TSV file as is available on loyce.club site? (all addresses with balance)
Thanks.


Title: Re: List of all Bitcoin addresses with a balance
Post by: ABCbits on August 16, 2023, 09:47:55 AM
Hello
if I run my own full node, how can I generate the same TSV file as is available on loyce.club site? (all addresses with balance)
Thanks.

It depends on which software you use to run full node. Some software (including Bitcoin Core) doesn't have address index, which force you to either perform workaround (e.g. load and parse UTXO) or install additional software (such as Electrum server or self-hosted mempool.space) which perform address index.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on August 16, 2023, 12:22:29 PM
if I run my own full node, how can I generate the same TSV file as is available on loyce.club site? (all addresses with balance)
As far as I know, all you need for this is the data in the "chainstate" directory, which isn't that much. There are a few parsers out there, just try to find which one you like.
I don't need more frequent updates, so I never setup my own parser.


Title: Re: List of all Bitcoin addresses with a balance
Post by: stas87 on August 20, 2023, 05:20:50 PM
Hi, thanks you for the ability to download all funded bitcoin addresses. Do you have such option for USDT (Tron) addresses? Or its blockchain is a way too heavy? Thanks


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on August 21, 2023, 05:51:17 AM
Hi, thanks you for the ability to download all funded bitcoin addresses. Do you have such option for USDT (Tron) addresses? Or its blockchain is a way too heavy? Thanks
Nope, only Bitcoin here.


Title: Re: List of all Bitcoin addresses with a balance
Post by: Mek on August 30, 2023, 08:31:59 AM
if I run my own full node, how can I generate the same TSV file as is available on loyce.club site? (all addresses with balance)
As far as I know, all you need for this is the data in the "chainstate" directory, which isn't that much. There are a few parsers out there, just try to find which one you like.
I don't need more frequent updates, so I never setup my own parser.
OK, can you please publish the way you generate the TSV file? The latest one on the website is already a week old. Maybe someone needs more frequent updates, or to have a backup plan should your website go down or stop being updated (these things happen...)


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on August 30, 2023, 09:14:32 AM
OK, can you please publish the way you generate the TSV file?
I did publish that, read the OP.

Quote
The latest one on the website is already a week old.
Thanks for this, I kinda forgot about this data project when I started another (big) download from Blockchair. Since I'm only allowed to download one connection at a time, it blocked other updates.
I'm running an update now, it should be available in about 3 hours.

Quote
Maybe someone needs more frequent updates, or to have a backup plan should your website go down or stop being updated (these things happen...)
If that happens, they'll have to figure it out on their own. If you're interested in discussing different parsers, it's probably best to open another topic about it.


Title: Re: List of all Bitcoin addresses with a balance
Post by: pbies on August 31, 2023, 01:23:12 PM
For all of you, a script which downloads and converts LoyceV addresses to brainflayer bloom filter.
Taken are all >1 mBTC addresses.

Code:
#!/usr/bin/env bash
set -e
echo "### Public addresses downloader and converter to bloom filter ###"

echo [01/12] Download the addresses
wget http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz

echo [02/12] Unpack the archive
pv blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz | gunzip > blockchair_bitcoin_addresses_and_balance_LATEST.tsv

echo [03/12] Check lowest balance line
line=$(grep -m 1 -bne $'\t99999$' blockchair_bitcoin_addresses_and_balance_LATEST.tsv | cut -d ":" -f 2)

echo [04/12] Get only with proper balance
head -c $line blockchair_bitcoin_addresses_and_balance_LATEST.tsv > int1.txt

echo [05/12] Get rid of first line
tail -n +2 int1.txt > int2.txt

echo [06/12] Get only addresses, sort and write to file
pv int2.txt | cut -f 1 | sort > gte1mBTC.txt

echo [07/12] Search for incompatible addresses
line=$(grep -m 1 -bne $'^bc1p' "gte1mBTC.txt" | cut -d ":" -f 2)

echo [08/12] Get only compatible addresses
head -c $line "gte1mBTC.txt" > int3.txt

echo "[09/12] Convert addresses to hash160 ($(wc -l < int3.txt) lines)"
pv int3.txt | ~/brainflayer/brainflayer -o int4.txt

echo [10/12] Sort and uniq hashes
pv -c int4.txt | sort -u | pv -c > hash160.txt

echo [11/12] Convert hex to bloom filter
~/brainflayer/hex2blf "hash160.txt" bloom.blf

echo [12/12] Finished!
>&2 echo -ne "\x7"


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on August 31, 2023, 01:35:36 PM
For all of you, a script which downloads and converts LoyceV addresses to brainflayer bloom filter.
I've added it to the OP:
If you want to create a brainflayer bloom filter: pbies wrote a guide (https://bitcointalk.org/index.php?topic=5254914.msg62774192#msg62774192).

Note that I haven't tested it, so as always, do your own research. You shouldn't be messing with bloom filters if you don't know what you're doing anyway ;)


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on September 11, 2023, 09:01:30 AM
Since HASH160 keeps coming up, I'll link World's fastest and simplest block parser for those who (only) need all HASH160 (https://bitcointalk.org/index.php?topic=5465751.0) here. This is for all addresses ever used though.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on September 29, 2023, 08:28:04 AM
I've moved this project to another (donated) server. Everything should be back online again, let me know if there are any issues.


Title: Re: List of all Bitcoin addresses with a balance
Post by: Ermitage on January 20, 2024, 06:38:33 AM
Hi,
Why do I see only the address bc1qpcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t on blockchair.com but nowhere else?
BR


Title: Re: List of all Bitcoin addresses with a balance
Post by: citb0in on January 20, 2024, 08:34:12 AM
Hi,
Why do I see only the address bc1qpcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t on blockchair.com but nowhere else?
BR

simply said, its an invalid bitcoin address. Just because blockchair displays it, doesn't mean it exists. Blockchair seems to accept any address via URL parameter, even invalid ones like the one you mentioned.

Valid characters in in bech32 addresses are: 023456789acdefghjklmnpqrstuvwxyz
So invalid characters are: 1bio

Try enter this URL and you will get a similar result of what you asked for:
https://blockchair.com/bitcoin/address/bc1q123testbio123testbio123

However, your question is off-topic. Please make sure that in future you select a suitable subforum in which your question fits. If in doubt, create a new thread.


Title: Re: List of all Bitcoin addresses with a balance
Post by: Ermitage on January 20, 2024, 09:29:24 AM
Sorry, I didn't mention that my question is related to the thread, but indeed.
The LoyceVs list contains the line. So, the question was "Why does the list contain the line if the address doesn't exist in the real blockchain?".
The same for these addresses (for convenience, the first one is also included):
bc1pqcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t
bc1sw50qgdz25j
bc1pewum2h6sap5ch5lf9qns4t0ufe9f6wf3z5s225
bc1zw508d6qejxtdg4y5r3zarvaryvaxxpcs
bc1pvw7t698cn379acgstnzwtnn8r6qxernnsrul3z
bc1pgptwf4gcrwy3qvq26p3cwapagaxyvsd8syuwhq
bc1putjagjerw0qw9qmy0wxu2pfk0v5t8gu7ranaxg
bc1pj7gr7lpxdnf6fm82jyy602t7xs9jjf3cp30yc7
bc1prkqmm25t4fn84k3j0dvlhkut4yeyuwurekel3c
bc1pxcsyvgrxyp8jqmeqtqs8sgpkyq7zq0sxpf58k
bc1p042rxsz3865566ykcd956pymkn0fxkn8lq96jx
bc1px5sy2gr9yp8zqm3q2us8wgp4yq4jq0gf5cpy9
bc1pxusywgr8ypgzqupqtys8jgphyq4zqgcmufa07
bc1p23jk6urvv96x2gp3yqszqgpqyqszqgqgxs8es
bc1p8ysyjgrfypfzqu3q9usrqgpeyqnzqfgv63qmh
bc1pxvsyxgrrypxzqmpq25s82gpnypajqlg7ux9e3
bc1pxqsrzgpjyqejqdpqx5srvgphyquzqwgc3wgd8
bc1pdv6mhmhayxp7xjxut0zhdpexskf7te5wfa53pv
bc1pjw7qg604staq70s86sfdq80zy5df2ljdl9n370
bc1p8qsysgrgypgjqufqtgs85gpcyqjzqsqul7r68
bc1pxysyzgrpyp9zq63q2vs8xgp3ypdjqhgfsx3d3
bc1pchjs0jhpga0lrqwwn26k6a9v5hj9pr8e522vq8
bc1pxgsyygrzyp9jq6eq2ss8ggpjyq5zq2g4sz4gk
bc1pxssyggryypxjqmfq2cs8vgp5yqsjq0ctxl0l2
bc1zqyqs3juw9m
bc1pm9jzmujvdqjj6y28hptk859zs3yyv78hlz84pm


Title: Re: List of all Bitcoin addresses with a balance
Post by: citb0in on January 20, 2024, 11:11:29 AM
Sorry, I didn't mention that my question is related to the thread, but indeed.
The LoyceVs list contains the line. So, the question was "Why does the list contain the line if the address doesn't exist in the real blockchain?".
The same for these addresses (for convenience, the first one is also included):
bc1pqcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t
bc1sw50qgdz25j
--snip--

I see. But I can't find the Bitcoin address bc1qpcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t that you originally mentioned in the current LoyceV funded addresses (http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz) list:

even up to this prefix there is no single hit
Code:
$ grep 'bc1qpcmm7k' blockchair_bitcoin_addresses_and_balance_LATEST.tsv
Quote
<nothing>

first hit occurs here, but this is not the address you mentioned before.
Code:
$ grep 'bc1qpcmm7' blockchair_bitcoin_addresses_and_balance_LATEST.tsv
Quote
bc1qpcmm74pjafundkxm79pk4hl42skcy7dswa7skq   18000

Now to the other addresses you have listed...

The same for these addresses (for convenience, the first one is also included):
bc1pqcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t
bc1sw50qgdz25j
--snip--

The first line has a typo, see the address you posted at the beginning and see this address you mentioned:
bc1qpcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t
bc1pqcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t

Apparently you have misspelled the address in your initial question. You have a typo :) Being that said, the address with the prefix 'bc1p' is in the LoyceV list:
Code:
$ grep 'bc1pqcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t' blockchair_bitcoin_addresses_and_balance_LATEST.tsv
Quote
bc1pqcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t   41354

Now we are one step further. These are special other address types. You can find more information HERE (https://bitcointalk.org/index.php?topic=5429436.0).

Hope this answers your question.
citb0in


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on January 20, 2024, 11:42:29 AM
Why do I see only the address bc1qpcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t on blockchair.com but nowhere else?
This address is empty (https://blockchair.com/bitcoin/address/bc1qpcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t) and not included in my list of funded addresses (http://addresses.loyce.club/).

The LoyceVs list contains the line.
No, it does not.

Quote
So, the question was "Why does the list contain the line if the address doesn't exist in the real blockchain?".
Search the differences:
https://blockchair.com/bitcoin/address/bc1qpcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t (https://blockchair.com/bitcoin/address/bc1qpcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t)
https://blockchair.com/bitcoin/address/bc1pqcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t (https://blockchair.com/bitcoin/address/bc1pqcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t)

Lesson learned: use copy paste (https://bitcointalk.org/index.php?topic=5190776.0) and carefully check each character. If I type an address on an offline computer, I create a checksum on both just to be sure I typed it correctly.


Title: Re: List of all Bitcoin addresses with a balance
Post by: Ermitage on January 20, 2024, 05:15:18 PM
Guys, thank you for your discovery and answers.
It looks a bit strange because I always use copy/paste and so on.
But the lines I see in the file I downloaded this morning.
I am ready to provide screenshots to all of mentioned addresses.
But, I already have an idea why it can be so. Bad if it affected not only me.
I just extracted the data using 7-zip and those text lines are in the result file. Maybe some bytes were mixed (shifted)
But I like more the idea of different witness versions.
Screenshot for some lines: https://ibb.co/tJJhGRL
Blockchain on the source site: https://blockchair.com/bitcoin/address/bc1pqcmm7krqrmsjr4k84r85v6s9rhnl9warprrp0t


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on January 20, 2024, 06:16:16 PM
I just extracted the data using 7-zip and those text lines are in the result file. Maybe some bytes were mixed (shifted)
If that's the case, write to 7-zip. I just use tar, and it works fine.


Title: Re: List of all Bitcoin addresses with a balance
Post by: efjay on February 01, 2024, 11:16:56 AM
Many thanks for making this available, liked it enough to login to BCT and post for the first time in years!!


I started playing with the all addresses with balance data and made a poor version of a full ranked list.  Simple one line command in bash or git bash (and others) if people are interested.

For those that want to, download the data file locally and use something like this (probably more efficient ways to do this but I was in a hurry :) ) :

zcat datafile.tar.gz | grep -v address | sort -k2 -nr | nl -w8 - > outputfilename.txt

This will rank from largest as #1 down to smallest (over 50 million addresses currently).


Unfortunately it seems the BTC addresses that I have are off-chain / micro wallets.  Is this an area that interests you to expand the dataset?  (I don't know much about how these sort of wallets/systems work under the covers at this stage so not sure what is in the art of the possible)

Assuming the places that run these wallets provide access to the data in some way of course, at this stage I haven't gone down the path of opening any discussion with blockchair directly.

Enjoy!


Title: Re: List of all Bitcoin addresses with a balance
Post by: citb0in on February 01, 2024, 11:37:39 AM
zcat datafile.tar.gz | grep -v address | sort -k2 -nr | nl -w8 - > outputfilename.txt

This will rank from largest as #1 down to smallest (over 50 million addresses currently).

I don't know which file you have downloaded but I am confused about your procedure because: the list, which also contains the balances, is sorted by default by balance with the highest at the top and the lowest at the bottom. So you don't have to sort anything ;) YMMV


Title: Re: List of all Bitcoin addresses with a balance
Post by: efjay on February 01, 2024, 11:57:51 AM

I don't know which file you have downloaded but I am confused about your procedure because: the list, which also contains the balances, is sorted by default by balance with the highest at the top and the lowest at the bottom. So you don't have to sort anything ;) YMMV

Fair comment.  guess it future proofs any changes to the sorting of the source data as long as the balance data is in the same column :)

Cheers



Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on February 01, 2024, 02:30:55 PM
Many thanks for making this available, liked it enough to login to BCT and post for the first time in years!!
You're welcome :)

Quote
Unfortunately it seems the BTC addresses that I have are off-chain / micro wallets.
What do you mean by "off-chain"? Bitcoin addresses can only be on-chain (unless they're unused, in that case they're nowhere).


Title: Re: List of all Bitcoin addresses with a balance
Post by: efjay on February 01, 2024, 02:49:50 PM
Quote
Unfortunately it seems the BTC addresses that I have are off-chain / micro wallets.
What do you mean by "off-chain"? Bitcoin addresses can only be on-chain (unless they're unused, in that case they're nowhere).

An example being a deposit address on Faucetpay.io is not in the list of 50+ million addresses (source data http://addresses.loyce.club-blockchair/bitcoin_addresses_and_balance_LATEST.tsv.gz from a few days ago at least).

In this particular case the address starts with 1EZASQnt79J and can be sent to from many locations (do they all have their own faucetpay integration feature outside of the BTC blockchain - I don't know).

Cheers


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on February 02, 2024, 09:07:46 AM
An example being a deposit address on Faucetpay.io is not in the list
That's because the address was never used on-chain. I don't use Faucetpay, but I guess they use the Bitcoin address as "account number". Anyone can do anything they want off-chain, there's no way to index that.


Title: Re: List of all Bitcoin addresses with a balance
Post by: efjay on February 03, 2024, 11:33:31 AM
An example being a deposit address on Faucetpay.io is not in the list
That's because the address was never used on-chain. I don't use Faucetpay, but I guess they use the Bitcoin address as "account number". Anyone can do anything they want off-chain, there's no way to index that.

Yeah I had a quick look at one of the addresses and blockchair has some history data, so I think they use it like a clearing account.

What I mean by that is that it is a real address available onchain but also offchain (some of my terminology might be a bit wrong but feel free to correct me if there's better ways to elaborate on how it works).

Offchain usage (or is that considered sidechain) is like the network of systems that can talk direct to the (in this case the one i looked at, faucetpay) network/system, allowing transfers to be fee free if the sending system chooses to do so (transactions don't show in blockchair in this case).

Onchain usage is normal real transfers via the block chain, registers in the historical data (eg on blockchair) and then at some stage (algo? time? amount? don't know) is moved out of that address to another address (hence the clearing) - but in the interface that I see the amount is still allocated to me.  So my balance to my (faucetpay) address is buried in another address, hopefully readily available to me when i need to spend it!

And that would explain why my address doesn't show in the list of all address and balances above zero, as when they clear it they move the whole balance out.

Cheers


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on February 03, 2024, 12:27:42 PM
What I mean by that is that it is a real address available onchain
It's easy to generate billions of "real" addresses, but they're meaningless until they receive an on-chain Bitcoin payment.

Quote
And that would explain why my address doesn't show in the list of all address and balances above zero
Bitcoin's blockchain is a ledger for anything that ever happened. Things made up by Faucetpay internally have nothing to do with Bitcoin.


Title: Re: List of all Bitcoin addresses with a balance
Post by: runlola on April 30, 2024, 11:17:56 AM
The files haven't been updated in over a week, though the source files on blockchair are still updated. Don't see any maintenance status mentioned on the download site or this topic.


Title: Re: List of all Bitcoin addresses with a balance
Post by: marssystems on May 09, 2024, 06:58:44 PM
The files haven't been updated in over a week
It looks like the daily updates (http://alladdresses.loyce.club/daily_updates/) didn't happen, which means the weekly update had no fresh data. I'm not sure why, I'll check it tomorrow.

Update: it's fixed. It was waiting for another stuck update (https://bitcointalk.org/index.php?topic=5265993.msg64025835#msg64025835). The weekly update is running now, this takes a few hours to complete. Thanks for reporting this.

Files haven't been updated since 19 April 2024.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on May 10, 2024, 07:04:30 AM
The files haven't been updated in over a week
I get this data from Blockchair.com (https://gz.blockchair.com/bitcoin/addresses/), and their last update is from April 19. Their other data (https://gz.blockchair.com/bitcoin/outputs/) is still being updated, so it looks like something is stuck with updating the Bitcoin addresses.

Files haven't been updated since 19 April 2024.
I posted both posts in the wrong topic. It's correct now, the other post is here (https://bitcointalk.org/index.php?topic=5265993.msg64056850#msg64056850). Both updates got stuck, one was waiting for the other, which was waiting for Blockchair. In short: the list of all addresses with a balance is not updated at the moment.


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on May 22, 2024, 01:14:34 PM
I get this data from Blockchair.com (https://gz.blockchair.com/bitcoin/addresses/), and their last update is from April 19. Their other data (https://gz.blockchair.com/bitcoin/outputs/) is still being updated, so it looks like something is stuck with updating the Bitcoin addresses.
The daily updates are back to normal :)


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on May 31, 2024, 05:51:37 AM
I tried a lot and could not put to work these commands on linux mint
I've deleted your post (https://loyce.club/archive/posts/6415/64150155.html) because of the unnecessary long quote.

Your screenshot already shows the problem:
https://i.ibb.co/WyDr7Rp/Capture.jpg[/quote]

Quote
comm -12 Bitcoin_addresses_LATEST.txt <(cat wallet.txt | sort | uniq)
The file with balances is indeed not sorted. And you can't use it, because comm matches whole lines. Remove the balances and sort it, or download the file without balances.

Quote
comm -12 Bitcoin_addresses_LATEST.txt <(cat wallet.txt | fromdos | sort | uniq)
If you need fromdos, install tofrodos, as it tells you to do.

Running this as root is bad practice.


Title: Re: List of all Bitcoin addresses with a balance
Post by: pbies on June 01, 2024, 02:52:52 AM
I use these 3 scripts:

Get addresses and balances and unpack them:

Code:
#!/usr/bin/env bash
wget http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz
pv blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz | gunzip > blockchair_bitcoin_addresses_and_balance_LATEST.tsv
>&2 echo -ne "\x7"

Take only addresses, sort and uniq them:

Code:
#!/usr/bin/env bash
pv -B 1M blockchair_bitcoin_addresses_and_balance_LATEST.tsv | cut -f 1 | sort -u > addrs-with-bal.txt
>&2 echo -ne "\x7"

Compare two files, one with addresses and second your one with addresses you are searching for:

Code:
#!/usr/bin/env bash
comm -12 <(pv -cN 1st "$1") <(pv -cN 2nd "$2")
>&2 echo -ne "\x7"


Title: Re: List of all Bitcoin addresses with a balance
Post by: silaslyma on June 02, 2024, 12:57:39 AM
Compare two files, one with addresses and second your one with addresses you are searching for:

Code:
#!/usr/bin/env bash
comm -12 <(pv -cN 1st "$1") <(pv -cN 2nd "$2")
>&2 echo -ne "\x7"


Sorry, i do not know what i am doing wrong,
I tested with several addresses and only found 1

https://i.ibb.co/R3v2fJk/1.jpg

https://i.ibb.co/M9NXhgL/2.jpg


Title: Re: List of all Bitcoin addresses with a balance
Post by: silaslyma on June 02, 2024, 01:33:30 AM

Quote
comm -12 Bitcoin_addresses_LATEST.txt <(cat wallet.txt | fromdos | sort | uniq)
Quote
If you need fromdos, install tofrodos, as it tells you to do.

Now it worked. Thank you very much


Title: Re: List of all Bitcoin addresses with a balance
Post by: Mek on June 02, 2024, 06:32:34 AM
The list is currently a few days behind again :(


Title: Re: List of all Bitcoin addresses with a balance
Post by: LoyceV on June 02, 2024, 07:50:09 AM
The list is currently a few days behind again :(
If blockchair's data source (https://gz.blockchair.com/bitcoin/addresses/) is outdated, my data is outdated too. Unfortunately I don't have the disk space anymore to scrape my own blockchain.