LoyceV (OP)
Legendary
Offline
Activity: 3304
Merit: 16637
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
June 11, 2020, 03:30:34 PM Last edit: August 31, 2023, 01:34:20 PM by LoyceV Merited by OmegaStarScream (12), PawGo (6), yogg (5), bitmover (5), mugt (5), suchmoon (4), Welsh (4), vapourminer (3), klarki (3), albert0bsd (3), Symmetrick (3), BrewMaster (2), BTCW (2), blauervogel (2), Cyrus (1), Coin-1 (1), Porfirii (1), citb0in (1), mynonce (1), COVID-19 (1), npuath (1) |
|
BackgroundThere's another very useful data dump on Blockchair that deserves attention: blockchair_bitcoin_addresses_latest.tsv.gz! On many occasions I found myself searching for a complete snapshot of all funded addresses, for instance to search for a balance when you only have a partial Bitcoin address. This file takes almost a day to download at 10 kB/s. The dataSee addresses.loyce.club. I keep 18 snapshots of Blockchair's daily data. Sample: blockchair_bitcoin_addresses_and_balance_DATE.tsv.gz (sorted by balance, highest first (in satoshis)): address balance 35hK24tcLEWcgNA4JxpvbkNkoAcDGqQPsP 25550215765875 3KZ526NxCVXbKwwP66RgM3pte6zW4gY1tD 10185724750535 37XuVSEpWW4trkfmvWzegTHQt7BdktSKUs 9450577254951 ....... m-3165957a315e3d9d2de76eccb1140cb8 1 127TnYq7APW8WfKewd7EdxA8gMUXEtr623 1 1E6NkSVsBewyz8Z8wJBYVTKgWmdqcUSWkS 1 This file is in TSV format. It's probably too large (30 million rows) to import into a spreadsheet. I create a new file with all funded addresses, without balances, and without "weird" addresses (the ones with "-" in it). Sample: Bitcoin_addresses_DATE.txt.gz (sorted in alphabetical order): 1111111111111111111114oLvT2 111111111111111111112BEH2ro 111111111111111111112xT3273 ....... bc1qzzzz6hthvpjgtl9pgqyepwwzfrky2ntmm5ccpc bc1qzzzzp3khtxe32q03qfm6epm5ytyytq7lfcakpn bc1zqyqsywvzqe This file is in TEXT format. The first addresses are burn addresses, nobody can access those funds. The (incomplete) last address somehow really got funded. Download speed should be around 1000 times faster than Blockchair. I don't offer uncompressed downloads. How to useThe most likely use is to check a long list of Bitcoin addresses for a remaining balance. On Linux, use this to find matching addresses (after extrating the compressed .gz file of course): comm -12 Bitcoin_addresses_LATEST.txt <(cat mylist.txt | sort | uniq) - Bitcoin_addresses_LATEST.txt: the extracted latest version downloaded from addresses.loyce.club.
- mylist.txt: your own list of addresses, one address per line.
This takes only seconds to check millions of addresses. If your text file has Microsoft formatting, you may need to use this instead: comm -12 Bitcoin_addresses_LATEST.txt <(cat mylist.txt | fromdos | sort | uniq) If you want the balances of many addresses: grep -f mylist.txt blockchair_bitcoin_addresses_and_balance_LATEST.tsv This is a bit slower and eats RAM: 500,000 input addresses uses 1.5 GB and took 50 seconds. You could mix it with the earlier command to only search the balance for addresses that aren't empty. If you want to know the balance of a certain address, but you don't want any block explorer or SPV wallet on the planet to know you're looking for that address, you can use this: wget -qO- http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz | gunzip | grep -m1 1NXYoJ5xU91Jp83XfVMHwwTUyZFK64BoAD; sleep 5m; kill -9 $$ This doesn't save the file on your local drive, stops downloading after the first match, and kills your Bash shell 5 minutes later (so no .bash_history gets saved). Note (January 12, 2021): I now realize my webserver could know when you stop downloading. If you don't want this, remove " -m1" from grep. If you want to search more than one address, please download the file first. If you want to create a brainflayer bloom filter: pbies wrote a guide. Data retention and updatesI'll provide daily updates. I keep the latest 6 daily snapshots, and the latest 12 monthly snapshots. One a day (or once a month) I delete the oldest files. New: Direct link to LATEST versionsblockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz (currently 862MB) Bitcoin_addresses_LATEST.txt.gz (currently 750 MB) New: Keeping track of the total number of funded addressesSee total_number_of_funded_addresses.txt. Starting December 2020, I'll add a daily total address count to this list. Long-term, this might prove useful (or at least produce nice graphs). CreditsBlockchair Database Dumps has a staggering amount of data, easily accessible (at 10 kB/s) with daily updates. All data presented in this topic comes from Blockchair. No spam please. Self-moderated against spam. Discussion and questions are welcome.
Related topicsBitcoin block data available in CSV formatList of all Bitcoin addresses with a balanceList of all Bitcoin addresses ever used[~500 GB] Bitcoin block data: inputs, outputs and transactions[800 GB] Ethereum data
|
|
|
|
|
|
|
|
|
Even if you use Bitcoin through Tor, the way transactions are handled by the network makes anonymity difficult to achieve. Do not expect your transactions to be anonymous unless you really know what you're doing.
|
|
|
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
|
|
|
|
BrewMaster
Legendary
Offline
Activity: 2114
Merit: 1292
There is trouble abrewing
|
how did you download these files? i've tried a couple of times using my cloud downloader but each time it failed near the end. i suspected the file might have been removed or changed since it took nearly a day to download. I'm not sure what the weird "address" format at the end is.
for some reason blockchair.com assigns these weird format addresses to any outscript that they see. this includes OP_RETURN outputs too which can be found in almost all of the coinbase transactions. the last one in your list for example is a simple P2PKH script that starts with a garbage being pushed to the stack followed by an OP_DROP which makes the pushto virtually not-exist. https://blockchair.com/bitcoin/transaction/48e7694b3de213e2ab36cfcfaaa3013a22e1a5dc28168772cb01be99f91a12fa
|
There is a FOMO brewing...
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3304
Merit: 16637
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
June 12, 2020, 09:15:33 AM |
|
how did you download these files? wgeti've tried a couple of times using my cloud downloader but each time it failed near the end. i suspected the file might have been removed or changed since it took nearly a day to download. As long as your download doesn't disconnect, I assume the file you're downloading doesn't change. However, one of my downloads got corrupted, so I now test each new .gz-file before sharing it, and before deleting the oldest one. I thought it must have been OP_RETURN indeed, but didn't look into it. As an easy "solution", I'll remove the 397,175 "addresses" that contain a "-" from Bitcoin_addresses_DATE.txt.gz. I won't change existing files for this, so updates start in about 2 days.
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3304
Merit: 16637
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
June 15, 2020, 09:25:06 AM Last edit: July 08, 2020, 11:28:45 AM by LoyceV |
|
In Ratimov's Russian translation, FontSeli asked how this can be useful for a normal person (thanks Google Translate). I'll answer this here. I can think of several other uses: - Say you have a list of your old Bitcoin addresses and want to know if they hold any balance. But you don't want to let any block explorer or Electrum server know which address you're looking at. By downloading all addresses, you can easily do a local search without anyone else knowing which address you're looking for.
- I've also seen cases where someone has an incomplete Bitcoin address, and wants to know if it's worth to pursue recovering the private key. Block explorers don't let you search for incomplete addresses. As a rediculous example, here are all funded addresses that contain "Loyce":
1LoyceVSbrv3CP3jT12qVYrNSbrqz4FL5q 1499000 1PKDzdoRNoP1LoyceZLT1VwNsvP2GBuxk6 100000 1F9Q7dzMyFBkrNQvdo4ECLL5DvLoycewSw 71917 17YTjpKVtRLAVP6igaztot5Loycegd5igR 70701 1MakvXVkoAGJXhbbTEAM3Loyce6LibBaEQ 31590
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3304
Merit: 16637
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
June 21, 2020, 11:57:57 AM |
|
still the choosing of any block explorer to pursue the addresses of your interest would lead to swifter result. All out good VPN would help to hide from the curiosity expressed by those who are behind the block explorer. For many users that will be true. I just offer one more option
|
|
|
|
bitmover
Legendary
Offline
Activity: 2296
Merit: 5942
bitcoindata.science
|
|
June 30, 2020, 02:41:59 AM |
|
Nice job LoyceV.
You are good with these kind of data.
I am searching for historical volume data from exchanges, BTC pairs. I tried in coinmarketcap but I couldn't find there. Do you have any idea where I can find it?
|
. .BLACKJACK ♠ FUN. | | | ███▄██████ ██████████████▀ ████████████ █████████████████ ████████████████▄▄ ░█████████████▀░▀▀ ██████████████████ ░██████████████ █████████████████▄ ░██████████████▀ ████████████ ███████████████░██ ██████████ | | CRYPTO CASINO & SPORTS BETTING | | │ | | │ | ▄▄███████▄▄ ▄███████████████▄ ███████████████████ █████████████████████ ███████████████████████ █████████████████████████ █████████████████████████ █████████████████████████ ███████████████████████ █████████████████████ ███████████████████ ▀███████████████▀ ███████████████████ | | .
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3304
Merit: 16637
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
June 30, 2020, 10:05:53 AM |
|
I am searching for historical volume data from exchanges, BTC pairs. I tried in coinmarketcap but I couldn't find there. Do you have any idea where I can find it? I haven't seen this data yet. I've seen sites that sell historic data, but I'm not sure if they include exchange volumes. And even if you do find it, the data must have come from the exchanges themselves, and some of them are known to provide fake data.
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3304
Merit: 16637
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
July 08, 2020, 11:33:57 AM Last edit: July 20, 2020, 08:36:54 AM by LoyceV Merited by vapourminer (1), malevolent (1) |
|
I can think of several other uses: I thought of another use: what if you wanted to know how many addresses might be ChipMixer chips? They have specific sizes of 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 or 8192 m BTC. A quick search (using yesterday's address list) shows: 1 m BTC: 266408 addresses 2 m BTC: 66301 addresses 4 m BTC: 30343 addresses 8 m BTC: 15695 addresses 16 m BTC: 8290 addresses 32 m BTC: 3894 addresses 64 m BTC: 1797 addresses 128 m BTC: 1167 addresses 256 m BTC: 595 addresses 512 m BTC: 259 addresses 1024 m BTC: 442 addresses 2048 m BTC: 120 addresses 4096 m BTC: 139 addresses 8192 m BTC: 59 addresses
|
|
|
|
bitwise
Newbie
Offline
Activity: 5
Merit: 5
|
AFAIK you don't need to download this if you run a node on any bitcoind node you can run dumptxoutset command and then just parse it and when https://github.com/bitcoin/bitcoin/pull/18689/ gets merged, you won't even need to parse it
|
|
|
|
btc_enigma
|
|
September 24, 2020, 09:43:56 AM |
|
AFAIK you don't need to download this if you run a node on any bitcoind node you can run dumptxoutset command and then just parse it and when https://github.com/bitcoin/bitcoin/pull/18689/ gets merged, you won't even need to parse it This feature would be really nice. Thanks for highlighting it
|
|
|
|
alevlaslo
|
|
October 19, 2020, 06:40:49 AM |
|
Hello, please divide the addresses into three groups, we can not import different types into the Vanitysearch program at the same time
|
Sale the first NFT of the first foto
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3304
Merit: 16637
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
October 19, 2020, 10:11:37 AM |
|
Hello, please divide the addresses into three groups I won't add different file formats for specific uses (because of restrictions on disk space and server load). You're free to do that yourself though, it's quite easy: For all Bech32 addresses: wget http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_October_19_2020.tsv.gz -O - | gunzip | grep "^b" > Bech32_bitcoin_addresses_and_balance_October_19_2020.tsv Replace the "b" by "1" or "3" for other addresses, and adjust the file date for the latest version.
I'm glad to see this addresses subdomain consumes more data than the rest of loyce.club combined, that means it's being used
|
|
|
|
alevlaslo
|
|
October 19, 2020, 11:37:06 AM |
|
thank you, but I have Windows, where do I need to set this command and what program to install?
|
Sale the first NFT of the first foto
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3304
Merit: 16637
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
October 19, 2020, 11:54:21 AM |
|
I have Windows, where do I need to set this command and what program to install? I have no idea So I got you this instead: http://alladdresses.loyce.club:20319/tmp/It won't be updated, and it's scheduled to be deleted in 7 days.
|
|
|
|
alevlaslo
|
|
October 19, 2020, 12:13:45 PM |
|
thank you very much!
|
Sale the first NFT of the first foto
|
|
|
alevlaslo
|
|
October 19, 2020, 01:03:12 PM |
|
how do I remove extra characters? The program does not accept with them, each line only must has an address only, there should be nothing else
|
Sale the first NFT of the first foto
|
|
|
bitmover
Legendary
Offline
Activity: 2296
Merit: 5942
bitcoindata.science
|
|
October 19, 2020, 02:02:29 PM Last edit: October 19, 2020, 02:31:08 PM by bitmover |
|
how do I remove extra characters? The program does not accept with them, each line only must has an address only, there should be nothing else
Which program are you using? You will struggle a lot to manipulate this kind of data in apps such as excel. personally, I could only open that 1GB files in my python environment. All other programs just froze Edit: I was able to open the larger file in Notepad++.
|
. .BLACKJACK ♠ FUN. | | | ███▄██████ ██████████████▀ ████████████ █████████████████ ████████████████▄▄ ░█████████████▀░▀▀ ██████████████████ ░██████████████ █████████████████▄ ░██████████████▀ ████████████ ███████████████░██ ██████████ | | CRYPTO CASINO & SPORTS BETTING | | │ | | │ | ▄▄███████▄▄ ▄███████████████▄ ███████████████████ █████████████████████ ███████████████████████ █████████████████████████ █████████████████████████ █████████████████████████ ███████████████████████ █████████████████████ ███████████████████ ▀███████████████▀ ███████████████████ | | .
|
|
|
|
alevlaslo
|
|
October 19, 2020, 03:37:14 PM |
|
I have Notepad++, but I don't have enough RAM, it freezes after selecting all the text and ALT+SHIFT+>>>
|
Sale the first NFT of the first foto
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3304
Merit: 16637
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
October 19, 2020, 04:05:01 PM |
|
how do I remove extra characters? You mean remove the balance? I just use cut. I got you 3 new files with only addresses.
|
|
|
|
alevlaslo
|
|
October 19, 2020, 06:35:31 PM |
|
thank you, I already figured it out, I have an old hard drive, I had to wait 1 hour.
|
Sale the first NFT of the first foto
|
|
|
|