List of all Bitcoin addresses ever used - weekly updates work again!

[LoyceV]

Any news from the provider, when your server is coming back up?

Nope, they're awefully quiet

Do yo have temp location I can download files from ?

I can boot up a pay-by-the-hour VPS for you and upload the files (recently updated, until July 26, 2021).
I have 2 versions:

1. All Bitcoin addresses ever used, in chronological order, without duplicates.
Sample: addresses_in_order_of_first_appearance.txt.gz: (Warning: 18 GB):

Code:

1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa
12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX
1HLoD9E4SDFFPDiYfNYnkBLQ85Y51J3Zb1
.......
3GFfFQAFgXKiA1qqUK6rqBpEpG4vZDos6t
3Mbtv47gZ2eN6Fy7owpgHHwSLYHS42P56P
38JyF2RQknBUMETyRT2yGndDJFYSp6hJNg

2. All Bitcoin addresses ever used, sorted by address, without duplicates.
Sample: addresses_sorted.txt.gz: (Warning: 16 GB):

Code:

1111111111111111111114oLvT2
111111111111111111112BEH2ro
111111111111111111112xT3273
.......
s-ffd80dee5966fb23c1a483b28f6bfcbc
s-fff5d0faa9628c188e97661f0e185fce
s-ffff291613d413b4ac128df96a462294

Which one would you prefer? The sorted version is much more practical for most uses, so unless you have a specific reason to want the addresses in chronological order, I'd say go for the sorted file.

[mike2077]

If you can and its not too hard for you, version 1 would be awesome!

1. All Bitcoin addresses ever used, in chronological order, without duplicates.
Sample: addresses_in_order_of_first_appearance.txt.gz: (Warning: 18 GB):
Code:

1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa
12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX
1HLoD9E4SDFFPDiYfNYnkBLQ85Y51J3Zb1
.......
3GFfFQAFgXKiA1qqUK6rqBpEpG4vZDos6t
3Mbtv47gZ2eN6Fy7owpgHHwSLYHS42P56P
38JyF2RQknBUMETyRT2yGndDJFYSp6hJNg

Thanks.

BTW,  I think mega - mega.co.nz give you something like 50GB of storage for free.

[LoyceV]

BTW,  I think mega - mega.co.nz give you something like 50GB of storage for free.

That's a terrible site, I used it once to download a large file, it forced me to install their program first. So I prefer a VPS.
I'll let you know when it's available.

Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again.
Update: link expired.

[NotATether]

BTW,  I think mega - mega.co.nz give you something like 50GB of storage for free.

That's a terrible site, I used it once to download a large file, it forced me to install their program first. So I prefer a VPS.

Most cloud storage sites can't upload files several GB large well without constantly breaking the connection, and will throttle the download speed even more which makes them unsuitable for downloading those files as well.

Also it's not 50GB of free storage, it's much smaller than that. Most of that free storage is temporary and expires after a year.

[PrimeNumber7]

Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again.

"Your download will take ~4 hours to complete"

IMO, you should upload the file to a GCS/AWS/Azure/Oracle/etc storage bucket, set the permissions to "anyone can access" but set the object so that the "requestor pays" for downloads. This will result in you paying under a dollar per month in storage costs, but anyone who accesses your file will pay a few dollars to get your data in seconds.

Maintainng a multigigabyte file that is accessible to the public for free, that can be accessed unlimited times is really not feasible.

[NotATether]

IMO, you should upload the file to a GCS/AWS/Azure/Oracle/etc storage bucket, set the permissions to "anyone can access" but set the object so that the "requestor pays" for downloads. This will result in you paying under a dollar per month in storage costs, but anyone who accesses your file will pay a few dollars to get your data in seconds. 

Or you can just ask me nicely and I'll host it on my site's public directory.

[LoyceV]

Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again.

"Your download will take ~4 hours to complete"

I get this (in England):

Code:

-                     0%[                    ] 139.10M  33.9MB/s    eta 10m 15s

IMO, you should upload the file to a GCS/AWS/Azure/Oracle/etc storage bucket, set the permissions to "anyone can access" but set the object so that the "requestor pays" for downloads. This will result in you paying under a dollar per month in storage costs, but anyone who accesses your file will pay a few dollars to get your data in seconds.

There's 2 problems with that: I don't want to use a creditcard, and I don't want anyone who downloads it to require a creditcard. If I need to charge a few dollars per download, I'd rather set it up myself to accept Bitcoin payments.

Maintainng a multigigabyte file that is accessible to the public for free, that can be accessed unlimited times is really not feasible.

My other project (List of all Bitcoin addresses with a balance) is closing in on it's 2 TB montly bandwidth limit. I'd hate to have to setup a payment system, especially since this is basically just mirroring data from Blockchair.com. I've never used Torrent from CLI, that might work. It's a privacy problem for the user though: Torrent shares IP addresses get shared with other downloaders, so they need a VPN again.

Or you can just ask me nicely and I'll host it on my site's public directory.

What bandwidth limitations do you have? I don't just want to make problem your problem.

I remember I had another offer:

For living we host high-end enterprise, just in case you need some space or mirrors, you’re welcome if ever in need.

Is this offer still valid?

[mike2077]

BTW,  I think mega - mega.co.nz give you something like 50GB of storage for free.

That's a terrible site, I used it once to download a large file, it forced me to install their program first. So I prefer a VPS.
I'll let you know when it's available.

Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again.

I've got the file, thank you so much for sharing.

[LoyceV]

I've got the file, thank you so much for sharing.

You're welcome

Judging by the 64 GB outgoing traffic, the file was downloaded 3 times. If anyone else wants it:

I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB.

I'll nuke this VPS tomorrow. It's gone, until I find a more permanent solution.

[PrimeNumber7]

Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again.

"Your download will take ~4 hours to complete"

I get this (in England):

Code:

-                     0%[                    ] 139.10M  33.9MB/s    eta 10m 15s

The 4 hour quote appears to be the result of my crappy WiFi connection on my back porch. I was able to reproduce a ~10 minute download estimate via a datacenter. I have been able to transfer ~a half terabyte worth of videos stored in a storage bucket in seconds.

IMO, you should upload the file to a GCS/AWS/Azure/Oracle/etc storage bucket, set the permissions to "anyone can access" but set the object so that the "requestor pays" for downloads. This will result in you paying under a dollar per month in storage costs, but anyone who accesses your file will pay a few dollars to get your data in seconds.

There's 2 problems with that: I don't want to use a creditcard, and I don't want anyone who downloads it to require a creditcard. If I need to charge a few dollars per download, I'd rather set it up myself to accept Bitcoin payments.

That is a reasonable desire, however it is something that is more difficult as you are making big data available to the public. Service providers have limited network infrastructure, and need to pay for data sent to the internet, regardless of if they have data caps, or charge you for egress/outgoing data. If you are freely sharing a 10 or 20 GB file(s) using a service provider that does not charge per data transferred, you will eventually get kicked off from that service provider.

Another point is that many people look at bitcoin-related data today. The fact that someone is looking at blockchain data is not the privacy leak that it might have been 10 years ago.

Maintainng a multigigabyte file that is accessible to the public for free, that can be accessed unlimited times is really not feasible.

My other project (List of all Bitcoin addresses with a balance) is closing in on it's 2 TB montly bandwidth limit. I'd hate to have to setup a payment system, especially since this is basically just mirroring data from Blockchair.com.

There is a reason why blockchair throttles downloads, and why they charge as much as they do for an API key.

[LoyceV]

Service providers have limited network infrastructure, and need to pay for data sent to the internet, regardless of if they have data caps, or charge you for egress/outgoing data. If you are freely sharing a 10 or 20 GB file(s) using a service provider that does not charge per data transferred, you will eventually get kicked off from that service provider.

Obviously, I won't use a host with "unlimited" bandwidth. That's never real. And I've seen shared hosts that don't allow hosting large files. But for a VPS, I pay for the bandwidth limit, and it's up to the provider to ensure it's profitable for them.

Another point is that many people look at bitcoin-related data today. The fact that someone is looking at blockchain data is not the privacy leak that it might have been 10 years ago.

I've never used a creditcard for anything crypto-related, and I want to keep it that way.
I found another reason to pay upfront in crypto instead of using my creditcard: The tale of the July 4th surprise $2700 AWS bill.

There is a reason why blockchair throttles downloads, and why they charge as much as they do for an API key.

They're also in the money making business, and their paying customers pay for the bandwidth used by, well, people like me

[NotATether]

Or you can just ask me nicely and I'll host it on my site's public directory.

What bandwidth limitations do you have? I don't just want to make problem your problem.

My provider gives me a cool 100TB monthly bandwidth cap so I will be fine  these boxes are designed for highly intensive torrenting.

[LoyceV]

My provider gives me a cool 100TB monthly bandwidth cap so I will be fine 

Interesting
Can it also handle the occasional update? I have "2 methods" now: one that takes a long time but works, and one that's much faster but gives different results once in a while. And because testing takes so long, I haven't found the problem yet.

[NotATether]

Can it also handle the occasional update? I have "2 methods" now: one that takes a long time but works, and one that's much faster but gives different results once in a while. And because testing takes so long, I haven't found the problem yet.

Depends on how much time is "long time".

Leaving it single threaded (as most shell commands already are) will probably be alright as long as it doesn't take more than a few hours.

But if it has obscene memory requirements then it'll be too much for my box. I only got 9 out of 16GB ram free and I hate force-rebooting the notatether.com webserver  

[PrimeNumber7]

Can it also handle the occasional update? I have "2 methods" now: one that takes a long time but works, and one that's much faster but gives different results once in a while. And because testing takes so long, I haven't found the problem yet.

Depends on how much time is "long time".

Leaving it single threaded (as most shell commands already are) will probably be alright as long as it doesn't take more than a few hours.

But if it has obscene memory requirements then it'll be too much for my box. I only got 9 out of 16GB ram free and I hate force-rebooting the notatether.com webserver 

Memory/thread constraints should not be an issue. Executing a script remotely on a server optimized for script requirements is trivial, and uploading an output file a single time to your server should not be an issue for a ~20 GB file.

[LoyceV]

Leaving it single threaded (as most shell commands already are) will probably be alright as long as it doesn't take more than a few hours.

But if it has obscene memory requirements then it'll be too much for my box. I only got 9 out of 16GB ram free and I hate force-rebooting the notatether.com webserver  

It uses a bunch of pipes, so the load it causes is more like 2-3 than 1. Memory requirements can be low (then sort uses more tmp files instead). It should be done within a few hours, and doesn't need frequent updates (once every 2 weeks will be fine).

[NotATether]

Memory requirements can be low (then sort uses more tmp files instead).

This is what I'm worried about. I remember you writing somewhere along the lines of the sort process takes an obscene amount of memory on this file, or maybe that was in the addresses-with-a-balance project. That's why I'm trying to figure out how much RAM it uses in the worst case.

[LoyceV]

I'm trying to figure out how much RAM it uses in the worst case.

I can limit sort's memory usage. I'm more worried about the grinding this causes on the hard drive. I have no idea how much data gets read and written to sort 30 GB, but I assume every bit gets pickup up at least several times.

[NotATether]

I'm trying to figure out how much RAM it uses in the worst case.

I can limit sort's memory usage. I'm more worried about the grinding this causes on the hard drive. I have no idea how much data gets read and written to sort 30 GB, but I assume every bit gets pickup up at least several times.

On second thought, I just had one of my servers' disks fail a couple days ago (all data was lost), so I'm not comfortable running these updating scripts on the rest of my hardware with all that grinding until I can set up a proper backup plan for my TBs of data.

[LoyceV]

I'm not comfortable running these updating scripts on the rest of my hardware

No worries, I'll just wait for the right hosting offer again.
This is the reason I don't want to do a lot of testing on my laptop SSD too.