LoyceV (OP)
Legendary
Offline
Activity: 3430
Merit: 17357
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 02, 2021, 02:03:40 PM |
|
Any news from the provider, when your server is coming back up? Nope, they're awefully quiet ![Sad](https://bitcointalk.org/Smileys/default/sad.gif) Do yo have temp location I can download files from ? I can boot up a pay-by-the-hour VPS for you and upload the files (recently updated, until July 26, 2021). I have 2 versions: 1. All Bitcoin addresses ever used, in chronological order, without duplicates. Sample: addresses_in_order_of_first_appearance.txt.gz: ( Warning: 18 GB): 1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa 12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX 1HLoD9E4SDFFPDiYfNYnkBLQ85Y51J3Zb1 ....... 3GFfFQAFgXKiA1qqUK6rqBpEpG4vZDos6t 3Mbtv47gZ2eN6Fy7owpgHHwSLYHS42P56P 38JyF2RQknBUMETyRT2yGndDJFYSp6hJNg 2. All Bitcoin addresses ever used, sorted by address, without duplicates. Sample: addresses_sorted.txt.gz: ( Warning: 16 GB): 1111111111111111111114oLvT2 111111111111111111112BEH2ro 111111111111111111112xT3273 ....... s-ffd80dee5966fb23c1a483b28f6bfcbc s-fff5d0faa9628c188e97661f0e185fce s-ffff291613d413b4ac128df96a462294 Which one would you prefer? The sorted version is much more practical for most uses, so unless you have a specific reason to want the addresses in chronological order, I'd say go for the sorted file.
|
|
|
|
mike2077
Newbie
Offline
Activity: 18
Merit: 10
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 03, 2021, 02:20:25 PM |
|
If you can and its not too hard for you, version 1 would be awesome! 1. All Bitcoin addresses ever used, in chronological order, without duplicates. Sample: addresses_in_order_of_first_appearance.txt.gz: (Warning: 18 GB): Code:
1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa 12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX 1HLoD9E4SDFFPDiYfNYnkBLQ85Y51J3Zb1 ....... 3GFfFQAFgXKiA1qqUK6rqBpEpG4vZDos6t 3Mbtv47gZ2eN6Fy7owpgHHwSLYHS42P56P 38JyF2RQknBUMETyRT2yGndDJFYSp6hJNg
Thanks. BTW, I think mega - mega.co.nz give you something like 50GB of storage for free.
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3430
Merit: 17357
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 03, 2021, 02:55:10 PM Last edit: June 05, 2021, 07:59:49 AM by LoyceV |
|
BTW, I think mega - mega.co.nz give you something like 50GB of storage for free. That's a terrible site, I used it once to download a large file, it forced me to install their program first. So I prefer a VPS. I'll let you know when it's available. Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again. Update: link expired.
|
|
|
|
NotATether
Legendary
Offline
Activity: 1722
Merit: 7248
In memory of o_e_l_e_o
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 04, 2021, 04:08:15 AM |
|
BTW, I think mega - mega.co.nz give you something like 50GB of storage for free. That's a terrible site, I used it once to download a large file, it forced me to install their program first. So I prefer a VPS. Most cloud storage sites can't upload files several GB large well without constantly breaking the connection, and will throttle the download speed even more which makes them unsuitable for downloading those files as well. Also it's not 50GB of free storage, it's much smaller than that. Most of that free storage is temporary and expires after a year.
|
|
|
|
PrimeNumber7
Copper Member
Legendary
Offline
Activity: 1652
Merit: 1900
Amazon Prime Member #7
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 04, 2021, 04:30:42 AM |
|
"Your download will take ~4 hours to complete" IMO, you should upload the file to a GCS/AWS/Azure/Oracle/etc storage bucket, set the permissions to "anyone can access" but set the object so that the "requestor pays" for downloads. This will result in you paying under a dollar per month in storage costs, but anyone who accesses your file will pay a few dollars to get your data in seconds. Maintainng a multigigabyte file that is accessible to the public for free, that can be accessed unlimited times is really not feasible.
|
|
|
|
NotATether
Legendary
Offline
Activity: 1722
Merit: 7248
In memory of o_e_l_e_o
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 04, 2021, 05:54:36 AM |
|
IMO, you should upload the file to a GCS/AWS/Azure/Oracle/etc storage bucket, set the permissions to "anyone can access" but set the object so that the "requestor pays" for downloads. This will result in you paying under a dollar per month in storage costs, but anyone who accesses your file will pay a few dollars to get your data in seconds.
Or you can just ask me nicely and I'll host it on my site's public directory.
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3430
Merit: 17357
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 04, 2021, 09:28:23 AM |
|
"Your download will take ~4 hours to complete" I get this (in England): - 0%[ ] 139.10M 33.9MB/s eta 10m 15s IMO, you should upload the file to a GCS/AWS/Azure/Oracle/etc storage bucket, set the permissions to "anyone can access" but set the object so that the "requestor pays" for downloads. This will result in you paying under a dollar per month in storage costs, but anyone who accesses your file will pay a few dollars to get your data in seconds. There's 2 problems with that: I don't want to use a creditcard, and I don't want anyone who downloads it to require a creditcard. If I need to charge a few dollars per download, I'd rather set it up myself to accept Bitcoin payments. Maintainng a multigigabyte file that is accessible to the public for free, that can be accessed unlimited times is really not feasible. My other project ( List of all Bitcoin addresses with a balance) is closing in on it's 2 TB montly bandwidth limit. I'd hate to have to setup a payment system, especially since this is basically just mirroring data from Blockchair.com. I've never used Torrent from CLI, that might work. It's a privacy problem for the user though: Torrent shares IP addresses get shared with other downloaders, so they need a VPN again. What bandwidth limitations do you have? I don't just want to make problem your problem.I remember I had another offer: For living we host high-end enterprise, just in case you need some space or mirrors, you’re welcome if ever in need. Is this offer still valid?
|
|
|
|
mike2077
Newbie
Offline
Activity: 18
Merit: 10
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 04, 2021, 02:29:59 PM |
|
BTW, I think mega - mega.co.nz give you something like 50GB of storage for free. That's a terrible site, I used it once to download a large file, it forced me to install their program first. So I prefer a VPS. I'll let you know when it's available. Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again. I've got the file, thank you so much for sharing.
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3430
Merit: 17357
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 04, 2021, 02:38:19 PM Last edit: June 05, 2021, 07:59:27 AM by LoyceV |
|
I've got the file, thank you so much for sharing. You're welcome ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif) Judging by the 64 GB outgoing traffic, the file was downloaded 3 times. If anyone else wants it: I'll nuke this VPS tomorrow. It's gone, until I find a more permanent solution.
|
|
|
|
PrimeNumber7
Copper Member
Legendary
Offline
Activity: 1652
Merit: 1900
Amazon Prime Member #7
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 04, 2021, 06:53:48 PM |
|
"Your download will take ~4 hours to complete" I get this (in England): - 0%[ ] 139.10M 33.9MB/s eta 10m 15s The 4 hour quote appears to be the result of my crappy WiFi connection on my back porch. I was able to reproduce a ~10 minute download estimate via a datacenter. I have been able to transfer ~a half terabyte worth of videos stored in a storage bucket in seconds. IMO, you should upload the file to a GCS/AWS/Azure/Oracle/etc storage bucket, set the permissions to "anyone can access" but set the object so that the "requestor pays" for downloads. This will result in you paying under a dollar per month in storage costs, but anyone who accesses your file will pay a few dollars to get your data in seconds. There's 2 problems with that: I don't want to use a creditcard, and I don't want anyone who downloads it to require a creditcard. If I need to charge a few dollars per download, I'd rather set it up myself to accept Bitcoin payments. That is a reasonable desire, however it is something that is more difficult as you are making big data available to the public. Service providers have limited network infrastructure, and need to pay for data sent to the internet, regardless of if they have data caps, or charge you for egress/outgoing data. If you are freely sharing a 10 or 20 GB file(s) using a service provider that does not charge per data transferred, you will eventually get kicked off from that service provider. Another point is that many people look at bitcoin-related data today. The fact that someone is looking at blockchain data is not the privacy leak that it might have been 10 years ago. Maintainng a multigigabyte file that is accessible to the public for free, that can be accessed unlimited times is really not feasible. My other project ( List of all Bitcoin addresses with a balance) is closing in on it's 2 TB montly bandwidth limit. I'd hate to have to setup a payment system, especially since this is basically just mirroring data from Blockchair.com. There is a reason why blockchair throttles downloads, and why they charge as much as they do for an API key.
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3430
Merit: 17357
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 05, 2021, 08:07:25 AM |
|
Service providers have limited network infrastructure, and need to pay for data sent to the internet, regardless of if they have data caps, or charge you for egress/outgoing data. If you are freely sharing a 10 or 20 GB file(s) using a service provider that does not charge per data transferred, you will eventually get kicked off from that service provider. Obviously, I won't use a host with "unlimited" bandwidth. That's never real. And I've seen shared hosts that don't allow hosting large files. But for a VPS, I pay for the bandwidth limit, and it's up to the provider to ensure it's profitable for them. Another point is that many people look at bitcoin-related data today. The fact that someone is looking at blockchain data is not the privacy leak that it might have been 10 years ago. I've never used a creditcard for anything crypto-related, and I want to keep it that way. I found another reason to pay upfront in crypto instead of using my creditcard: The tale of the July 4th surprise $2700 AWS bill.There is a reason why blockchair throttles downloads, and why they charge as much as they do for an API key. They're also in the money making business, and their paying customers pay for the bandwidth used by, well, people like me ![Tongue](https://bitcointalk.org/Smileys/default/tongue.gif)
|
|
|
|
NotATether
Legendary
Offline
Activity: 1722
Merit: 7248
In memory of o_e_l_e_o
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 05, 2021, 02:45:47 PM |
|
What bandwidth limitations do you have? I don't just want to make problem your problem.My provider gives me a cool 100TB monthly bandwidth cap so I will be fine ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif) these boxes are designed for highly intensive torrenting.
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3430
Merit: 17357
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 05, 2021, 06:29:35 PM |
|
My provider gives me a cool 100TB monthly bandwidth cap so I will be fine ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif) Interesting ![Cheesy](https://bitcointalk.org/Smileys/default/cheesy.gif) Can it also handle the occasional update? I have "2 methods" now: one that takes a long time but works, and one that's much faster but gives different results once in a while. And because testing takes so long, I haven't found the problem yet.
|
|
|
|
NotATether
Legendary
Offline
Activity: 1722
Merit: 7248
In memory of o_e_l_e_o
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 06, 2021, 06:06:14 AM |
|
Can it also handle the occasional update? I have "2 methods" now: one that takes a long time but works, and one that's much faster but gives different results once in a while. And because testing takes so long, I haven't found the problem yet.
Depends on how much time is "long time". Leaving it single threaded (as most shell commands already are) will probably be alright as long as it doesn't take more than a few hours. But if it has obscene memory requirements then it'll be too much for my box. I only got 9 out of 16GB ram free and I hate force-rebooting the notatether.com webserver ![Undecided](https://bitcointalk.org/Smileys/default/undecided.gif)
|
|
|
|
PrimeNumber7
Copper Member
Legendary
Offline
Activity: 1652
Merit: 1900
Amazon Prime Member #7
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 06, 2021, 07:14:23 AM |
|
Can it also handle the occasional update? I have "2 methods" now: one that takes a long time but works, and one that's much faster but gives different results once in a while. And because testing takes so long, I haven't found the problem yet.
Depends on how much time is "long time". Leaving it single threaded (as most shell commands already are) will probably be alright as long as it doesn't take more than a few hours. But if it has obscene memory requirements then it'll be too much for my box. I only got 9 out of 16GB ram free and I hate force-rebooting the notatether.com webserver ![Undecided](https://bitcointalk.org/Smileys/default/undecided.gif) Memory/thread constraints should not be an issue. Executing a script remotely on a server optimized for script requirements is trivial, and uploading an output file a single time to your server should not be an issue for a ~20 GB file.
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3430
Merit: 17357
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 06, 2021, 02:29:51 PM |
|
Leaving it single threaded (as most shell commands already are) will probably be alright as long as it doesn't take more than a few hours. But if it has obscene memory requirements then it'll be too much for my box. I only got 9 out of 16GB ram free and I hate force-rebooting the notatether.com webserver ![Undecided](https://bitcointalk.org/Smileys/default/undecided.gif) It uses a bunch of pipes, so the load it causes is more like 2-3 than 1. Memory requirements can be low (then sort uses more tmp files instead). It should be done within a few hours, and doesn't need frequent updates (once every 2 weeks will be fine).
|
|
|
|
NotATether
Legendary
Offline
Activity: 1722
Merit: 7248
In memory of o_e_l_e_o
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 06, 2021, 06:51:54 PM |
|
Memory requirements can be low (then sort uses more tmp files instead).
This is what I'm worried about. I remember you writing somewhere along the lines of the sort process takes an obscene amount of memory on this file, or maybe that was in the addresses-with-a-balance project. That's why I'm trying to figure out how much RAM it uses in the worst case.
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3430
Merit: 17357
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 06, 2021, 08:01:35 PM |
|
I'm trying to figure out how much RAM it uses in the worst case. I can limit sort's memory usage. I'm more worried about the grinding this causes on the hard drive. I have no idea how much data gets read and written to sort 30 GB, but I assume every bit gets pickup up at least several times.
|
|
|
|
NotATether
Legendary
Offline
Activity: 1722
Merit: 7248
In memory of o_e_l_e_o
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 12, 2021, 07:59:14 AM |
|
I'm trying to figure out how much RAM it uses in the worst case. I can limit sort's memory usage. I'm more worried about the grinding this causes on the hard drive. I have no idea how much data gets read and written to sort 30 GB, but I assume every bit gets pickup up at least several times. On second thought, I just had one of my servers' disks fail a couple days ago (all data was lost), so I'm not comfortable running these updating scripts on the rest of my hardware with all that grinding until I can set up a proper backup plan for my TBs of data.
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3430
Merit: 17357
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 12, 2021, 08:16:31 AM |
|
I'm not comfortable running these updating scripts on the rest of my hardware No worries, I'll just wait for the right hosting offer again. This is the reason I don't want to do a lot of testing on my laptop SSD too.
|
|
|
|
|