Bitcoin Forum
March 28, 2024, 09:42:56 AM *
News: Latest Bitcoin Core release: 26.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 2 3 4 [5] 6 7 8 »  All
  Print  
Author Topic: List of all Bitcoin addresses ever used - weekly updates work again!  (Read 3587 times)
This is a self-moderated topic. If you do not want to be moderated by the person who started this topic, create a new topic. (3 posts by 1+ user deleted.)
LoyceV (OP)
Legendary
*
Offline Offline

Activity: 3262
Merit: 16315


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
June 02, 2021, 02:03:40 PM
 #81

Any news from the provider, when your server is coming back up?
Nope, they're awefully quiet Sad

Quote
Do yo have temp location I can download files from ?
I can boot up a pay-by-the-hour VPS for you and upload the files (recently updated, until July 26, 2021).
I have 2 versions:
1. All Bitcoin addresses ever used, in chronological order, without duplicates.
Sample: addresses_in_order_of_first_appearance.txt.gz: (Warning: 18 GB):
Code:
1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa
12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX
1HLoD9E4SDFFPDiYfNYnkBLQ85Y51J3Zb1
.......
3GFfFQAFgXKiA1qqUK6rqBpEpG4vZDos6t
3Mbtv47gZ2eN6Fy7owpgHHwSLYHS42P56P
38JyF2RQknBUMETyRT2yGndDJFYSp6hJNg

2. All Bitcoin addresses ever used, sorted by address, without duplicates.
Sample: addresses_sorted.txt.gz: (Warning: 16 GB):
Code:
1111111111111111111114oLvT2
111111111111111111112BEH2ro
111111111111111111112xT3273
.......
s-ffd80dee5966fb23c1a483b28f6bfcbc
s-fff5d0faa9628c188e97661f0e185fce
s-ffff291613d413b4ac128df96a462294
Which one would you prefer? The sorted version is much more practical for most uses, so unless you have a specific reason to want the addresses in chronological order, I'd say go for the sorted file.

█▀▀▀











█▄▄▄
▀▀▀▀▀▀▀▀▀▀▀
e
▄▄▄▄▄▄▄▄▄▄▄
█████████████
████████████▄███
██▐███████▄█████▀
█████████▄████▀
███▐████▄███▀
████▐██████▀
█████▀█████
███████████▄
████████████▄
██▄█████▀█████▄
▄█████████▀█████▀
███████████▀██▀
████▀█████████
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
c.h.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
▀▀▀█











▄▄▄█
▄██████▄▄▄
█████████████▄▄
███████████████
███████████████
███████████████
███████████████
███░░█████████
███▌▐█████████
█████████████
███████████▀
██████████▀
████████▀
▀██▀▀
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
mike2077
Newbie
*
Offline Offline

Activity: 18
Merit: 10


View Profile
June 03, 2021, 02:20:25 PM
 #82

If you can and its not too hard for you, version 1 would be awesome!

1. All Bitcoin addresses ever used, in chronological order, without duplicates.
Sample: addresses_in_order_of_first_appearance.txt.gz: (Warning: 18 GB):
Code:

1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa
12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX
1HLoD9E4SDFFPDiYfNYnkBLQ85Y51J3Zb1
.......
3GFfFQAFgXKiA1qqUK6rqBpEpG4vZDos6t
3Mbtv47gZ2eN6Fy7owpgHHwSLYHS42P56P
38JyF2RQknBUMETyRT2yGndDJFYSp6hJNg

Thanks.

BTW,  I think mega - mega.co.nz give you something like 50GB of storage for free.
LoyceV (OP)
Legendary
*
Offline Offline

Activity: 3262
Merit: 16315


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
June 03, 2021, 02:55:10 PM
Last edit: June 05, 2021, 07:59:49 AM by LoyceV
Merited by mike2077 (1)
 #83

BTW,  I think mega - mega.co.nz give you something like 50GB of storage for free.
That's a terrible site, I used it once to download a large file, it forced me to install their program first. So I prefer a VPS.
I'll let you know when it's available.

Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again.
Update: link expired.

█▀▀▀











█▄▄▄
▀▀▀▀▀▀▀▀▀▀▀
e
▄▄▄▄▄▄▄▄▄▄▄
█████████████
████████████▄███
██▐███████▄█████▀
█████████▄████▀
███▐████▄███▀
████▐██████▀
█████▀█████
███████████▄
████████████▄
██▄█████▀█████▄
▄█████████▀█████▀
███████████▀██▀
████▀█████████
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
c.h.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
▀▀▀█











▄▄▄█
▄██████▄▄▄
█████████████▄▄
███████████████
███████████████
███████████████
███████████████
███░░█████████
███▌▐█████████
█████████████
███████████▀
██████████▀
████████▀
▀██▀▀
NotATether
Legendary
*
Offline Offline

Activity: 1554
Merit: 6532


bitcoincleanup.com / bitmixlist.org


View Profile WWW
June 04, 2021, 04:08:15 AM
 #84

BTW,  I think mega - mega.co.nz give you something like 50GB of storage for free.
That's a terrible site, I used it once to download a large file, it forced me to install their program first. So I prefer a VPS.

Most cloud storage sites can't upload files several GB large well without constantly breaking the connection, and will throttle the download speed even more which makes them unsuitable for downloading those files as well.

Also it's not 50GB of free storage, it's much smaller than that. Most of that free storage is temporary and expires after a year.

..JAMBLER.io..Create Your Bitcoin Mixing
Business Now for   F R E E 
▄█████████████████████████████
█████████████████████████
████▀████████████████████
███▀█████▄█▀███▀▀▀██████
██▀█████▄█▄██████████████
██▄▄████▀▄▄▄▀▀▀▀▀▄▄██████
█████▄▄▄██████████▀▄████
█████▀▄█▄██████▀█▄█████
███████▀▄█▀█▄██▀█▄███████
█████████▄█▀▄█▀▄█████████
█████████████████████████
█████████████████████████
▀█████████████████████████████
█████████████████████████████████████████████████
.
      OUR      
PARTNERS

.
█████████████████████████████████████████████████
████▄
██
██
██
██
██
██
██
██
██
██
██
████▀
▄█████████████████████████████
████████▀▀█████▀▀████████
█████▀█████████████▀█████
████████████████████████
███████████████▄█████████
█████████████████████████
█████████████████████████
█████████████████████████
███████████████▀█████████
████████████████████████
█████▄█████████████▄█████
████████▄▄█████▄▄████████
▀█████████████████████████████
█████████████████████████████████████████████████
.
   INVEST   
BITCOIN

.
█████████████████████████████████████████████████
████▄
██
██
██
██
██
██
██
██
██
██
██
████▀
PrimeNumber7
Copper Member
Legendary
*
Offline Offline

Activity: 1610
Merit: 1899

Amazon Prime Member #7


View Profile
June 04, 2021, 04:30:42 AM
 #85

Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again.
"Your download will take ~4 hours to complete"

IMO, you should upload the file to a GCS/AWS/Azure/Oracle/etc storage bucket, set the permissions to "anyone can access" but set the object so that the "requestor pays" for downloads. This will result in you paying under a dollar per month in storage costs, but anyone who accesses your file will pay a few dollars to get your data in seconds.

Maintainng a multigigabyte file that is accessible to the public for free, that can be accessed unlimited times is really not feasible.
NotATether
Legendary
*
Offline Offline

Activity: 1554
Merit: 6532


bitcoincleanup.com / bitmixlist.org


View Profile WWW
June 04, 2021, 05:54:36 AM
 #86

IMO, you should upload the file to a GCS/AWS/Azure/Oracle/etc storage bucket, set the permissions to "anyone can access" but set the object so that the "requestor pays" for downloads. This will result in you paying under a dollar per month in storage costs, but anyone who accesses your file will pay a few dollars to get your data in seconds. 

Or you can just ask me nicely and I'll host it on my site's public directory.

..JAMBLER.io..Create Your Bitcoin Mixing
Business Now for   F R E E 
▄█████████████████████████████
█████████████████████████
████▀████████████████████
███▀█████▄█▀███▀▀▀██████
██▀█████▄█▄██████████████
██▄▄████▀▄▄▄▀▀▀▀▀▄▄██████
█████▄▄▄██████████▀▄████
█████▀▄█▄██████▀█▄█████
███████▀▄█▀█▄██▀█▄███████
█████████▄█▀▄█▀▄█████████
█████████████████████████
█████████████████████████
▀█████████████████████████████
█████████████████████████████████████████████████
.
      OUR      
PARTNERS

.
█████████████████████████████████████████████████
████▄
██
██
██
██
██
██
██
██
██
██
██
████▀
▄█████████████████████████████
████████▀▀█████▀▀████████
█████▀█████████████▀█████
████████████████████████
███████████████▄█████████
█████████████████████████
█████████████████████████
█████████████████████████
███████████████▀█████████
████████████████████████
█████▄█████████████▄█████
████████▄▄█████▄▄████████
▀█████████████████████████████
█████████████████████████████████████████████████
.
   INVEST   
BITCOIN

.
█████████████████████████████████████████████████
████▄
██
██
██
██
██
██
██
██
██
██
██
████▀
LoyceV (OP)
Legendary
*
Offline Offline

Activity: 3262
Merit: 16315


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
June 04, 2021, 09:28:23 AM
 #87

Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again.
"Your download will take ~4 hours to complete"
I get this (in England):
Code:
-                     0%[                    ] 139.10M  33.9MB/s    eta 10m 15s

Quote
IMO, you should upload the file to a GCS/AWS/Azure/Oracle/etc storage bucket, set the permissions to "anyone can access" but set the object so that the "requestor pays" for downloads. This will result in you paying under a dollar per month in storage costs, but anyone who accesses your file will pay a few dollars to get your data in seconds.
There's 2 problems with that: I don't want to use a creditcard, and I don't want anyone who downloads it to require a creditcard. If I need to charge a few dollars per download, I'd rather set it up myself to accept Bitcoin payments.

Quote
Maintainng a multigigabyte file that is accessible to the public for free, that can be accessed unlimited times is really not feasible.
My other project (List of all Bitcoin addresses with a balance) is closing in on it's 2 TB montly bandwidth limit. I'd hate to have to setup a payment system, especially since this is basically just mirroring data from Blockchair.com. I've never used Torrent from CLI, that might work. It's a privacy problem for the user though: Torrent shares IP addresses get shared with other downloaders, so they need a VPN again.

What bandwidth limitations do you have? Cheesy I don't just want to make problem your problem.

I remember I had another offer:
For living we host high-end enterprise, just in case you need some space or mirrors, you’re welcome if ever in need.
Is this offer still valid?

█▀▀▀











█▄▄▄
▀▀▀▀▀▀▀▀▀▀▀
e
▄▄▄▄▄▄▄▄▄▄▄
█████████████
████████████▄███
██▐███████▄█████▀
█████████▄████▀
███▐████▄███▀
████▐██████▀
█████▀█████
███████████▄
████████████▄
██▄█████▀█████▄
▄█████████▀█████▀
███████████▀██▀
████▀█████████
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
c.h.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
▀▀▀█











▄▄▄█
▄██████▄▄▄
█████████████▄▄
███████████████
███████████████
███████████████
███████████████
███░░█████████
███▌▐█████████
█████████████
███████████▀
██████████▀
████████▀
▀██▀▀
mike2077
Newbie
*
Offline Offline

Activity: 18
Merit: 10


View Profile
June 04, 2021, 02:29:59 PM
 #88

BTW,  I think mega - mega.co.nz give you something like 50GB of storage for free.
That's a terrible site, I used it once to download a large file, it forced me to install their program first. So I prefer a VPS.
I'll let you know when it's available.

Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again.

I've got the file, thank you so much for sharing.
LoyceV (OP)
Legendary
*
Offline Offline

Activity: 3262
Merit: 16315


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
June 04, 2021, 02:38:19 PM
Last edit: June 05, 2021, 07:59:27 AM by LoyceV
 #89

I've got the file, thank you so much for sharing.
You're welcome Smiley

Judging by the 64 GB outgoing traffic, the file was downloaded 3 times. If anyone else wants it:
I'll nuke this VPS tomorrow. It's gone, until I find a more permanent solution.

█▀▀▀











█▄▄▄
▀▀▀▀▀▀▀▀▀▀▀
e
▄▄▄▄▄▄▄▄▄▄▄
█████████████
████████████▄███
██▐███████▄█████▀
█████████▄████▀
███▐████▄███▀
████▐██████▀
█████▀█████
███████████▄
████████████▄
██▄█████▀█████▄
▄█████████▀█████▀
███████████▀██▀
████▀█████████
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
c.h.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
▀▀▀█











▄▄▄█
▄██████▄▄▄
█████████████▄▄
███████████████
███████████████
███████████████
███████████████
███░░█████████
███▌▐█████████
█████████████
███████████▀
██████████▀
████████▀
▀██▀▀
PrimeNumber7
Copper Member
Legendary
*
Offline Offline

Activity: 1610
Merit: 1899

Amazon Prime Member #7


View Profile
June 04, 2021, 06:53:48 PM
Merited by LoyceV (5)
 #90

Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again.
"Your download will take ~4 hours to complete"
I get this (in England):
Code:
-                     0%[                    ] 139.10M  33.9MB/s    eta 10m 15s
The 4 hour quote appears to be the result of my crappy WiFi connection on my back porch. I was able to reproduce a ~10 minute download estimate via a datacenter. I have been able to transfer ~a half terabyte worth of videos stored in a storage bucket in seconds.

Quote from: PN7
IMO, you should upload the file to a GCS/AWS/Azure/Oracle/etc storage bucket, set the permissions to "anyone can access" but set the object so that the "requestor pays" for downloads. This will result in you paying under a dollar per month in storage costs, but anyone who accesses your file will pay a few dollars to get your data in seconds.
There's 2 problems with that: I don't want to use a creditcard, and I don't want anyone who downloads it to require a creditcard. If I need to charge a few dollars per download, I'd rather set it up myself to accept Bitcoin payments.
That is a reasonable desire, however it is something that is more difficult as you are making big data available to the public. Service providers have limited network infrastructure, and need to pay for data sent to the internet, regardless of if they have data caps, or charge you for egress/outgoing data. If you are freely sharing a 10 or 20 GB file(s) using a service provider that does not charge per data transferred, you will eventually get kicked off from that service provider.

Another point is that many people look at bitcoin-related data today. The fact that someone is looking at blockchain data is not the privacy leak that it might have been 10 years ago.

Quote from: PN7
Maintainng a multigigabyte file that is accessible to the public for free, that can be accessed unlimited times is really not feasible.
My other project (List of all Bitcoin addresses with a balance) is closing in on it's 2 TB montly bandwidth limit. I'd hate to have to setup a payment system, especially since this is basically just mirroring data from Blockchair.com.
There is a reason why blockchair throttles downloads, and why they charge as much as they do for an API key.

LoyceV (OP)
Legendary
*
Offline Offline

Activity: 3262
Merit: 16315


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
June 05, 2021, 08:07:25 AM
 #91

Service providers have limited network infrastructure, and need to pay for data sent to the internet, regardless of if they have data caps, or charge you for egress/outgoing data. If you are freely sharing a 10 or 20 GB file(s) using a service provider that does not charge per data transferred, you will eventually get kicked off from that service provider.
Obviously, I won't use a host with "unlimited" bandwidth. That's never real. And I've seen shared hosts that don't allow hosting large files. But for a VPS, I pay for the bandwidth limit, and it's up to the provider to ensure it's profitable for them.

Quote
Another point is that many people look at bitcoin-related data today. The fact that someone is looking at blockchain data is not the privacy leak that it might have been 10 years ago.
I've never used a creditcard for anything crypto-related, and I want to keep it that way.
I found another reason to pay upfront in crypto instead of using my creditcard: The tale of the July 4th surprise $2700 AWS bill.

Quote
There is a reason why blockchair throttles downloads, and why they charge as much as they do for an API key.
They're also in the money making business, and their paying customers pay for the bandwidth used by, well, people like me Tongue

█▀▀▀











█▄▄▄
▀▀▀▀▀▀▀▀▀▀▀
e
▄▄▄▄▄▄▄▄▄▄▄
█████████████
████████████▄███
██▐███████▄█████▀
█████████▄████▀
███▐████▄███▀
████▐██████▀
█████▀█████
███████████▄
████████████▄
██▄█████▀█████▄
▄█████████▀█████▀
███████████▀██▀
████▀█████████
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
c.h.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
▀▀▀█











▄▄▄█
▄██████▄▄▄
█████████████▄▄
███████████████
███████████████
███████████████
███████████████
███░░█████████
███▌▐█████████
█████████████
███████████▀
██████████▀
████████▀
▀██▀▀
NotATether
Legendary
*
Offline Offline

Activity: 1554
Merit: 6532


bitcoincleanup.com / bitmixlist.org


View Profile WWW
June 05, 2021, 02:45:47 PM
 #92

What bandwidth limitations do you have? Cheesy I don't just want to make problem your problem.

My provider gives me a cool 100TB monthly bandwidth cap so I will be fine  Smiley these boxes are designed for highly intensive torrenting.

..JAMBLER.io..Create Your Bitcoin Mixing
Business Now for   F R E E 
▄█████████████████████████████
█████████████████████████
████▀████████████████████
███▀█████▄█▀███▀▀▀██████
██▀█████▄█▄██████████████
██▄▄████▀▄▄▄▀▀▀▀▀▄▄██████
█████▄▄▄██████████▀▄████
█████▀▄█▄██████▀█▄█████
███████▀▄█▀█▄██▀█▄███████
█████████▄█▀▄█▀▄█████████
█████████████████████████
█████████████████████████
▀█████████████████████████████
█████████████████████████████████████████████████
.
      OUR      
PARTNERS

.
█████████████████████████████████████████████████
████▄
██
██
██
██
██
██
██
██
██
██
██
████▀
▄█████████████████████████████
████████▀▀█████▀▀████████
█████▀█████████████▀█████
████████████████████████
███████████████▄█████████
█████████████████████████
█████████████████████████
█████████████████████████
███████████████▀█████████
████████████████████████
█████▄█████████████▄█████
████████▄▄█████▄▄████████
▀█████████████████████████████
█████████████████████████████████████████████████
.
   INVEST   
BITCOIN

.
█████████████████████████████████████████████████
████▄
██
██
██
██
██
██
██
██
██
██
██
████▀
LoyceV (OP)
Legendary
*
Offline Offline

Activity: 3262
Merit: 16315


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
June 05, 2021, 06:29:35 PM
 #93

My provider gives me a cool 100TB monthly bandwidth cap so I will be fine  Smiley
Interesting Cheesy
Can it also handle the occasional update? I have "2 methods" now: one that takes a long time but works, and one that's much faster but gives different results once in a while. And because testing takes so long, I haven't found the problem yet.

█▀▀▀











█▄▄▄
▀▀▀▀▀▀▀▀▀▀▀
e
▄▄▄▄▄▄▄▄▄▄▄
█████████████
████████████▄███
██▐███████▄█████▀
█████████▄████▀
███▐████▄███▀
████▐██████▀
█████▀█████
███████████▄
████████████▄
██▄█████▀█████▄
▄█████████▀█████▀
███████████▀██▀
████▀█████████
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
c.h.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
▀▀▀█











▄▄▄█
▄██████▄▄▄
█████████████▄▄
███████████████
███████████████
███████████████
███████████████
███░░█████████
███▌▐█████████
█████████████
███████████▀
██████████▀
████████▀
▀██▀▀
NotATether
Legendary
*
Offline Offline

Activity: 1554
Merit: 6532


bitcoincleanup.com / bitmixlist.org


View Profile WWW
June 06, 2021, 06:06:14 AM
 #94

Can it also handle the occasional update? I have "2 methods" now: one that takes a long time but works, and one that's much faster but gives different results once in a while. And because testing takes so long, I haven't found the problem yet.

Depends on how much time is "long time".

Leaving it single threaded (as most shell commands already are) will probably be alright as long as it doesn't take more than a few hours.

But if it has obscene memory requirements then it'll be too much for my box. I only got 9 out of 16GB ram free and I hate force-rebooting the notatether.com webserver  Undecided

..JAMBLER.io..Create Your Bitcoin Mixing
Business Now for   F R E E 
▄█████████████████████████████
█████████████████████████
████▀████████████████████
███▀█████▄█▀███▀▀▀██████
██▀█████▄█▄██████████████
██▄▄████▀▄▄▄▀▀▀▀▀▄▄██████
█████▄▄▄██████████▀▄████
█████▀▄█▄██████▀█▄█████
███████▀▄█▀█▄██▀█▄███████
█████████▄█▀▄█▀▄█████████
█████████████████████████
█████████████████████████
▀█████████████████████████████
█████████████████████████████████████████████████
.
      OUR      
PARTNERS

.
█████████████████████████████████████████████████
████▄
██
██
██
██
██
██
██
██
██
██
██
████▀
▄█████████████████████████████
████████▀▀█████▀▀████████
█████▀█████████████▀█████
████████████████████████
███████████████▄█████████
█████████████████████████
█████████████████████████
█████████████████████████
███████████████▀█████████
████████████████████████
█████▄█████████████▄█████
████████▄▄█████▄▄████████
▀█████████████████████████████
█████████████████████████████████████████████████
.
   INVEST   
BITCOIN

.
█████████████████████████████████████████████████
████▄
██
██
██
██
██
██
██
██
██
██
██
████▀
PrimeNumber7
Copper Member
Legendary
*
Offline Offline

Activity: 1610
Merit: 1899

Amazon Prime Member #7


View Profile
June 06, 2021, 07:14:23 AM
 #95

Can it also handle the occasional update? I have "2 methods" now: one that takes a long time but works, and one that's much faster but gives different results once in a while. And because testing takes so long, I haven't found the problem yet.

Depends on how much time is "long time".

Leaving it single threaded (as most shell commands already are) will probably be alright as long as it doesn't take more than a few hours.

But if it has obscene memory requirements then it'll be too much for my box. I only got 9 out of 16GB ram free and I hate force-rebooting the notatether.com webserver  Undecided
Memory/thread constraints should not be an issue. Executing a script remotely on a server optimized for script requirements is trivial, and uploading an output file a single time to your server should not be an issue for a ~20 GB file.

LoyceV (OP)
Legendary
*
Offline Offline

Activity: 3262
Merit: 16315


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
June 06, 2021, 02:29:51 PM
 #96

Leaving it single threaded (as most shell commands already are) will probably be alright as long as it doesn't take more than a few hours.

But if it has obscene memory requirements then it'll be too much for my box. I only got 9 out of 16GB ram free and I hate force-rebooting the notatether.com webserver  Undecided
It uses a bunch of pipes, so the load it causes is more like 2-3 than 1. Memory requirements can be low (then sort uses more tmp files instead). It should be done within a few hours, and doesn't need frequent updates (once every 2 weeks will be fine).

█▀▀▀











█▄▄▄
▀▀▀▀▀▀▀▀▀▀▀
e
▄▄▄▄▄▄▄▄▄▄▄
█████████████
████████████▄███
██▐███████▄█████▀
█████████▄████▀
███▐████▄███▀
████▐██████▀
█████▀█████
███████████▄
████████████▄
██▄█████▀█████▄
▄█████████▀█████▀
███████████▀██▀
████▀█████████
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
c.h.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
▀▀▀█











▄▄▄█
▄██████▄▄▄
█████████████▄▄
███████████████
███████████████
███████████████
███████████████
███░░█████████
███▌▐█████████
█████████████
███████████▀
██████████▀
████████▀
▀██▀▀
NotATether
Legendary
*
Offline Offline

Activity: 1554
Merit: 6532


bitcoincleanup.com / bitmixlist.org


View Profile WWW
June 06, 2021, 06:51:54 PM
 #97

Memory requirements can be low (then sort uses more tmp files instead).

This is what I'm worried about. I remember you writing somewhere along the lines of the sort process takes an obscene amount of memory on this file, or maybe that was in the addresses-with-a-balance project. That's why I'm trying to figure out how much RAM it uses in the worst case.

..JAMBLER.io..Create Your Bitcoin Mixing
Business Now for   F R E E 
▄█████████████████████████████
█████████████████████████
████▀████████████████████
███▀█████▄█▀███▀▀▀██████
██▀█████▄█▄██████████████
██▄▄████▀▄▄▄▀▀▀▀▀▄▄██████
█████▄▄▄██████████▀▄████
█████▀▄█▄██████▀█▄█████
███████▀▄█▀█▄██▀█▄███████
█████████▄█▀▄█▀▄█████████
█████████████████████████
█████████████████████████
▀█████████████████████████████
█████████████████████████████████████████████████
.
      OUR      
PARTNERS

.
█████████████████████████████████████████████████
████▄
██
██
██
██
██
██
██
██
██
██
██
████▀
▄█████████████████████████████
████████▀▀█████▀▀████████
█████▀█████████████▀█████
████████████████████████
███████████████▄█████████
█████████████████████████
█████████████████████████
█████████████████████████
███████████████▀█████████
████████████████████████
█████▄█████████████▄█████
████████▄▄█████▄▄████████
▀█████████████████████████████
█████████████████████████████████████████████████
.
   INVEST   
BITCOIN

.
█████████████████████████████████████████████████
████▄
██
██
██
██
██
██
██
██
██
██
██
████▀
LoyceV (OP)
Legendary
*
Offline Offline

Activity: 3262
Merit: 16315


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
June 06, 2021, 08:01:35 PM
 #98

I'm trying to figure out how much RAM it uses in the worst case.
I can limit sort's memory usage. I'm more worried about the grinding this causes on the hard drive. I have no idea how much data gets read and written to sort 30 GB, but I assume every bit gets pickup up at least several times.

█▀▀▀











█▄▄▄
▀▀▀▀▀▀▀▀▀▀▀
e
▄▄▄▄▄▄▄▄▄▄▄
█████████████
████████████▄███
██▐███████▄█████▀
█████████▄████▀
███▐████▄███▀
████▐██████▀
█████▀█████
███████████▄
████████████▄
██▄█████▀█████▄
▄█████████▀█████▀
███████████▀██▀
████▀█████████
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
c.h.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
▀▀▀█











▄▄▄█
▄██████▄▄▄
█████████████▄▄
███████████████
███████████████
███████████████
███████████████
███░░█████████
███▌▐█████████
█████████████
███████████▀
██████████▀
████████▀
▀██▀▀
NotATether
Legendary
*
Offline Offline

Activity: 1554
Merit: 6532


bitcoincleanup.com / bitmixlist.org


View Profile WWW
June 12, 2021, 07:59:14 AM
 #99

I'm trying to figure out how much RAM it uses in the worst case.
I can limit sort's memory usage. I'm more worried about the grinding this causes on the hard drive. I have no idea how much data gets read and written to sort 30 GB, but I assume every bit gets pickup up at least several times.

On second thought, I just had one of my servers' disks fail a couple days ago (all data was lost), so I'm not comfortable running these updating scripts on the rest of my hardware with all that grinding until I can set up a proper backup plan for my TBs of data.

..JAMBLER.io..Create Your Bitcoin Mixing
Business Now for   F R E E 
▄█████████████████████████████
█████████████████████████
████▀████████████████████
███▀█████▄█▀███▀▀▀██████
██▀█████▄█▄██████████████
██▄▄████▀▄▄▄▀▀▀▀▀▄▄██████
█████▄▄▄██████████▀▄████
█████▀▄█▄██████▀█▄█████
███████▀▄█▀█▄██▀█▄███████
█████████▄█▀▄█▀▄█████████
█████████████████████████
█████████████████████████
▀█████████████████████████████
█████████████████████████████████████████████████
.
      OUR      
PARTNERS

.
█████████████████████████████████████████████████
████▄
██
██
██
██
██
██
██
██
██
██
██
████▀
▄█████████████████████████████
████████▀▀█████▀▀████████
█████▀█████████████▀█████
████████████████████████
███████████████▄█████████
█████████████████████████
█████████████████████████
█████████████████████████
███████████████▀█████████
████████████████████████
█████▄█████████████▄█████
████████▄▄█████▄▄████████
▀█████████████████████████████
█████████████████████████████████████████████████
.
   INVEST   
BITCOIN

.
█████████████████████████████████████████████████
████▄
██
██
██
██
██
██
██
██
██
██
██
████▀
LoyceV (OP)
Legendary
*
Offline Offline

Activity: 3262
Merit: 16315


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
June 12, 2021, 08:16:31 AM
 #100

I'm not comfortable running these updating scripts on the rest of my hardware
No worries, I'll just wait for the right hosting offer again.
This is the reason I don't want to do a lot of testing on my laptop SSD too.

█▀▀▀











█▄▄▄
▀▀▀▀▀▀▀▀▀▀▀
e
▄▄▄▄▄▄▄▄▄▄▄
█████████████
████████████▄███
██▐███████▄█████▀
█████████▄████▀
███▐████▄███▀
████▐██████▀
█████▀█████
███████████▄
████████████▄
██▄█████▀█████▄
▄█████████▀█████▀
███████████▀██▀
████▀█████████
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
c.h.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
▀▀▀█











▄▄▄█
▄██████▄▄▄
█████████████▄▄
███████████████
███████████████
███████████████
███████████████
███░░█████████
███▌▐█████████
█████████████
███████████▀
██████████▀
████████▀
▀██▀▀
Pages: « 1 2 3 4 [5] 6 7 8 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!