All used addresses

[naufragus]

please donate to me instead( LOL), i never received a donation lol
no i am not selling data
[it will take me some days (not many, though) to upload every thing to git as of today]

and i think this is a much powerful proof we are NOT
really interested in money, except some private companies may


only one liners,  yeah oh... go learn some more
@loyce you didnt post the resulting lists, how can anybody be sure about; 
@windows/osx ;people please remove windows or mac os

[1713954737]

1713954737

[1713954737]

1713954737

[1713954737]

1713954737

[LoyceV]

I made List of all Bitcoin addresses ever used.

one feature of my lists is i tried to keep the original order in which addresses first appeared in the blockchair dumps..

It works with awk:

Code:

awk '!a[$0]++'

But this requires far too much memory. I can use this on data per day, but not on all data.
So for now, I gave up trying to keep addresses in chronological order. I'll keep the original data in case I find a different solution (or enough RAM) later.

[naufragus]

I made List of all Bitcoin addresses ever used.

one feature of my lists is i tried to keep the original order in which addresses first appeared in the blockchair dumps..

It works with awk:

Code:

awk '!a[$0]++'

But this requires far too much memory. I can use this on data per day, but not on all data.
So for now, I gave up trying to keep addresses in chronological order. I'll keep the original data in case I find a different solution (or enough RAM) later.

Hey that is very nice , bro.
You have got the means, work hard and you are very good at it.
I knew that awk one-liner you wrote, tho i tried using perl because thought that might need less ram..
The sort command can keep chronological order without using ram but it needs a large temp directory (/tmp will not work as it is limited by the system ram value).
ok, cheers!

[LoyceV]

The sort command can keep chronological order without using ram

How? I haven't found that option.

[bob123]

The sort command can keep chronological order without using ram

How? I haven't found that option.

The command sort only takes roughly 50% of your available RAM.
If you are running out of memory using sort on a large file, most likely there isn't enough space on your hard drive.

What sort does, when the file is larger than your available RAM, is that it creates temporary files on the hard drive which are then merge-sorted at the end.
So the overall capacity needed is the sizes of the file times 3 (if you keep the original one) or 2 (if you overwrite the original file).


I might have just misunderstood the problem, might want to elaborate the actual issue?

[LoyceMobile]

I might have just misunderstood the problem, might want to elaborate the actual issue?

I want to remove duplicate lines without changing the order, so only keeping the first occurrence.

Say:
A
G
D
A
B
C
D

I want to keep:
A
G
D
B
C

[bob123]

I want to remove duplicate lines without changing the order, so only keeping the first occurrence.

If i am not mistaken, the following should work:

Code:

cat -n input.txt | sort -uk2 | sort -nk1 | cut -f2- > output.txt

None of these commands needs to hold the file in memory all at once.
But as mentioned previously, sort does need quite some disk space to create temporary files. So that might be a bottleneck, depending on your system specs.

[LoyceV]

Code:

cat -n input.txt | sort -uk2 | sort -nk1 | cut -f2- > output.txt

This looks genius in simplicity! It worked on a small sample, I'm currently transfering and extracting 31 GB of data to do the full test. There's no way the double sort will fit the 100 GB VPS, but this should work without using a lot of RAM.
I'll continu my test results in my own topic: List of all Bitcoin addresses ever used.

[naufragus]

i wrote my methodology for my addresses list with more details in my github repo ..
Here is how i did the sorting..

Code:

$ export TMPDIR='/large/tmp/dir'
$ export LC_ALL=C 
$ nl concat.txt | sort -k2 -u | sort -n | cut -f2 > final.txt

Note that using LC_ALL=C will greatly speed up sorting!