List of all Bitcoin addresses ever used - weekly updates work again!

[pbies]

To download dump with crontab I am using this script:

Code:

#!/usr/bin/env bash
# Dependencies: aria2, pv (optional), pigz (optional for faster gzip)
# Purpose: Robust, resumable fetch + verified unpack of LoyceV's latest addresses dump.

set -Eeuo pipefail

URL="http://addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz"
GZ="Bitcoin_addresses_LATEST.txt.gz"
OUT="addrs-with-bal.txt"

# Temporary files
TMP_GZ="$(mktemp --suffix=".gz")"
TMP_OUT="$(mktemp --suffix=".txt")"

cleanup() {
	# Remove only temp files if they still exist
	[[ -f "$TMP_GZ" ]] && rm -f -- "$TMP_GZ" || true
	[[ -f "$TMP_OUT" ]] && rm -f -- "$TMP_OUT" || true
}
trap cleanup EXIT

echo "Checking remote headers..."
# -fSIL: fail on HTTP errors, silent, show headers only, follow redirects
headers="$(curl -fSIL "$URL")"

# Extract Content-Length (if provided)
cl="$(printf '%s\n' "$headers" | awk -F': ' 'BEGIN{IGNORECASE=1} /^Content-Length:/ {print $2; exit}')"
cl="${cl//[$'\r\n ']/}"

if [[ -z "${cl:-}" ]] || ! [[ "$cl" =~ ^[0-9]+$ ]]; then
	echo "No valid Content-Length; continuing without size pre-check." >&2
	cl=""
fi

# If existing file matches remote size -> skip download
if [[ -n "$cl" && -f "$GZ" ]]; then
	local_size="$(stat -c%s "$GZ" 2>/dev/null || echo 0)"
	if [[ "$local_size" == "$cl" ]]; then
		echo "Already up-to-date (size match)."
		# Proceed to unpack existing file
		use_existing=1
		exit
	else
		use_existing=0
	fi
else
	use_existing=0
fi

if [[ "$use_existing" -eq 0 ]]; then
	echo "Downloading with aria2c..."
	# -x16: 16 connections per server, -s16: 16 splits, -k1M: 1 MiB piece size, -c: continue
	aria2c -x16 -s16 -k1M -c -o "$(basename "$TMP_GZ")" -d "$(dirname "$TMP_GZ")" "$URL"

	# Verify size if Content-Length known
	if [[ -n "$cl" ]]; then
		got="$(stat -c%s "$TMP_GZ")"
		if [[ "$got" != "$cl" ]]; then
			echo "Size mismatch: expected $cl, got $got" >&2
			exit 1
		fi
	fi

	# Atomic replace
	mv -f -- "$TMP_GZ" "$GZ"
fi

echo "Unpacking..."
if command -v pigz >/dev/null 2>&1; then
	# Fast parallel gzip
	if command -v pv >/dev/null 2>&1; then
		pv "$GZ" | pigz -dc > "$TMP_OUT"
	else
		pigz -dc "$GZ" > "$TMP_OUT"
	fi
else
	# Fallback to gzip
	if command -v pv >/dev/null 2>&1; then
		pv "$GZ" | gzip -dc > "$TMP_OUT"
	else
		gzip -dc "$GZ" > "$TMP_OUT"
	fi
fi

# Atomic move for final output
mv -f -- "$TMP_OUT" "$OUT"

echo "Sorting..."
./srtu "$OUT"

echo "Done."
# Terminal bell
>&2 printf '\a'

srtu script is:

Code:

#!/usr/bin/env bash
# srtu file_to_sort.txt # file is replaced
FILESIZE=$(stat -c%s "$1") || exit 1
TH=$(echo $(nproc --all)-2 | bc)
echo -n "File size = "
LC_NUMERIC=en_US.UTF-8 printf "%'d" $FILESIZE
F=$(df -h . | tail -1 | awk '{print $4}')
echo " bytes ; Using $TH threads ; Free here: $F"
time pv -cN input -B 200M "$1" | dos2unix -f | TMPDIR=. LC_ALL=C sort -S80% -u --parallel=$TH | pv -cN output -s $FILESIZE > "$1.sorted~"
if [[ -s "$1.sorted~" ]]
then
	mv "$1.sorted~" "$1"
	echo Done.
else
	echo Error!
fi
>&2 printf '\a'

[femtocat]

Worst case,  you'll have to sort it yourself again to get "the specific sort order" you're looking for.

So, i decided to follow your advice, and got a little bit carried away while doing that: instead of doing reasonable thing, deleting some porn and running ordinary 'sort' on the file, i decided to write my own script that will re-sort the whole list in-place.

That script is available here: http://femtocat.ru/bigbeautifulsorter.py
Just run it like this:

Code:

python3 bigbeautifulsorter.py all_Bitcoin_addresses_ever_used_sorted.txt.gz

and it should, after a while, 'repair' this file, making it so that it that passes the 'sort -k' check and is also ~1.8 Gb smaller. While doing that, it will consume ~100Mb of RAM.
Here is the output of /usr/bin/time --verbose i got during the last run:

Code:

	Command being timed: "python3.10 /home/femtocat/yolotrum/bigbeautifulsorter.py all_Bitcoin_addresses_ever_used_sorted.txt.gz"
	User time (seconds): 15464.49
	System time (seconds): 112.53
	Percent of CPU this job got: 91%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 4:44:23
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 96296
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 13228792
	Voluntary context switches: 51772
	Involuntary context switches: 233986
	Swaps: 0
	File system inputs: 144586448
	File system outputs: 185097296
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

There are a few obvious caveats, though:

• 1. Since it modifies the list in-place, any interruption, be it from the power outage or some bug being present in the code, will irreversibly corrupt the file
• 2. It will take a while: as you can see, on my computer it was working for almost 5 hours.
  Interestingly enough, limiting factor here seems to be the compression speed, not the IO performance, so it is possible that on a processor better than my ancient E5-2689 it will work faster.
• 3. The reason script consumes so little (relatively speaking) RAM is because it shuffles data around on-disk. In other words, the load on your hard drive/sdd will be substantial. Be aware of that.

Important note: script is designed to fix the list of addresses that is already mostly sorted. Trying to use it on the list that is not will almost certainly cause it to eat all the memory, crash and corrupt the file it was working on.

[LoyceV]

My server is currently suffering from high load:

Code:

load average: 103.27, 96.75, 99.0

This isn't the first time. I suspect someone is downloading this data with many, MANY threads at the same time. That doesn't work: the data is on SAN storage, which is slow and cheap. Because of this, the server's output is just a few MB/s, much less than it would be if only one thread was used.

If whoever is doing this reads this: please stop it. Just download one thread at a time. Alternatively, I'll be forced to kick IPs, or limit threads per IP in Apache2. I'd rather spend my time on things I enjoy more

Update:
I set iptables to reject more than 10 connections from the same IP address. Let's see how this goes: chances are the connection isn't dropped instantly, so even someone with just one connection may run into this limitation.
I've also DROPped one IP-address from AWS Virginia. Why would someone set curl to connect to the same download 23 times within a second?

[pbies]

...

So, i decided to follow your advice, and got a little bit carried away while doing that: instead of doing reasonable thing, deleting some porn and running ordinary 'sort' on the file, i decided to write my own script that will re-sort the whole list in-place.

That script is available here: http://femtocat.ru/bigbeautifulsorter.py
Just run it like this:

Code:

python3 bigbeautifulsorter.py all_Bitcoin_addresses_ever_used_sorted.txt.gz

and it should, after a while, 'repair' this file, making it so that it that passes the 'sort -k' check and is also ~1.8 Gb smaller. While doing that, it will consume ~100Mb of RAM.
Here is the output of /usr/bin/time --verbose i got during the last run:

Code:

	Command being timed: "python3.10 /home/femtocat/yolotrum/bigbeautifulsorter.py all_Bitcoin_addresses_ever_used_sorted.txt.gz"
	User time (seconds): 15464.49
	System time (seconds): 112.53
	Percent of CPU this job got: 91%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 4:44:23
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 96296
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 13228792
	Voluntary context switches: 51772
	Involuntary context switches: 233986
	Swaps: 0
	File system inputs: 144586448
	File system outputs: 185097296
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

There are a few obvious caveats, though:

• 1. Since it modifies the list in-place, any interruption, be it from the power outage or some bug being present in the code, will irreversibly corrupt the file
• 2. It will take a while: as you can see, on my computer it was working for almost 5 hours.
  Interestingly enough, limiting factor here seems to be the compression speed, not the IO performance, so it is possible that on a processor better than my ancient E5-2689 it will work faster.
• 3. The reason script consumes so little (relatively speaking) RAM is because it shuffles data around on-disk. In other words, the load on your hard drive/sdd will be substantial. Be aware of that.

Important note: script is designed to fix the list of addresses that is already mostly sorted. Trying to use it on the list that is not will almost certainly cause it to eat all the memory, crash and corrupt the file it was working on.

That's a too robust solution for a too little problem.

Also faults are not acceptable at this size of a script.

You need to rework your script few times.

[LoyceV]

The reason it's slow, is because my server has 1 TB NVMe and 2 TB "SAN" network storage. The 2 TB is much slower, and that's where I store these files. I just tested it:

Code:

-                     0%[                    ]  67.00M  2.40MB/s    eta 4h 50m

So I guess you'll just have to wait a few hours. I agree this speed is far too slow, but this is just a "fun" data project and I don't really want to spend more on hosting.

The speed temporarily improved when I was migrated to another server, but it got very slow again. Even slower than the above test.
So I moved this data from slow SAN storage to NVMe. When I start downloading, I get 30 MB/s. But after a few seconds it drops, and keeps dropping until it's just a few MB/s again. When I test the disk with hdparm, I (only) get 200-500 MB/s. But that's not slow enough to explain the very slow downloads. Maybe the webhost is just maxed out on it's bandwith. They offer "unlimited" packages, so I guess I'm just competing against all other users.