List of all Bitcoin addresses ever used - weekly updates work again!

[pbies]

To download dump with crontab I am using this script:

Code:

#!/usr/bin/env bash
# Dependencies: aria2, pv (optional), pigz (optional for faster gzip)
# Purpose: Robust, resumable fetch + verified unpack of LoyceV's latest addresses dump.

set -Eeuo pipefail

URL="http://addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz"
GZ="Bitcoin_addresses_LATEST.txt.gz"
OUT="addrs-with-bal.txt"

# Temporary files
TMP_GZ="$(mktemp --suffix=".gz")"
TMP_OUT="$(mktemp --suffix=".txt")"

cleanup() {
	# Remove only temp files if they still exist
	[[ -f "$TMP_GZ" ]] && rm -f -- "$TMP_GZ" || true
	[[ -f "$TMP_OUT" ]] && rm -f -- "$TMP_OUT" || true
}
trap cleanup EXIT

echo "Checking remote headers..."
# -fSIL: fail on HTTP errors, silent, show headers only, follow redirects
headers="$(curl -fSIL "$URL")"

# Extract Content-Length (if provided)
cl="$(printf '%s\n' "$headers" | awk -F': ' 'BEGIN{IGNORECASE=1} /^Content-Length:/ {print $2; exit}')"
cl="${cl//[$'\r\n ']/}"

if [[ -z "${cl:-}" ]] || ! [[ "$cl" =~ ^[0-9]+$ ]]; then
	echo "No valid Content-Length; continuing without size pre-check." >&2
	cl=""
fi

# If existing file matches remote size -> skip download
if [[ -n "$cl" && -f "$GZ" ]]; then
	local_size="$(stat -c%s "$GZ" 2>/dev/null || echo 0)"
	if [[ "$local_size" == "$cl" ]]; then
		echo "Already up-to-date (size match)."
		# Proceed to unpack existing file
		use_existing=1
		exit
	else
		use_existing=0
	fi
else
	use_existing=0
fi

if [[ "$use_existing" -eq 0 ]]; then
	echo "Downloading with aria2c..."
	# -x16: 16 connections per server, -s16: 16 splits, -k1M: 1 MiB piece size, -c: continue
	aria2c -x16 -s16 -k1M -c -o "$(basename "$TMP_GZ")" -d "$(dirname "$TMP_GZ")" "$URL"

	# Verify size if Content-Length known
	if [[ -n "$cl" ]]; then
		got="$(stat -c%s "$TMP_GZ")"
		if [[ "$got" != "$cl" ]]; then
			echo "Size mismatch: expected $cl, got $got" >&2
			exit 1
		fi
	fi

	# Atomic replace
	mv -f -- "$TMP_GZ" "$GZ"
fi

echo "Unpacking..."
if command -v pigz >/dev/null 2>&1; then
	# Fast parallel gzip
	if command -v pv >/dev/null 2>&1; then
		pv "$GZ" | pigz -dc > "$TMP_OUT"
	else
		pigz -dc "$GZ" > "$TMP_OUT"
	fi
else
	# Fallback to gzip
	if command -v pv >/dev/null 2>&1; then
		pv "$GZ" | gzip -dc > "$TMP_OUT"
	else
		gzip -dc "$GZ" > "$TMP_OUT"
	fi
fi

# Atomic move for final output
mv -f -- "$TMP_OUT" "$OUT"

echo "Sorting..."
./srtu "$OUT"

echo "Done."
# Terminal bell
>&2 printf '\a'

srtu script is:

Code:

#!/usr/bin/env bash
# srtu file_to_sort.txt # file is replaced
FILESIZE=$(stat -c%s "$1") || exit 1
TH=$(echo $(nproc --all)-2 | bc)
echo -n "File size = "
LC_NUMERIC=en_US.UTF-8 printf "%'d" $FILESIZE
F=$(df -h . | tail -1 | awk '{print $4}')
echo " bytes ; Using $TH threads ; Free here: $F"
time pv -cN input -B 200M "$1" | dos2unix -f | TMPDIR=. LC_ALL=C sort -S80% -u --parallel=$TH | pv -cN output -s $FILESIZE > "$1.sorted~"
if [[ -s "$1.sorted~" ]]
then
	mv "$1.sorted~" "$1"
	echo Done.
else
	echo Error!
fi
>&2 printf '\a'

[femtocat]

Worst case,  you'll have to sort it yourself again to get "the specific sort order" you're looking for.

So, i decided to follow your advice, and got a little bit carried away while doing that: instead of doing reasonable thing, deleting some porn and running ordinary 'sort' on the file, i decided to write my own script that will re-sort the whole list in-place.

That script is available here: http://femtocat.ru/bigbeautifulsorter.py
Just run it like this:

Code:

python3 bigbeautifulsorter.py all_Bitcoin_addresses_ever_used_sorted.txt.gz

and it should, after a while, 'repair' this file, making it so that it that passes the 'sort -k' check and is also ~1.8 Gb smaller. While doing that, it will consume ~100Mb of RAM.
Here is the output of /usr/bin/time --verbose i got during the last run:

Code:

	Command being timed: "python3.10 /home/femtocat/yolotrum/bigbeautifulsorter.py all_Bitcoin_addresses_ever_used_sorted.txt.gz"
	User time (seconds): 15464.49
	System time (seconds): 112.53
	Percent of CPU this job got: 91%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 4:44:23
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 96296
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 13228792
	Voluntary context switches: 51772
	Involuntary context switches: 233986
	Swaps: 0
	File system inputs: 144586448
	File system outputs: 185097296
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

There are a few obvious caveats, though:

• 1. Since it modifies the list in-place, any interruption, be it from the power outage or some bug being present in the code, will irreversibly corrupt the file
• 2. It will take a while: as you can see, on my computer it was working for almost 5 hours.
  Interestingly enough, limiting factor here seems to be the compression speed, not the IO performance, so it is possible that on a processor better than my ancient E5-2689 it will work faster.
• 3. The reason script consumes so little (relatively speaking) RAM is because it shuffles data around on-disk. In other words, the load on your hard drive/sdd will be substantial. Be aware of that.

Important note: script is designed to fix the list of addresses that is already mostly sorted. Trying to use it on the list that is not will almost certainly cause it to eat all the memory, crash and corrupt the file it was working on.