List of all Bitcoin addresses with a balance

[LoyceV]

Guys I'm currently at 5.5 TB bandwidth in 8 days. If that continues, I'll reach my monthly limit before the month is over.
I checked some of the most active IP addresses in Apache2's access.log: 4 occur more than 1000 times, 12 occur 100 to 900 times, and many more try less frequently. And that's in less than 24 hours! Who's checking 3 times per minute to see if there's an update, and doing so 24/7?! Why?!

I think I found the abuser again:

Code:

[13/Jun/2026:13:07:36 +0200] "GET /Bitcoin_addresses_LATEST.txt.gz HTTP/1.1" 200 1450300093 "-" "python-requests/2.32.4"

This happens every 3 minutes. It turns out it's the same IP address I DROPped last year:

My bandwidth consumption so far this month is almost 1 TB, which means it should end up at 4-5 TB per month.

Update almost at the end of the month:

Code:

3.92 TB of 16.6 TB Used / 12.68 TB Free

I still can't believe one guy was burning more than this allowed bandwidth!

That was some kid which needed fresh data but didn't made any proper research.

Guess he didn't learn anything in the past 7 months! In a way it's my own fault: I rebooted the server for an update, and didn't restore the same IP tables. I've added it to rc.local now.

[pbies]

@LoyceV

I am running my script (in crontab, each 30 minutes) which I shared earlier,

but it is first checking for file size before update,

so I shouldn't be the one that's downloading constantly.

Other users should use the same script.

Or at least check for different file size before downloading

and do not download if the file size are equal.

The updated script is:

Code:

#!/usr/bin/env bash
# Dependencies: aria2, pv (optional), pigz (optional for faster gzip)
# Purpose: Robust, resumable fetch + verified unpack of LoyceV's latest addresses dump.

set -Eeuo pipefail

URL="http://addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz"
GZ="Bitcoin_addresses_LATEST.txt.gz"
OUT="addrs-with-bal.txt"

# Temporary files
TMP_GZ="$(mktemp --suffix=".gz")"
TMP_OUT="$(mktemp --suffix=".txt")"

cleanup() {
	# Remove only temp files if they still exist
	[[ -f "$TMP_GZ" ]] && rm -f -- "$TMP_GZ" || true
	[[ -f "$TMP_OUT" ]] && rm -f -- "$TMP_OUT" || true
}
trap cleanup EXIT

echo "Checking remote headers..."
# -fSIL: fail on HTTP errors, silent, show headers only, follow redirects
headers="$(curl -fSIL "$URL")"

# Extract Content-Length (if provided)
cl="$(printf '%s\n' "$headers" | awk -F': ' 'BEGIN{IGNORECASE=1} /^Content-Length:/ {print $2; exit}')"
cl="${cl//[$'\r\n ']/}"

if [[ -z "${cl:-}" ]] || ! [[ "$cl" =~ ^[0-9]+$ ]]; then
	echo "No valid Content-Length; continuing without size pre-check." >&2
	cl=""
fi

# If existing file matches remote size -> skip download
if [[ -n "$cl" && -f "$GZ" ]]; then
	local_size="$(stat -c%s "$GZ" 2>/dev/null || echo 0)"
	if [[ "$local_size" == "$cl" ]]; then
		echo "Already up-to-date (size match)."
		# Proceed to unpack existing file
		use_existing=1
		exit
	else
		use_existing=0
	fi
else
	use_existing=0
fi

if [[ "$use_existing" -eq 0 ]]; then
	echo "Downloading with aria2c..."
	# -x16: 16 connections per server, -s16: 16 splits, -k1M: 1 MiB piece size, -c: continue
	aria2c -x16 -s16 -k1M -c -o "$(basename "$TMP_GZ")" -d "$(dirname "$TMP_GZ")" "$URL"

	# Verify size if Content-Length known
	if [[ -n "$cl" ]]; then
		got="$(stat -c%s "$TMP_GZ")"
		if [[ "$got" != "$cl" ]]; then
			echo "Size mismatch: expected $cl, got $got" >&2
			exit 1
		fi
	fi

	# Atomic replace
	mv -f -- "$TMP_GZ" "$GZ"
fi

echo "Unpacking..."
if command -v pigz >/dev/null 2>&1; then
	# Fast parallel gzip
	if command -v pv >/dev/null 2>&1; then
		pv "$GZ" | pigz -dc > "$TMP_OUT"
	else
		pigz -dc "$GZ" > "$TMP_OUT"
	fi
else
	# Fallback to gzip
	if command -v pv >/dev/null 2>&1; then
		pv "$GZ" | gzip -dc > "$TMP_OUT"
	else
		gzip -dc "$GZ" > "$TMP_OUT"
	fi
fi

# Atomic move for final output
mv -f -- "$TMP_OUT" "$OUT"

echo "Sorting..."
./srtu "$OUT"

echo "Done."
# Terminal bell
>&2 printf '\a'

srtu script:

Code:

#!/usr/bin/env bash
# srtu file_to_sort.txt # file is replaced
[[ $# -eq 0 ]] && { echo "Usage: srtu <file>"; exit 1; }
FILESIZE=$(stat -c%z "$1") || exit 1
TH=$(( $(nproc --all) - 2 ))
(( TH < 1 )) && TH=1
F=$(df -h . | tail -1 | awk '{print $4}')
echo "File size = $(numfmt --grouping "$FILESIZE") bytes ; Using $TH threads ; Free here: $F"
time pv -cN input -B 200M "$1" | dos2unix -f | TMPDIR="$(dirname "$1")" LC_ALL=C sort -S80% -u --parallel=$TH | pv -cN output -s "$FILESIZE" > "$1.sorted~"
if [[ -s "$1.sorted~" ]]
then
	cp "$1.sorted~" "$1" && rm -f "$1.sorted~"
	OUTSIZE=$(stat -c%z "$1")
	echo -n "Written: "
	echo "$(numfmt --grouping "$OUTSIZE") bytes"
	echo Done.
else
	echo Error!
	exit 1
fi
>&2 printf '\a'

[LoyceV]

Guys I'm currently at 5.5 TB bandwidth in 8 days. If that continues, I'll reach my monthly limit before the month is over.

Despite banning the abuser, I received an email from my host again:

We are sending you this email because you have exceeded more than 90% of your bandwidth allocation on the virtual server listed below:

I checked my logs: someone downloaded the same 1.7 GB 10 times today. I'll let this one slide.
Someone else downloads it every 15 minutes. That's 112 GB bandwidth in 18 hours:

Code:

- - [01/Jul/2026:18:54:06 +0200] "GET /blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz HTTP/1.1" 200 1766525025 "-" "Python-urllib/3.13

The last abuser I found did this every 3 minutes, now it's 15 minutes apart. I guess it's an improvement, but I've dropped his IP anyway.

To everyone: if you want to download 1.7 GB every 15 minutes or even more frequently, please drop me a PM with an offer, so I can create a server just for you to do this

A new month has started, 412.79 GB bandwidth used so far (today). I'll keep an eye on this, if it keeps getting out of hand, I'll set up automated IP bans. Please behave and test your download scripts so you don't abuse my server.

[pbies]

@LoyceV

That's bad.

They are bad at understanding how bandwidth works.

Were you thinking about ovh? I think they don't have bandwith limit for small VPSes. I didn't seen that for my VPS. I am paying a little over 40 PLN for month, that's ~$10.

[LoyceV]

Were you thinking about ovh?

No, it was just my (failed) sarcasm to point out how stupid it is, but I'd gladly provide them with their own server if they're willing to pay for it

I think they don't have bandwith limit for small VPSes.

That usually means there's a "fair use limit", so I prefer to just know the hard limit. But more bandwidth isn't really a solution for stupidity: downloading the same large file every 3 minutes comes down to 80 Mbit/s continuously. That's just crazy I wouldn't even be surprised if there's no delay in between downloads, and he just downloads it again the moment it finishes.

Looking at my logs, there are many "checks" per minutes from different IP-addresses, to see if it's updated yet. So when I update the file, there's a clear spike in bandwidth each day, and many people bots are downloading the same file simultaneously.

Since my last post here, I've uploaded 277 GB in about 15 hours. That extrapolates to 13.3 TB/month, and should just fit my bandwidth allocation

[examplens]

Looking at my logs, there are many "checks" per minutes from different IP-addresses, to see if it's updated yet. So when I update the file, there's a clear spike in bandwidth each day, and many people bots are downloading the same file simultaneously.

Are you sure this is some kind of abuse and that it's a bad actor?
Is it possible that someone is trying to use this data in another application, and what you see is an automated bot that keeps the information up to date?
How feasible is it to integrate the API and have access enabled only with a specific API key

[LoyceV]

Looking at my logs, there are many "checks" per minutes from different IP-addresses, to see if it's updated yet. So when I update the file, there's a clear spike in bandwidth each day, and many people bots are downloading the same file simultaneously.

Are you sure this is some kind of abuse and that it's a bad actor?

I didn't say that Those are the normal users waiting patiently (and checking frequently) for the file to be updated.

Is it possible that someone is trying to use this data in another application, and what you see is an automated bot that keeps the information up to date?

Yes.

How feasible is it to integrate the API and have access enabled only with a specific API key

I've never created an API, and I kinda don't want to restrict access to data.