Bitcoin Forum
September 22, 2026, 07:34:23 AM *
News: Latest Bitcoin Core release: 31.1 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 [18]  All
  Print  
Author Topic: [back on new server] List of all Bitcoin addresses with a balance  (Read 13243 times)
This is a self-moderated topic. If you do not want to be moderated by the person who started this topic, create a new topic. (17 posts by 4+ users deleted.)
LoyceV (OP)
Legendary
*
Offline

Activity: 4172
Merit: 22767


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
September 03, 2026, 08:13:38 AM
Last edit: September 03, 2026, 10:52:42 AM by LoyceV
 #341

1. Cloudflare considered to behave just like MITM attack among some people with serious privacy or security concern. But it could be "easy" way to reduce successful abuse attempt.
It could also break any automated scripts that download it. And I really dislike Cloudflare.

Quote
2. Limit the maximum bandwidth per connection or IP. People who only need your data one time or on few occasion more likely to tolerate it than people who regularly download latest file.
I doubt that's going to stop people. Automated scripts don't mind the slower download, and most of the bandwidth goes to different IPs now.



I've restarted Apache2. The moment it's online, downloads started.



These are 10 second averages of outgoing bandwidth (in Mbyte/s):
80,183,775
74,460,381
77,756,118
75,537,366
78,059,662
82,381,260
80,515,415
75,926,504
74,763,770
83,997,099
84,626,537
82,675,131
80,837,847
79,588,349
80,269,984
81,104,141
80,695,840
72,825,527
77,831,898
83,120,619
74,202,390
76,550,071
80,254,527
76,089,717
76,394,077

There's no abuse anymore (like someone downloading the same file every 3 minutes). It's all different IP-addresses. It just looks like demand for this data went up A LOT recently.

I've added a warning on http://addresses.loyce.club/ about outages. Next time I take it offline (which won't take long), I'll just remove the "LATEST" symlink, forcing users to manually check things.

¡uʍop ǝpᴉsdn pɐǝɥ ɹnoʎ ɥʇᴉʍ ʎuunɟ ʞool no⅄
[MUSIC] The Ballad of LoyceV
LoyceV (OP)
Legendary
*
Offline

Activity: 4172
Merit: 22767


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
September 08, 2026, 10:22:04 AM
Last edit: September 09, 2026, 08:43:07 AM by LoyceV
 #342

Torrent is not a bad idea but some short unique name and date in format YYMMDD, like 260902.
If I use torrent, I was thinking of using the same filename for each update. That way, anyone can download the small torrent file as often as they want, and the torrent system does the rest.
I've created Torrent files, let's try this for a while:
addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz.torrent (1689 MB)
addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz.torrent (1393 MB)



I've only turned the LATEST versions into .torrent files. This will for now break all automated daily (or more frequent) downloads, forcing users to check the site to continue.
My server's upload (per thread) is still terribly slow. For now, staying within my bandwidth limits without taking it offline is my main priority.
The LATEST .torrent files are updated daily. I'm hoping someone will post a script to automatically download the LATEST version through Torrent. Don't keep older Torrents seeding, let them die out the next day.



Feel free to suggest more Trackers to add to the .torrent.
Feel free to post anything that can be improved, I don't have much experience creating Torrents.
Please try the torrents! I need some downloaders Smiley



Earlier, while testing, I could connect to the .torrent, but now it doesn't connect anymore. Suggestions?

Update (a day later)
Everything seems to work fine: the new .torrent is created automatically, and starts downloading the moment I click it.
I've only had a few .torrent downloads yesterday, but I expect more users to realize the change in the coming days.

How to download the Torrent
On Linux, I use transmission-daemon with transmission-remote (part of transmission-cli) to create and upload Torrents. You can use the same to download them. Don't forget to remove the old Torrents before downloading the latest version:
Code:
transmission-remote -t all --remove

Bandwidth
My bandwidth consumption dropped about 90% to a few MByte/s. I've also wiped my iptables DROPlist. If "3 minute guy" wants to come back to get a 404 every 3 minutes, that's okay Smiley
Note that the only change I made, was for LATEST, not for all other files. Those are still available for incidental downloads. If someone clicks the link and downloads it, that's totally fine. If you want an update a month later, that's also fine. But if my bandwidth consumption skyrockets again, I'll start logging IP-addresses and automatically DROPping IP-addresses for anyone who should have used the Torrent (and by the way I drop them, it will also DROP your access to my website and Torrent daemon). So just don't abuse it Tongue

I'll update the OP when I decided how to keep this service long-term.

¡uʍop ǝpᴉsdn pɐǝɥ ɹnoʎ ɥʇᴉʍ ʎuunɟ ʞool no⅄
[MUSIC] The Ballad of LoyceV
bitmover
Legendary
*
Offline

Activity: 3164
Merit: 7771


Trêvoid █ No KYC-AML Crypto Swaps


View Profile WWW
September 09, 2026, 10:37:51 AM
 #343

There's no abuse anymore (like someone downloading the same file every 3 minutes). It's all different IP-addresses. It just looks like demand for this data went up A LOT recently.

I've added a warning on http://addresses.loyce.club/ about outages. Next time I take it offline (which won't take long), I'll just remove the "LATEST" symlink, forcing users to manually check things.

I just noticed you have a competitor:
https://bitref.com/bitcoin-addresses-with-balance/

But he charges $199 USD to have access to the information (all addresses with balance).
Maybe charging something wouldn't be a bad idea.

▄▄████████████████████▄▄
▄███████▀▀██████▀▀███████▄
████████████████████████
████████▄▄██████▄▄██████

████████████████████████
██▄▄█████████████▄▄██████
██▀▀██████████████████▄▄██
██████▀▀██████████████▀▀██
██████████████████████████
██████▀▀██████▀▀████████
████████████████████████
▀███████▄▄██████▄▄███████▀
▀▀████████████████████▀▀
 
 DΞX.fo 
▄▄██████
█████████
██████████
█████████
██████████
█████████
▀▀██████

▄███████
▄██████████
████████████
█████████████
█████████████
|
▄▄█
▄████▀
▄███▀
▄██▀▄██
█████▀▀
███████
████████
▀██▄████
▄████▄▄
▄█████▀███
▄█████▀████
█████▀███████
▀██▀█████████
|  BTC     XMR  
  DAI     LTC  
   Fees  0.8%    
LoyceV (OP)
Legendary
*
Offline

Activity: 4172
Merit: 22767


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
September 09, 2026, 11:06:19 AM
 #344

But he charges $199 USD to have access to the information (all addresses with balance).
If someone pays that much for something they can create or download for free, that's on them Tongue

Quote
Maybe charging something wouldn't be a bad idea.
I'd do that, if I'd know how. It was discussed on the previous page:
A thousand downloads per day wouldn't be any problem at all if they all pay $1 per year.

¡uʍop ǝpᴉsdn pɐǝɥ ɹnoʎ ɥʇᴉʍ ʎuunɟ ʞool no⅄
[MUSIC] The Ballad of LoyceV
bitmover
Legendary
*
Offline

Activity: 3164
Merit: 7771


Trêvoid █ No KYC-AML Crypto Swaps


View Profile WWW
September 09, 2026, 11:21:09 AM
 #345

But he charges $199 USD to have access to the information (all addresses with balance).
If someone pays that much for something they can create or download for free, that's on them Tongue

Quote
Maybe charging something wouldn't be a bad idea.
I'd do that, if I'd know how. It was discussed on the previous page:
A thousand downloads per day wouldn't be any problem at all if they all pay $1 per year.

It will demand some work from your side. AI certainly helps.

Easy way:
Put your email and give them the link to download once they send you the btc.

Longer way:
You would probably have to create an account/login system.
Genetate a single address for each account( maybe use btcpayserver or a bitcoin library)

▄▄████████████████████▄▄
▄███████▀▀██████▀▀███████▄
████████████████████████
████████▄▄██████▄▄██████

████████████████████████
██▄▄█████████████▄▄██████
██▀▀██████████████████▄▄██
██████▀▀██████████████▀▀██
██████████████████████████
██████▀▀██████▀▀████████
████████████████████████
▀███████▄▄██████▄▄███████▀
▀▀████████████████████▀▀
 
 DΞX.fo 
▄▄██████
█████████
██████████
█████████
██████████
█████████
▀▀██████

▄███████
▄██████████
████████████
█████████████
█████████████
|
▄▄█
▄████▀
▄███▀
▄██▀▄██
█████▀▀
███████
████████
▀██▄████
▄████▄▄
▄█████▀███
▄█████▀████
█████▀███████
▀██▀█████████
|  BTC     XMR  
  DAI     LTC  
   Fees  0.8%    
LoyceV (OP)
Legendary
*
Offline

Activity: 4172
Merit: 22767


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
September 09, 2026, 12:36:50 PM
 #346

It will demand some work from your side. AI certainly helps.
I don't really like using things I don't understand.

Quote
Easy way:
Put your email and give them the link to download once they send you the btc.
Then I'll have to keep track of who shares their link with whom, and it's terrible for privacy.

Quote
Longer way:
You would probably have to create an account/login system.
Genetate a single address for each account( maybe use btcpayserver or a bitcoin library)
It should still work for automated downloads, as that's how most people use this.

But there's a bigger "problem": I never created this with the intention of earning money from it. I kinda like how this data has become so established that even people on Github link to addresses.loyce.club. Charging money for it would scare off the large majority of users, even if it's just a small amount.
So for now, I'll just try the Torrent solution for a while.

¡uʍop ǝpᴉsdn pɐǝɥ ɹnoʎ ɥʇᴉʍ ʎuunɟ ʞool no⅄
[MUSIC] The Ballad of LoyceV
bitmover
Legendary
*
Offline

Activity: 3164
Merit: 7771


Trêvoid █ No KYC-AML Crypto Swaps


View Profile WWW
September 09, 2026, 01:47:20 PM
 #347

It will demand some work from your side. AI certainly helps.
I don't really like using things I don't understand.

I agree with you on this.

However, you don't have to do use AI this way (asking it do to things you don't understand).

You can use AI to guide you, and AI can teach you how to do it. And do little by little with AI support.

AI is a basically a dumb developer with infinite patience, you can ask AI any question, to the most basic level of understanding.

I am a much better developer since I started using AI, I learned a lot from it and learned much faster than I would without it.

Quote
Quote
Easy way:
Put your email and give them the link to download once they send you the btc.
Then I'll have to keep track of who shares their link with whom, and it's terrible for privacy.

You already have the IP of anyone who access your data. But you don't know which addresses they are searching for, so I don't see any additional privacy problem.

I also don't think people would be trying to cheat your system...

Quote
But there's a bigger "problem": I never created this with the intention of earning money from it. I kinda like how this data has become so established that even people on Github link to addresses.loyce.club. Charging money for it would scare off the large majority of users, even if it's just a small amount.
So for now, I'll just try the Torrent solution for a while.

That is indeed a good point. I also got many backlinks from chatgpt and github to bitcoindata.science, because my tools work and they are free to use.

But, there might be some way to earn from your tool (if you want to). Maybe free up to 5 downloads per month, and then ask for money after 5 downloads, or some other model. Or you can just let everything how it is, I just saw someone charging $199 for the same thing and remembered about your tool.

▄▄████████████████████▄▄
▄███████▀▀██████▀▀███████▄
████████████████████████
████████▄▄██████▄▄██████

████████████████████████
██▄▄█████████████▄▄██████
██▀▀██████████████████▄▄██
██████▀▀██████████████▀▀██
██████████████████████████
██████▀▀██████▀▀████████
████████████████████████
▀███████▄▄██████▄▄███████▀
▀▀████████████████████▀▀
 
 DΞX.fo 
▄▄██████
█████████
██████████
█████████
██████████
█████████
▀▀██████

▄███████
▄██████████
████████████
█████████████
█████████████
|
▄▄█
▄████▀
▄███▀
▄██▀▄██
█████▀▀
███████
████████
▀██▄████
▄████▄▄
▄█████▀███
▄█████▀████
█████▀███████
▀██▀█████████
|  BTC     XMR  
  DAI     LTC  
   Fees  0.8%    
LoyceV (OP)
Legendary
*
Offline

Activity: 4172
Merit: 22767


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
September 09, 2026, 01:58:15 PM
 #348

But, there might be some way to earn from your tool (if you want to). Maybe free up to 5 downloads per month, and then ask for money after 5 downloads, or some other model.
I enjoy projects much more without dealing with accounting systems Wink

In theory, the Torrent solution would be perfect: if 1000 people per day download it, they'll share the majority of bandwidth consumption amongst themselves and my problem is gone Smiley But so far, I only get a few downloads per day. Let's see if this is a deal breaker before trying more drastic measures Smiley

Quote
However, you don't have to do use AI this way (asking it do to things you don't understand).
I've never used and always ignored all chatbots I see Smiley

¡uʍop ǝpᴉsdn pɐǝɥ ɹnoʎ ɥʇᴉʍ ʎuunɟ ʞool no⅄
[MUSIC] The Ballad of LoyceV
pbies
Sr. Member
****
Offline

Activity: 446
Merit: 272



View Profile
September 10, 2026, 02:34:02 PM
 #349

What about some CDN = Content Delivery Network?
For example Amazon has such.

They should host some data with unlimited bw.

It may be more expensive but should work flawlessly...

BTC: bc1qmrexlspd24kevspp42uvjg7sjwm8xcf9w86h5k
LoyceV (OP)
Legendary
*
Offline

Activity: 4172
Merit: 22767


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
September 10, 2026, 05:21:13 PM
 #350

It may be more expensive
You answered your own question. And they don't accept crypto.

¡uʍop ǝpᴉsdn pɐǝɥ ɹnoʎ ɥʇᴉʍ ʎuunɟ ʞool no⅄
[MUSIC] The Ballad of LoyceV
LoyceV (OP)
Legendary
*
Offline

Activity: 4172
Merit: 22767


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
September 12, 2026, 07:34:37 AM
 #351

I was afraid it would come to this:
Note that the only change I made, was for LATEST, not for all other files. Those are still available for incidental downloads. If someone clicks the link and downloads it, that's totally fine. If you want an update a month later, that's also fine. But if my bandwidth consumption skyrockets again, I'll start logging IP-addresses and automatically DROPping IP-addresses for anyone who should have used the Torrent (and by the way I drop them, it will also DROP your access to my website and Torrent daemon). So just don't abuse it Tongue
Instead of just playing nice and implementing the Torrent download, more and more people are downloading "September_12_2026.txt.gz" directly.

I hate that it has to come down to this. I'll collect data for a while, then implement automatic IP DROPs for the users who do this every day. I can already expect the next problem to be people using random IPs.

¡uʍop ǝpᴉsdn pɐǝɥ ɹnoʎ ɥʇᴉʍ ʎuunɟ ʞool no⅄
[MUSIC] The Ballad of LoyceV
LoyceV (OP)
Legendary
*
Offline

Activity: 4172
Merit: 22767


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
September 16, 2026, 08:14:28 AM
Last edit: September 18, 2026, 07:26:13 AM by LoyceV
 #352

The LATEST .torrent files are updated daily. I'm hoping someone will post a script to automatically download the LATEST version through Torrent. Don't keep older Torrents seeding, let them die out the next day.
How's this (quick and dirty):
Code:
#!/bin/bash

# Requirements:
# sudo apt install transmission-daemon transmission-cli curl
# systemctl disable --now transmission-daemon   # Running the daemon as a user makes it easier to connect to

mkdir -p ~/addresses.loyce.club/incomplete; cd ~/addresses.loyce.club/

if [ -z "`ps aux | grep 'transmission-daemon' | grep -v grep | grep -v 'color=auto'`" ]
then  transmission-daemon --incomplete-dir ~/addresses.loyce.club/incomplete --download-dir ~/addresses.loyce.club                                                                                                                                    # Start the daemon if it's not running yet
fi

while test 1
do
      # Bitcoin_addresses_LATEST.txt.gz.torrent                                                                                                                                                                                                       # Remove this block if you only want the other file
            checksumaddresses=$(cat Bitcoin_addresses_LATEST.txt.gz.torrent 2> /dev/null | md5sum)
            curl -s -o Bitcoin_addresses_LATEST.txt.gz.torrent -z Bitcoin_addresses_LATEST.txt.gz.torrent http://addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz.torrent                                                                         # Download when newer
            if [ "$checksumaddresses" != "$(cat Bitcoin_addresses_LATEST.txt.gz.torrent | md5sum)" ]
            then  transmission-remote -t $(transmission-remote --list | grep Bitcoin_addresses_ | cut -c1-8 | cut -d'*' -f1 | sed s/\ //g) --remove 2>/dev/null 1>/dev/null                                                                           # Remove old torrent
                  transmission-remote -a ~/addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz.torrent                                                                                                                                               # Start new torrent
            fi

      # blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz.torrent                                                                                                                                                                                # Remove this block if you only want the other file
            checksumbalances=$(cat blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz.torrent 2> /dev/null | md5sum)
            curl -s -o blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz.torrent -z blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz.torrent http://addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz.torrent    # Download when newer
            if [ "$checksumbalances" !=  "$(cat blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz.torrent | md5sum)" ]
            then  transmission-remote -t $(transmission-remote --list | grep blockchair_bitcoin_addresses_and_balance_ | cut -c1-8 | cut -d'*' -f1 | sed s/\ //g) --remove 2>/dev/null 1>/dev/null                                                    # Remove old torrent
                  transmission-remote -a ~/addresses.loyce.club/blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz.torrent                                                                                                                        # Start new torrent
            fi

      sleep 18m
done

# Monitoring
# while test 1; do clear; transmission-remote -t all  --info-peers; echo; transmission-remote --list; sleep 10; done
I've tested the above script on a second server. It downloads both LATEST versions daily.
Start the script once and keep it running in the background. If there's a newer torrent available, it stops seeding the older one and start downloading the latest version. When it's done, it's moved to ~/addresses.loyce.club/.
I need a few testers before publishing it on my site.



My current VPS will be replaced by a new one. I'm hoping that will improve the download speed, but it means it'll be down for a while.

¡uʍop ǝpᴉsdn pɐǝɥ ɹnoʎ ɥʇᴉʍ ʎuunɟ ʞool no⅄
[MUSIC] The Ballad of LoyceV
Vod
Legendary
*
Offline

Activity: 4536
Merit: 3713


Licking my boob since 1970


View Profile WWW
September 17, 2026, 02:08:58 AM
 #353

I tried to add both torrent links to seed but it got stuck on "Retrieving metadata...".  Have you moved or restricted it?

Edit - site is moving.

███████████████████████████
███████▄████████████▄██████
████████▄████████▄████████
███▀█████▀▄███▄▀█████▀███
█████▀█▀▄██▀▀▀██▄▀█▀█████
███████▄███████████▄███████
███████████████████████████
███████▀███████████▀███████
████▄██▄▀██▄▄▄██▀▄██▄████
████▄████▄▀███▀▄████▄████
██▄███▀▀█▀██████▀█▀███▄███
██▀█▀████████████████▀█▀███
███████████████████████████
.
.Duelbits PREDICT..
█████████████████████████
█████████████████████████
███████████▀▀░░░░▀▀██████
██████████░░▄████▄░░████
█████████░░████████░░████
█████████░░████████░░████
█████████▄▀██████▀▄████
████████▀▀░░░▀▀▀▀░░▄█████
██████▀░░░░██▄▄▄▄████████
████▀░░░░▄███████████████
█████▄▄█████████████████
█████████████████████████
█████████████████████████
.
.WHERE EVERYTHING IS A MARKET..
█████
██
██







██
██
██████
Will Bitcoin hit $200,000
before January 1st 2027?

    No @1.15         Yes @6.00    
█████
██
██







██
██
██████

  CHECK MORE > 
LoyceV (OP)
Legendary
*
Offline

Activity: 4172
Merit: 22767


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
September 17, 2026, 08:07:39 AM
Last edit: September 17, 2026, 09:55:50 AM by LoyceV
 #354

Edit - site is moving.
My site is back! Feel free to start torrenting Smiley
Today's data update will take a few hours to complete.

¡uʍop ǝpᴉsdn pɐǝɥ ɹnoʎ ɥʇᴉʍ ʎuunɟ ʞool no⅄
[MUSIC] The Ballad of LoyceV
Vod
Legendary
*
Offline

Activity: 4536
Merit: 3713


Licking my boob since 1970


View Profile WWW
September 17, 2026, 06:55:11 PM
 #355

Edit - site is moving.
My site is back! Feel free to start torrenting Smiley
Today's data update will take a few hours to complete.

Running now.  I have a 3GB upload so I'll seed for you as needed.

███████████████████████████
███████▄████████████▄██████
████████▄████████▄████████
███▀█████▀▄███▄▀█████▀███
█████▀█▀▄██▀▀▀██▄▀█▀█████
███████▄███████████▄███████
███████████████████████████
███████▀███████████▀███████
████▄██▄▀██▄▄▄██▀▄██▄████
████▄████▄▀███▀▄████▄████
██▄███▀▀█▀██████▀█▀███▄███
██▀█▀████████████████▀█▀███
███████████████████████████
.
.Duelbits PREDICT..
█████████████████████████
█████████████████████████
███████████▀▀░░░░▀▀██████
██████████░░▄████▄░░████
█████████░░████████░░████
█████████░░████████░░████
█████████▄▀██████▀▄████
████████▀▀░░░▀▀▀▀░░▄█████
██████▀░░░░██▄▄▄▄████████
████▀░░░░▄███████████████
█████▄▄█████████████████
█████████████████████████
█████████████████████████
.
.WHERE EVERYTHING IS A MARKET..
█████
██
██







██
██
██████
Will Bitcoin hit $200,000
before January 1st 2027?

    No @1.15         Yes @6.00    
█████
██
██







██
██
██████

  CHECK MORE > 
pbies
Sr. Member
****
Offline

Activity: 446
Merit: 272



View Profile
September 18, 2026, 01:43:08 AM
Last edit: September 18, 2026, 02:30:41 AM by pbies
 #356

@LoyceV

1. When to start the script you've provided? (crontab, daily? which hour is the best?)
2. Will it redownload the same torrent? (stupid question but need to ask; I need postprocessing)

I am getting when manually run:

Unexpected response: <h1>401: Unauthorized</h1>
Unexpected response: <h1>401: Unauthorized</h1>
Unexpected response: <h1>401: Unauthorized</h1>
Unexpected response: <h1>401: Unauthorized</h1>


Fixed script:

Code:
#!/bin/bash

# Keeps the two addresses.loyce.club torrents up to date.
#
# Requirements:
#   apt install transmission-daemon transmission-cli curl
#
# One-time setup (Ubuntu 24.04 / Transmission 4.1.x):
#   systemctl disable --now transmission-daemon          # packaged daemon holds 9091 and requires auth
#   cat > /etc/apparmor.d/local/transmission-daemon <<'EOF'
#   /root/addresses.loyce.club/ rw,
#   /root/addresses.loyce.club/** rwk,
#   EOF
#   apparmor_parser -r /etc/apparmor.d/transmission
#
# The AppArmor profile confines transmission-daemon to
# /var/lib/transmission-daemon/.config/transmission-daemon/ by default; without the
# local override above the daemon cannot read or write $BASE.

set -u

BASE="${BASE:-$HOME/addresses.loyce.club}"
URL="http://addresses.loyce.club"
RPC_PORT="${RPC_PORT:-9092}"
PEER_PORT="${PEER_PORT:-51420}"
CONF="$BASE/config"
LOG="$BASE/transmission.log"
INTERVAL="${INTERVAL:-18m}"
TR=(transmission-remote "127.0.0.1:$RPC_PORT")
FILES=(
"Bitcoin_addresses_LATEST.txt.gz.torrent"
"blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz.torrent"
)

log() { echo "$(date '+%F %T') $*"; }

mkdir -p "$BASE/incomplete" "$CONF" || exit 1
cd "$BASE" || exit 1

write_settings() {
[ -f "$CONF/settings.json" ] && return
cat > "$CONF/settings.json" <<EOF
{
"rpc-enabled": true,
"rpc-port": $RPC_PORT,
"rpc-bind-address": "127.0.0.1",
"rpc-authentication-required": false,
"rpc-whitelist-enabled": true,
"rpc-whitelist": "127.0.0.1,::1",
"download-dir": "$BASE",
"incomplete-dir": "$BASE/incomplete",
"incomplete-dir-enabled": true,
"peer-port": $PEER_PORT,
"peer-port-random-on-start": false,
"dht-enabled": true,
"pex-enabled": true,
"lpd-enabled": true,
"utp-enabled": true,
"encryption": 1,
"download-queue-enabled": false,
"queue-stalled-enabled": false
}
EOF
}

rpc_up() { "${TR[@]}" --session-info >/dev/null 2>&1; }

start_daemon() {
write_settings
transmission-daemon --config-dir "$CONF" --logfile "$LOG"
for _ in $(seq 30); do
rpc_up && return 0
sleep 1
done
return 1
}

if ! rpc_up; then
if ! start_daemon; then
log "transmission RPC not reachable on 127.0.0.1:$RPC_PORT" >&2
tail -n 20 "$LOG" >&2 2>/dev/null
dmesg -T 2>/dev/null | grep -i 'apparmor.*DENIED' | tail -5 >&2
exit 1
fi
log "daemon started (rpc $RPC_PORT, peer $PEER_PORT)"
fi

# IDs of torrents whose name contains $1
get_ids() {
"${TR[@]}" --list 2>/dev/null | awk -v n="$1" 'NR > 1 && $1 != "Sum:" && index($0, n) { gsub(/\*/, "", $1); print $1 }'
}

update() {
local f="$1"
local prefix="${f%%LATEST*}"
local tmp="$BASE/$f.tmp"
local cond=()
local id

[ -f "$f" ] && cond=(-z "$f")

rm -f "$tmp"
if ! curl -fsSL -R "${cond[@]}" -o "$tmp" "$URL/$f"; then
log "download failed: $f" >&2
rm -f "$tmp"
return
fi

# 304 Not Modified, or empty body
if [ ! -s "$tmp" ]; then
rm -f "$tmp"
return
fi

# Unchanged content
if [ -f "$f" ] && cmp -s "$tmp" "$f"; then
rm -f "$tmp"
return
fi

mv -f "$tmp" "$f"

# Drop the old torrent; data stays on disk and the new torrent re-verifies it,
# fetching only the changed pieces.
for id in $(get_ids "$prefix"); do
"${TR[@]}" -t "$id" --remove >/dev/null 2>&1
done

if "${TR[@]}" -a "$BASE/$f" >/dev/null 2>&1; then
log "added: $f"
else
log "add failed: $f" >&2
fi
}

while true; do
for f in "${FILES[@]}"; do
update "$f"
done
sleep "$INTERVAL"
done

# Monitoring:
#   watch -n10 'transmission-remote 127.0.0.1:9092 --list'
#   transmission-remote 127.0.0.1:9092 -t all --info

Defects corrected from the original:

ps aux | grep daemon check matched unrelated processes and the script's own pipeline; replaced with pgrep -u <uid> -f.
No wait for RPC startup — the first transmission-remote call after launching the daemon raced and failed silently.
curl -z against a nonexistent file made -z misbehave on first run; now conditional.
curl -s swallowed HTTP errors; a 404/500 body overwrote the good torrent file. -f plus temp-file staging prevents corruption.
md5sum on a missing file compared garbage; replaced with cmp on staged temp file.
transmission-remote -t $(...) ran with an empty -t argument when no torrent matched, which targets all torrents in some versions; now a loop over parsed IDs, skipping the header and Sum: row.
cut -c1-8 truncated names and broke for IDs of varying width; replaced with awk field extraction.
Duplicated blocks collapsed to a loop over FILES.
Missing set-level failure handling on cd.

BTC: bc1qmrexlspd24kevspp42uvjg7sjwm8xcf9w86h5k
LoyceV (OP)
Legendary
*
Offline

Activity: 4172
Merit: 22767


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
September 18, 2026, 07:35:29 AM
 #357

1. When to start the script you've provided? (crontab, daily? which hour is the best?)
Just start it once:
Start the script once and keep it running in the background. If there's a newer torrent available, it stops seeding the older one and start downloading the latest version. When it's done, it's moved to ~/addresses.loyce.club/.
The time doesn't matter, it gets the latest version.

2. Will it redownload the same torrent? (stupid question but need to ask; I need postprocessing)
No.

Quote
I am getting when manually run:

Unexpected response: <h1>401: Unauthorized</h1>
Unexpected response: <h1>401: Unauthorized</h1>
Unexpected response: <h1>401: Unauthorized</h1>
Unexpected response: <h1>401: Unauthorized</h1>
It requires authentication when transmission-daemon runs as a system service. Just kill it and run it as a user.

Quote
Fixed script:
I prefer a short script, that makes it easier to read. This is a good addition though:
Code:
systemctl disable --now transmission-daemon

Quote
grep daemon check matched unrelated processes and the script's own pipeline
I can't account for all local processes, but grep -v grep makes sure it can't grep itself.

Quote
curl -z against a nonexistent file made -z misbehave on first run; now conditional.
Here too, I prefer a shorter script. It works when the file doesn't exist yet, it tells you about it, and you'll never see that message again. I'm okay with that Smiley

Quote
md5sum on a missing file compared garbage; replaced with cmp on staged temp file.
It worked Wink Your script got 4 times longer, which makes it harder to read.

Quote
cut -c1-8 truncated names and broke for IDs of varying width; replaced with awk field extraction.
This one was annoying indeed. I couldn't just remove the torrent from transmission-daemon based on it's name.

¡uʍop ǝpᴉsdn pɐǝɥ ɹnoʎ ɥʇᴉʍ ʎuunɟ ʞool no⅄
[MUSIC] The Ballad of LoyceV
LoyceV (OP)
Legendary
*
Offline

Activity: 4172
Merit: 22767


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
September 20, 2026, 07:54:23 AM
 #358

~
Why is a subscription needed if it's 100% free?
I've removed your quote with a backlink to the chatbot spammer's website.

¡uʍop ǝpᴉsdn pɐǝɥ ɹnoʎ ɥʇᴉʍ ʎuunɟ ʞool no⅄
[MUSIC] The Ballad of LoyceV
pbies
Sr. Member
****
Offline

Activity: 446
Merit: 272



View Profile
Today at 02:43:51 AM
 #359

I fixed the script but it needs external preparations.

Run with:

Code:
nohup ./script_filename &

Code:
#!/bin/bash

# Keeps the two addresses.loyce.club torrents up to date.
#
# Requirements:
#   apt install transmission-daemon transmission-cli curl
#
# One-time setup (Ubuntu 24.04 / Transmission 4.1.x):
#   systemctl disable --now transmission-daemon          # packaged daemon holds 9091 and requires auth
#   cat > /etc/apparmor.d/local/transmission-daemon <<'EOF'
#   /root/addresses.loyce.club/ rw,
#   /root/addresses.loyce.club/** rwk,
#   EOF
#   apparmor_parser -r /etc/apparmor.d/transmission
#
# The AppArmor profile confines transmission-daemon to
# /var/lib/transmission-daemon/.config/transmission-daemon/ by default; without the
# local override above the daemon cannot read or write $BASE.

set -u

BASE="${BASE:-$HOME/addresses.loyce.club}"
URL="http://addresses.loyce.club"
RPC_PORT="${RPC_PORT:-9092}"
PEER_PORT="${PEER_PORT:-51420}"
CONF="$BASE/config"
LOG="$BASE/transmission.log"
INTERVAL="${INTERVAL:-18m}"
TR=(transmission-remote "127.0.0.1:$RPC_PORT")
FILES=(
      "Bitcoin_addresses_LATEST.txt.gz.torrent"
      "blockchair_bitcoin_addresses_and_balance_LATEST.tsv.gz.torrent"
)

log() { echo "$(date '+%F %T') $*"; }

mkdir -p "$BASE/incomplete" "$CONF" || exit 1
cd "$BASE" || exit 1

write_settings() {
      [ -f "$CONF/settings.json" ] && return
      cat > "$CONF/settings.json" <<EOF
{
      "rpc-enabled": true,
      "rpc-port": $RPC_PORT,
      "rpc-bind-address": "127.0.0.1",
      "rpc-authentication-required": false,
      "rpc-whitelist-enabled": true,
      "rpc-whitelist": "127.0.0.1,::1",
      "download-dir": "$BASE",
      "incomplete-dir": "$BASE/incomplete",
      "incomplete-dir-enabled": true,
      "peer-port": $PEER_PORT,
      "peer-port-random-on-start": false,
      "dht-enabled": true,
      "pex-enabled": true,
      "lpd-enabled": true,
      "utp-enabled": true,
      "encryption": 1,
      "download-queue-enabled": false,
      "queue-stalled-enabled": false
}
EOF
}

rpc_up() { "${TR[@]}" --session-info >/dev/null 2>&1; }

start_daemon() {
      write_settings
      transmission-daemon --config-dir "$CONF" --logfile "$LOG"
      for _ in $(seq 30); do
            rpc_up && return 0
            sleep 1
      done
      return 1
}

if ! rpc_up; then
      if ! start_daemon; then
            log "transmission RPC not reachable on 127.0.0.1:$RPC_PORT" >&2
            tail -n 20 "$LOG" >&2 2>/dev/null
            dmesg -T 2>/dev/null | grep -i 'apparmor.*DENIED' | tail -5 >&2
            exit 1
      fi
      log "daemon started (rpc $RPC_PORT, peer $PEER_PORT)"
fi

# IDs of torrents whose name contains $1
get_ids() {
      "${TR[@]}" --list 2>/dev/null | awk -v n="$1" 'NR > 1 && $1 != "Sum:" && index($0, n) { gsub(/\*/, "", $1); print $1 }'
}

update() {
      local f="$1"
      local prefix="${f%%LATEST*}"
      local tmp="$BASE/$f.tmp"
      local cond=()
      local id

      [ -f "$f" ] && cond=(-z "$f")

      rm -f "$tmp"
      if ! curl -fsSL -R "${cond[@]}" -o "$tmp" "$URL/$f"; then
            log "download failed: $f" >&2
            rm -f "$tmp"
            return
      fi

      # 304 Not Modified, or empty body
      if [ ! -s "$tmp" ]; then
            rm -f "$tmp"
            return
      fi

      # Unchanged content
      if [ -f "$f" ] && cmp -s "$tmp" "$f"; then
            rm -f "$tmp"
            return
      fi

      mv -f "$tmp" "$f"

      # Drop the old torrent; data stays on disk and the new torrent re-verifies it,
      # fetching only the changed pieces.
      for id in $(get_ids "$prefix"); do
            "${TR[@]}" -t "$id" --remove >/dev/null 2>&1
      done

      if "${TR[@]}" -a "$BASE/$f" >/dev/null 2>&1; then
            log "added: $f"
      else
            log "add failed: $f" >&2
      fi
}

while true; do
      for f in "${FILES[@]}"; do
            update "$f"
      done
      sleep "$INTERVAL"
done

# Monitoring:
#   watch -n10 'transmission-remote 127.0.0.1:9092 --list'
#   transmission-remote 127.0.0.1:9092 -t all --info

BTC: bc1qmrexlspd24kevspp42uvjg7sjwm8xcf9w86h5k
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 [18]  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!