[TOOL] ⭐Bitcointalk BRDb score

[dkbit98]

As promised, I've added an emoji representing the country of the localboard in question next to the user's uid.

This is a step in a good direction, but I can't see flag or letters in userscript profiles.
Even if I click on BRDb score in same usercript nothing is shown in pop up window.
So only way for me to see something is to visit profile page on website, and that is time consuming.
I also saw mistakes about other profiles posted earlier, so maybe you should set minimum number of posts that would count someone as local member.

[DYING_S0UL]

As promised, I've added an emoji representing the country of the localboard in question next to the user's uid.

This is a step in a good direction, but I can't see flag or letters in userscript profiles.
Even if I click on BRDb score in same usercript nothing is shown in pop up window.
So only way for me to see something is to visit profile page on website, and that is time consuming.
I also saw mistakes about other profiles posted earlier, so maybe you should set minimum number of posts that would count someone as local member.

None can, because he didn't add it into the userscript's window, he only added the flags to the BRDb site.
Also, if you look carefully you'll notice some of these users aren't assigned any flags yet as they don't met the minimum requirements. Example, admin.
He needs to work hard on setting the right conditions for determining the local, otherwise it would make a big mess, like it did with mine, and many others. So he can't just add it into the userscript yet.

*Ace*, the profile loading became slower, it takes about 3-5 sec to load everything. It just feels slower.

[*Ace*]

As promised, I've added an emoji representing the country of the localboard in question next to the user's uid.

This is a step in a good direction, but I can't see flag or letters in userscript profiles.
Even if I click on BRDb score in same usercript nothing is shown in pop up window.
So only way for me to see something is to visit profile page on website, and that is time consuming.
I also saw mistakes about other profiles posted earlier, so maybe you should set minimum number of posts that would count someone as local member.

I haven't added the tags to the userscript window yet. The fixes I've made are temporary; it's very difficult to find a formula that works for the vast majority of users, so I wait a little longer to see if any more bugs appear, then I put it in the userscript
 

*Ace*, the profile loading became slower, it takes about 3-5 sec to load everything. It just feels slower.

Yes, I've seen it too, but I haven't fixed it yet. Unfortunately, every new implementation makes requests to the DB and slows it down.

[*Ace*]

Have any of you ever encountered the 403 Forbidden error? I was doing some testing but apparently I exceeded the 1 request per second by a long way. So now I'm stuck with the scraper stuck. Do you know how long this ban lasts?

LoyceV you who are more expert than me

[LoyceV]

Have any of you ever encountered the 403 Forbidden error? I was doing some testing but apparently I exceeded the 1 request per second by a long way.

Isn't this just blocking a certain request? If you're going too fast and get rate limited, you get error 429. This lasts only a second as far as I know. If you go even crazier than that (or hit a bug), Cloudflare gives you Error 1015. This one lasts an hour.
I don't remember seeing error 403 on Bitcointalk.

[*Ace*]

Have any of you ever encountered the 403 Forbidden error? I was doing some testing but apparently I exceeded the 1 request per second by a long way.

Isn't this just blocking a certain request? If you're going too fast and get rate limited, you get error 429. This lasts only a second as far as I know. If you go even crazier than that (or hit a bug), Cloudflare gives you Error 1015. This one lasts an hour.
I don't remember seeing error 403 on Bitcointalk.

The other time I got error 429, but it lasted almost a whole day. Today, however, first 503, then 403. Now it seems that this block has been removed, but it lasted almost 4 hours. I need to review the duration of the requests, possibly lengthen them, but a complete cycle already lasts 3 minutes, all this seems strange to me.

[LoyceV]

The other time I got error 429, but it lasted almost a whole day.

Does that mean you keep scraping too fast, or does it persist after you stopped?
I'll test it: if I open 50 topics within a few seconds, I get:

Too fast / overloaded (503)

Most likely, you are accessing the forum too quickly. You are only allowed to access at a rate of one pageload per second on average (somewhat higher bursts).

It is also possible that the server is overloaded or temporarily down.

503 via master1

So it's not Error 429. But it works again right after that.

[*Ace*]

The other time I got error 429, but it lasted almost a whole day.

Does that mean you keep scraping too fast, or does it persist after you stopped?
I'll test it: if I open 50 topics within a few seconds, I get:

Too fast / overloaded (503)

Most likely, you are accessing the forum too quickly. You are only allowed to access at a rate of one pageload per second on average (somewhat higher bursts).

It is also possible that the server is overloaded or temporarily down.

503 via master1

So it's not Error 429. But it works again right after that.

I don't know what's going on inside the scraper. I make a request directly from the recent posts page, and between each page it pauses for 3 seconds. The entire cycle also includes Merit scraping and lasts about 3 minutes. I'm obviously doing something wrong somewhere.

[LoyceV]

I don't know what's going on inside the scraper. I make a request directly from the recent posts page, and between each page it pauses for 3 seconds. The entire cycle also includes Merit scraping and lasts about 3 minutes. I'm obviously doing something wrong somewhere.

Time for some troubleshooting? Log all scrape-requests with it's date, see what you're scraping.

[fillippone]

I saw something strange today:
On a thread, I saw this very strange score for a well-known user:



But of course, when I went into the profile, the result was different, and probably correct.



Going back to the post, I saw the wrong score again.

[*Ace*]

I don't know what's going on inside the scraper. I make a request directly from the recent posts page, and between each page it pauses for 3 seconds. The entire cycle also includes Merit scraping and lasts about 3 minutes. I'm obviously doing something wrong somewhere.

Time for some troubleshooting? Log all scrape-requests with it's date, see what you're scraping.

Code:

[2026-10-20T19:39:33.534Z] OK 923ms https://bitcointalk.org/index.php?action=merit;stats=recent
[2026-10-20T19:39:34.174Z] OK 155ms https://bitcointalk.org/index.php?action=recent;start=0
[2026-10-20T19:40:07.845Z] OK 303ms https://bitcointalk.org/index.php?action=recent;start=10
[2026-10-20T19:40:26.249Z] OK 204ms https://bitcointalk.org/index.php?action=recent;start=20
[2026-10-20T19:40:29.418Z] OK 147ms https://bitcointalk.org/index.php?action=recent;start=30
[2026-10-20T19:40:32.599Z] OK 156ms https://bitcointalk.org/index.php?action=recent;start=40
[2026-10-20T19:40:35.793Z] OK 177ms https://bitcointalk.org/index.php?action=recent;start=50
[2026-10-20T19:40:38.975Z] OK 159ms https://bitcointalk.org/index.php?action=recent;start=60
[2026-10-20T19:40:42.149Z] OK 150ms https://bitcointalk.org/index.php?action=recent;start=70
[2026-10-20T19:40:45.334Z] OK 166ms https://bitcointalk.org/index.php?action=recent;start=80
[2026-10-20T19:40:48.521Z] OK 164ms https://bitcointalk.org/index.php?action=recent;start=90

This is what happens during a normal scraping cycle for merit+post.
On average, it takes about 3 minutes, sometimes even 4 or 5, but in the meantime I’ve encountered the problem we’ve already discussed, even though I’m well below 1 request per second. I make about 11–12 requests over the entire cycle, all spaced about 3 seconds apart, except when the ‘merit’ scraping finishes and the ‘posts’ scraping starts – there’s a cooldown of about 30 seconds.

I saw something strange today:
On a thread, I saw this very strange score for a well-known user:



But of course, when I went into the profile, the result was different, and probably correct.



Going back to the post, I saw the wrong score again.

It was most likely a temporary glitch; I can see the correct score in that thread. Something went wrong within the database at that moment, as the BRDB score doesn’t start at 0.0 but ranges from 1.0 to 10.0; so the fact that 0 appears suggests that the userscript didn’t receive the correct data, though I don’t know why – I’ll have to look into it.

[LoyceV]

This is what happens during a normal scraping cycle for merit+post.
On average, it takes about 3 minutes, sometimes even 4 or 5, but in the meantime I’ve encountered the problem we’ve already discussed, even though I’m well below 1 request per second. I make about 11–12 requests over the entire cycle, all spaced about 3 seconds apart, except when the ‘merit’ scraping finishes and the ‘posts’ scraping starts – there’s a cooldown of about 30 seconds.

None of this looks like it should cause any problems. Are you sure there isn't a background process doing the same thing many more times or something?

[*Ace*]

This is what happens during a normal scraping cycle for merit+post.
On average, it takes about 3 minutes, sometimes even 4 or 5, but in the meantime I’ve encountered the problem we’ve already discussed, even though I’m well below 1 request per second. I make about 11–12 requests over the entire cycle, all spaced about 3 seconds apart, except when the ‘merit’ scraping finishes and the ‘posts’ scraping starts – there’s a cooldown of about 30 seconds.

None of this looks like it should cause any problems. Are you sure there isn't a background process doing the same thing many more times or something?

You're right, there were other processes active, but they don't explain the temporary ban.
In this case, there are three other processes: the modlog scraper, which makes a request every hour, the seclog scraper, which makes a request every hour, and the whoisonline scraper, which makes a request every minute.
So, in total, 14 requests per minute, well below the limit of 1 per second. Perhaps the only problem is that these last 3 overlap with the main scraper, and where possibly 4 processes make the request in the same second, but it is a sporadic peak

[LoyceV]

Perhaps the only problem is that these last 3 overlap with the main scraper, and where possibly 4 processes make the request in the same second, but it is a sporadic peak

That shouldn't be a problem either.

I tried this:

Code:

i=5597000; echo $i; while test $i -le 5597025; do echo "firefox https://bitcointalk.org/index.php?topic=$i"; firefox https://bitcointalk.org/index.php?topic=$i& i=$((i+1)); done

The first 25 pages open correctly, the next 5 give "Too fast / overloaded (503)".

[TryNinja]

You're right, there were other processes active, but they don't explain the temporary ban.
In this case, there are three other processes: the modlog scraper, which makes a request every hour, the seclog scraper, which makes a request every hour, and the whoisonline scraper, which makes a request every minute.
So, in total, 14 requests per minute, well below the limit of 1 per second. Perhaps the only problem is that these last 3 overlap with the main scraper, and where possibly 4 processes make the request in the same second, but it is a sporadic peak

When you used the BitList API, I would see a lot of repeated requests in a short time span that didn't make any sense in my head (i.e querying the same user, in the same date period, 5 times in a row and very fast). Even when you said you stopped querying BitList I would still see a lot of requests from your API Key. At some point, since you said you were moving to building your own solution and because requests kept coming, I had to delete your API Key.

I felt like there was something wrong on your end. Maybe multiple scrapers runing at the same time (i.e you change your code and run the new version, but the old version also keeps running), idk...

[*Ace*]

You're right, there were other processes active, but they don't explain the temporary ban.
In this case, there are three other processes: the modlog scraper, which makes a request every hour, the seclog scraper, which makes a request every hour, and the whoisonline scraper, which makes a request every minute.
So, in total, 14 requests per minute, well below the limit of 1 per second. Perhaps the only problem is that these last 3 overlap with the main scraper, and where possibly 4 processes make the request in the same second, but it is a sporadic peak

When you used the BitList API, I would see a lot of repeated requests in a short time span that didn't make any sense in my head (i.e querying the same user, in the same date period, 5 times in a row and very fast). Even when you said you stopped querying BitList I would still see a lot of requests from your API Key. At some point, since you said you were moving to building your own solution and because requests kept coming, I had to delete your API Key.

I felt like there was something wrong on your end. Maybe multiple scrapers runing at the same time (i.e you change your code and run the new version, but the old version also keeps running), idk...

I swear it wasn't my fault this time. Bitcointalk went down, but it's not *ace*'s fault. 

Jokes aside, I'm new to scraping, so I don't know all the dynamics perfectly, I often do things that don't make sense and I realize it. In the first version of BRDb I had more scraping which I thought I had disabled but obviously they were active, plus I had Claude help me and he had inserted backfills everywhere without my knowledge

[libert19]

Yo, can we have timezone next to wherever time is shown (it appears to be local time, but for the further clarity, like in terms of UTC? Example ^[1.1]). Secondly, what's meant by offline tag next to usernames ^[1.2]? Thirdly, Cyrus has avatar, but BRDB doesn't show it ^[2].

Lastly, suppose, you are checking, 'statistics' of a particular user, could link change to smth like, https://brdb.cc/profile/78147/stats — and when somebody clicks such link, page automatically scrolls there (i.e. stats in this case)?


[1] https://talkimg.com/image/a2GMUz
[2] https://talkimg.com/image/a2GTQq (no need to thank me for the highlight).

[*Ace*]

Yo, can we have timezone next to wherever time is shown (it appears to be local time, but for the further clarity, like in terms of UTC? Example ^[1.1]). Secondly, what's meant by offline tag next to usernames ^[1.2]? Thirdly, Cyrus has avatar, but BRDB doesn't show it ^[2].

Lastly, suppose, you are checking, 'statistics' of a particular user, could link change to smth like, https://brdb.cc/profile/78147/stats — and when somebody clicks such link, page automatically scrolls there (i.e. stats in this case)?


[1] https://talkimg.com/image/a2GMUz
[2] https://talkimg.com/image/a2GTQq (no need to thank me for the highlight).

The time displayed is that of the scraper server, I would not like to touch it to avoid problems for now.

The offline/online tag indicates when the user is seen by the scraper when looking at whoisonline, it rotates every minute, but if you have privacy settings the scraper will never see you. S For example, if you check my profile now, you'll see the green online tag. More up-to-date, last seen: Just now, or if I log out, last seen: 1 minute ago, etc. etc.

For the avatar I have to investigate, it should display exactly the one from Bitcointalk because I take them directly from there

[*Ace*]

brdb update

The avatar bug has been fixed. I assumed all avatars were in .png format, Now even .jpg and .jpeg files are displayed without errors.
I made a new addition to the tool, now you can also see the written posts on the profile page, the links go to Bitcointalk and if you click on the icon next to the title a post viewer opens directly on brdb


https://brdb.cc/explorer/post/67268474

[DYING_S0UL]

brdb update

The avatar bug has been fixed. I assumed all avatars were in .png format, Now even .jpg and .jpeg files are displayed without errors.
I made a new addition to the tool, now you can also see the written posts on the profile page, the links go to Bitcointalk and if you click on the icon next to the title a post viewer opens directly on brdb


https://brdb.cc/explorer/post/67268474

What did you mean by written posts? Like from the archive or directly from the forum? By any chance are you implying Bitlists' archive feature? Or did you simply mean it fetches the posts from BTT and then shows it inside brdb?

Out of curiosity, I tested the URL you provided, and changed the ID and it led me to another valid post. I'm assuming they weren't from your database right? But came from the forum..!

Man I keep forgetting things ! You remember the last time I said that the whole BRDb dashboard takes too much space and I cannot view it? Well it still exists, I still cannot see the graph, unless I decrease the page size to 80%.