[TOOL] ⭐Bitcointalk BRDb score

[*Ace*]

I’ll most likely remove the badge; personally, it was meant to give proper recognition to those who try to keep this forum active every day. I’m sure that acting as a source of merit is no small undertaking and requires consistency.
I don’t have a list of users who are sources of merit, but I could get hold of one, and I don’t think I’ll be sharing it 
Just so I don’t give too much away.
Next week I’ll be heading back to my Batcave to try and sort out the tool once and for all – or break it forever xD

[fillippone]

I’ll most likely remove the badge; personally, it was meant to give proper recognition to those who try to keep this forum active every day. I’m sure that acting as a source of merit is no small undertaking and requires consistency.
I don’t have a list of users who are sources of merit, but I could get hold of one, and I don’t think I’ll be sharing it 
Just so I don’t give too much away.
Next week I’ll be heading back to my Batcave to try and sort out the tool once and for all – or break it forever xD

The point is your algorithm is broken, as you cannot mtracecmerit sources like this.
Even with a generosity index in the positive territory you have a sufficient condition, not a necessary one!

[*Ace*]

I’ll most likely remove the badge; personally, it was meant to give proper recognition to those who try to keep this forum active every day. I’m sure that acting as a source of merit is no small undertaking and requires consistency.
I don’t have a list of users who are sources of merit, but I could get hold of one, and I don’t think I’ll be sharing it 
Just so I don’t give too much away.
Next week I’ll be heading back to my Batcave to try and sort out the tool once and for all – or break it forever xD

The point is your algorithm is broken, as you cannot mtracecmerit sources like this.
Even with a generosity index in the positive territory you have a sufficient condition, not a necessary one!

Unfortunately, you’re absolutely right – the algorithm is faulty. As soon as I get back from holiday, I’ll either sort it out or scrap the whole thing.
In the meantime, I’ll remove this badge – it’s not really necessary, even though it looked nice, but if it causes an error, it becomes a problem.

[*Ace*]

I have a question for those more experienced than me.
Given that I no longer use Bitlist as my data source, but scraping directly from Bitcointalk for thousands and thousands of users would also take a long time, what would be the best way to obtain the necessary data for all users in the shortest possible time?
If I’m not mistaken, the forum allows one request per second; exceeding this limit results in a ban.
Thanks in advance to anyone who can give me a hand with this.

[LoyceV]

what would be the best way to obtain the necessary data for all users in the shortest possible time?

Based on what you need, you're going to have to be selective. You can't update everything all the time. BPIP for instance scrapes profiles on a regular basis, depending on how active the user is.

If I’m not mistaken, the forum allows one request per second; exceeding this limit results in a ban.

You'll get "rate limited", not banned. After a few seconds it should work again. Unless you really push it, in that case Cloudflare bans you for an hour.

[*Ace*]

what would be the best way to obtain the necessary data for all users in the shortest possible time?

Based on what you need, you're going to have to be selective. You can't update everything all the time. BPIP for instance scrapes profiles on a regular basis, depending on how active the user is.

If I’m not mistaken, the forum allows one request per second; exceeding this limit results in a ban.

You'll get "rate limited", not banned. After a few seconds it should work again. Unless you really push it, in that case Cloudflare bans you for an hour.

That’s exactly my problem – being too selective.
What occurred to me was to use the ‘recent posts’ section to scrape profiles based on their most recent posts, without making too many requests on the forum.
However, this wouldn’t update the scores of profiles that haven’t posted for a long time.
I’m not sure if this idea would work well

[TryNinja]

That’s exactly my problem – being too selective.
What occurred to me was to use the ‘recent posts’ section to scrape profiles based on their most recent posts, without making too many requests on the forum.
However, this wouldn’t update the scores of profiles that haven’t posted for a long time.
I’m not sure if this idea would work well

You can have a list on your database with the last activity of an user.

Check recent posts page -> loops posts -> sees TryNinja posted just now -> marks TryNinja with "last post: [date]"

Then you can have a worker process that checks your list once every day. In your list you know that TryNinja posted on [date], if that was 6 month ago, you lower his score or whatever.

[*Ace*]

That’s exactly my problem – being too selective.
What occurred to me was to use the ‘recent posts’ section to scrape profiles based on their most recent posts, without making too many requests on the forum.
However, this wouldn’t update the scores of profiles that haven’t posted for a long time.
I’m not sure if this idea would work well

You can have a list on your database with the last activity of an user.

Check recent posts page -> loops posts -> sees TryNinja posted just now -> marks TryNinja with "last post: [date]"

Then you can have a worker process that checks your list once every day. In your list you know that TryNinja posted on [date], if that was 6 month ago, you lower his score or whatever.

Thank you so much for the advice; I’ll do as you suggested – it seems like a good compromise.
I’ll run continuous loops (which I actually already do for both new posts and merit points), but I’ll tag them by date; after that, I’m not sure whether to check the profile data or not. I’ll have a think about it over the next few days.

[*Ace*]

Good morning everyone, I need to let you know that brdbscore has been temporarily put on hold.
So at the moment you’ll still see the same score, but it won’t be retrieving any more data from the forum until I find a stable solution.
Sorry for the inconvenience

[DYING_S0UL]

Good morning everyone, I need to let you know that brdbscore has been temporarily put on hold.
-snip-

Why is that? Wrong formula?  

Btw, it's probably nothing major, but a simple visual glitch, however a bug is still a bug, so reporting it.
I noticed it yesterday. As you can see from the gif, the score shouldn't be there. And when I refreshed, it went away for a sec, and again came back.
Here is the thread link (https://bitcointalk.org/index.php?topic=5589389.msg66978403#msg66978403).

Note: It only appears, if you try to view the topic from "Show the last topics started by this person (in this case, Me)", but if you go directly to the topic via that link, it won't show.

[*Ace*]

Good morning everyone, I need to let you know that brdbscore has been temporarily put on hold.
-snip-

Why is that? Wrong formula?  

Btw, it's probably nothing major, but a simple visual glitch, however a bug is still a bug, so reporting it.
I noticed it yesterday. As you can see from the gif, the score shouldn't be there. And when I refreshed, it went away for a sec, and again came back.
Here is the thread link (https://bitcointalk.org/index.php?topic=5589389.msg66978403#msg66978403).

Note: It only appears, if you try to view the topic from "Show the last topics started by this person (in this case, Me)", but if you go directly to the topic via that link, it won't show.

Very strange, it should never appear there. And I haven't made any changes to the userscript lately, so I think the bug stems from somewhere else. Have you updated your script manager? If so, which one? Most likely this could be the reason, but to be sure I need this answer from you

[DYING_S0UL]

Very strange, it should never appear there. And I haven't made any changes to the userscript lately, so I think the bug stems from somewhere else. Have you updated your script manager? If so, which one? Most likely this could be the reason, but to be sure I need this answer from you

So have you looked into that post (from your end)? How it behaved? And still behaving? That visual glitch only appears at that particular post only. And it's clickable.
As for updating, No! I haven't updated anything in a while now. The manager is Tampermonkey v5.5.0 and the Userscript is v0.21.0. Note that I'm not using many extensions right now, it's mainly BPIP, then Bitcointalk Modern Theme from fillippone and yours. But I have tested this with those extensions ON/OFF - and they are not the reason for this anomaly.

[TryNinja]

Very strange, it should never appear there. And I haven't made any changes to the userscript lately, so I think the bug stems from somewhere else. Have you updated your script manager? If so, which one? Most likely this could be the reason, but to be sure I need this answer from you

The problem is when you're looking at the thread from "Show the last topics started by this person":

isProfile equals true, because you're on https://bitcointalk.org/index.php?action=profile;threads;u=xxx;sa=showPosts (contains action=profile)

Code:

const isProfile = location.search.includes('action=profile');

So this runs

Code:

mainProfile(false);

Which inserts the score next to an element with the action=merit url:

Code:

document.querySelector('a[href*="action=merit"]');

There is no profile sidebar on that page, so it finds this url:

|

সাম্প্রতিক মেরিটসমূহ

|

Last 500 merit transactions

|

And appends the score next to it.

[dkbit98]

Good morning everyone, I need to let you know that brdbscore has been temporarily put on hold.
So at the moment you’ll still see the same score, but it won’t be retrieving any more data from the forum until I find a stable solution.
Sorry for the inconvenience

This means that we can safely disable it right?
I hope you come up with some solution, this can be very useful tool, so you should try to collaborate with other developers in forum.

[*Ace*]

Very strange, it should never appear there. And I haven't made any changes to the userscript lately, so I think the bug stems from somewhere else. Have you updated your script manager? If so, which one? Most likely this could be the reason, but to be sure I need this answer from you

The problem is when you're looking at the thread from "Show the last topics started by this person":

isProfile equals true, because you're on https://bitcointalk.org/index.php?action=profile;threads;u=xxx;sa=showPosts (contains action=profile)

Code:

const isProfile = location.search.includes('action=profile');

So this runs

Code:

mainProfile(false);

Which inserts the score next to an element with the action=merit url:

Code:

document.querySelector('a[href*="action=merit"]');

There is no profile sidebar on that page, so it finds this url:

|

সাম্প্রতিক মেরিটসমূহ

|

Last 500 merit transactions

|

And appends the score next to it.

Ok, basically the tool thinks it's on the user profile page, I'll have to make a regex so that it excludes some pages where scores are inserted
For example, it mustn’t contain (action=profile;threads)
I reckon it might work

Good morning everyone, I need to let you know that brdbscore has been temporarily put on hold.
So at the moment you’ll still see the same score, but it won’t be retrieving any more data from the forum until I find a stable solution.
Sorry for the inconvenience

This means that we can safely disable it right?
I hope you come up with some solution, this can be very useful tool, so you should try to collaborate with other developers in forum.

For now, the scraper has been disabled until I find a solution, however the tool works and displays scores but they are not updated daily. If anyone wants to work on the project I am happy to give away all the source codes. Maybe they can provide more stability to the project

[Mitchell]

So what exactly is the problem? You need a VPS to run the scraper on?

[*Ace*]

So what exactly is the problem? You need a VPS to run the scraper on?

Yes, basically the problem is this, I need a VPS to run the scraper on.  I looked around and in any case the costs are not low for a small VPS, costs that unfortunately are not covered by the tool's earnings, the tool does not produce any profit. For this reason I stopped it, I already burned a Raspberry Pi, where I initially ran the scraper, then I switched to termux on a mobile device but it is not reliable. The only way is to buy a VPS

[Mitchell]

I'm happy to either host it or send some $$ over to support you

LowEndBox has a lot of cheap VPS' listed. It is where I found my VPS that I still use to this day (among others). Some suggestions I found whilst browsing:

CloudServer

CloudServer has a really cheap plan at $1.50 USD a month:

Code:

LEB LA 512MB SSD Plan
1x vCPU Core
512MB RAM
30GB SSD Storage
1x IPv4
1TB Bandwidth
Linux OS

It is very limited RAM/CPU/storage, but should probably hopefully be enough for a scraper.

LuxVPS

If you need more power, LuxVPS seems decent for €6.99 per month:

Code:

4 vCores (Intel Xeon Gold 6150 / E5-2697 v2 class)
16 GB RAM
100 GB Raid 1 NVMe
1 Gbit internet speed | 20 TB Traffic
1x IPv4
1x /64 IPv6
3.2Tbit Premium DDoS Protection

DISCLAIMER: I have no idea how good these providers are. I am not associated with LowEndBox, CloudServer or LuxVPS. Furthermore, I give zero guarantees if I am to host the scraper.

[*Ace*]

I'm happy to either host it or send some $$ over to support you

LowEndBox has a lot of cheap VPS' listed. It is where I found my VPS that I still use to this day (among others). Some suggestions I found whilst browsing:

CloudServer

CloudServer has a really cheap plan at $1.50 USD a month:

Code:

LEB LA 512MB SSD Plan
1x vCPU Core
512MB RAM
30GB SSD Storage
1x IPv4
1TB Bandwidth
Linux OS

It is very limited RAM/CPU/storage, but should probably hopefully be enough for a scraper.

LuxVPS

If you need more power, LuxVPS seems decent for €6.99 per month:

Code:

4 vCores (Intel Xeon Gold 6150 / E5-2697 v2 class)
16 GB RAM
100 GB Raid 1 NVMe
1 Gbit internet speed | 20 TB Traffic
1x IPv4
1x /64 IPv6
3.2Tbit Premium DDoS Protection

DISCLAIMER: I have no idea how good these providers are. I am not associated with LowEndBox, CloudServer or LuxVPS. Furthermore, I give zero guarantees if I am to host the scraper.

I thank you for your availability and kindness, but I don't want money. It would be like begging for me! That's not what I want.The problem was that I only found VPSs from €7 and up, so almost €90 a year without the tool being able to cover it alone. But now you've given me a fantastic clue, a really cheap VPS! I think the features should do it, it just needs to run a script written in bash in a loop. N Nothing heavy, Termux could do it, this VPS should be able to do it too.
Could you send me a PM with the link where to buy the VPS or put the link directly here?
Hoping it doesn't have any geographical limitations

Edit: sorry, I didn't see the links already in your post.

[Mitchell]

Could you send me a PM with the link where to buy the VPS or put the link directly here?
Hoping it doesn't have any geographical limitations

The links are in the post already, just click on CloudServer (or here it is directly: https://cloudserver.net/billing/index.php?rp=/store/custom-packages/leb-la-512mb-ssd-plan).

If you pick Debian, I would recommend following an upgrade guide to get it to Debian 13.